Information processing device, information processing method, program

Abstract

This technology provides the ability to evaluate techniques for estimating skeletal information and other data. [Solution] An input image to be processed by a pre-trained model is input, and the system is controlled to display the results of the inference processing on the objects in the input image by the pre-trained model, which has been trained to identify the feature points of objects. In the first setting, the feature points of the objects in the input image identified by the inference processing by the pre-trained model are displayed in association with each other, and in the second setting, the system is controlled to display the feature points of the objects in the input image identified by the inference processing by the pre-trained model in association with the feature points of the ground truth data related to those objects.

Description

Technical Field

[0001] The present invention relates to an information processing system, a control method thereof, and a program.

Background Art

[0002] It is known that there is a technology for estimating the position of joints and skeletons from image data.

[0003] Patent Document 1 discloses that, using a finger prediction model, skeleton information (minor skeleton information) of each finger is extracted from an arbitrary hand region image based on its reference point and image features.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Disclosure of the Invention

Problems to be Solved by the Invention

[0005] In order to grasp accurate skeleton information, a high-performance estimation technology is required. As a prerequisite, a mechanism (evaluation mechanism) for the user to grasp the performance of the estimation technology is necessary. Although Patent Document 1 discloses outputting the extracted skeleton information, it does not disclose the content related to evaluation.

[0006] Therefore, an object of the present invention is to provide a mechanism for evaluating a technology for estimating skeleton information and the like.

Means for Solving the Problems

[0007] The system comprises an input means for inputting an input image to be subjected to inference processing by a trained model, and a display control means for controlling the display of the results of inference processing on an object relating to the input image by a trained model that has been trained to identify the feature points of an object, wherein in a first setting, the display control means displays the feature points of the object relating to the input image identified by the inference processing by the trained model in association with each other, and in a second setting, it controls the display to show the feature points of the object relating to the input image identified by the inference processing by the trained model in association with the feature points relating to the correct data for that object. [Effects of the Invention]

[0008] According to the present invention, it becomes possible to provide a mechanism for evaluating techniques for estimating skeletal information and the like. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the configuration of a posture estimation system in an embodiment of the present invention. [Figure 2] This is a block diagram showing an example of the hardware configuration of a client terminal 101 in an embodiment of the present invention. [Figure 3] This flowchart shows an example of a process for switching the display of inference results in an embodiment of the present invention. [Figure 4] This figure shows an example of a user operation screen for displaying inference results in an embodiment of the present invention. [Figure 5] This figure shows an example of a screen for displaying image data in an embodiment of the present invention. [Figure 6] This figure shows an example of an image of the folder for storing trained models in an embodiment of the present invention. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0011] Figure 1 shows an example of the configuration of a posture estimation system in an embodiment of the present invention.

[0012] The client terminal 101 and the server 102 are configured to be connected via the network 100, enabling communication between them.

[0013] The client terminal 101 can be any device that incorporates the functions shown in Figure 2, and may be, for example, a personal computer (hereinafter referred to as PC), a mobile device such as a smartphone, or a tablet device.

[0014] Network 100 can take the form of a wired LAN, wireless LAN, USB, or other configurations depending on the physical interface of server 102.

[0015] Server 102 can store trained models, configuration files for storing setting parameters used during training, dataset information, and inference results in the inference dataset. The trained models and other items that can be stored in Server 102 may also be stored in the ROM 202 or external memory 211 of the client terminal 101.

[0016] Figure 2 is a block diagram showing an example of the hardware configuration of a client terminal 101 in an embodiment of the present invention.

[0017] Figure 2 is a block diagram showing an example of the hardware configuration of a client terminal 101 (a client terminal is an example of an information processing device) in an embodiment of the present invention. The file server 102 has a similar configuration.

[0018] As shown in FIG. 2, each information processing device has a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an input controller 205, a video controller 206, a memory controller 207, and a communication I / F controller 208 connected via a system bus 204.

[0019] The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

[0020] The ROM 202 or the external memory 211 holds a BIOS (Basic Input / Output System), an OS (Operating System), which are control programs executed by the CPU 201, a computer-readable and executable program for implementing this information processing method, and various necessary data (including data tables).

[0021] The RAM 203 functions as the main memory, work area, etc. of the CPU 201. When executing processing, the CPU 201 loads a program, etc. necessary for the execution from the ROM 202 or the external memory 211 into the RAM 203, and realizes various operations by executing the loaded program.

[0022] The input controller 205 controls inputs from input devices such as a keyboard 209, a mouse (not shown), and a pointing device such as a touch pad. When the input device is a touch panel, it is assumed that the user can give various instructions by pressing (touching with a finger, etc.) in accordance with icons, cursors, buttons, etc. displayed on the touch panel.

[0023] Also, the touch panel may be a touch panel capable of detecting positions touched with multiple fingers, such as a multi-touch screen.

[0024] The video controller 206 controls the display to an external output device such as the display 210. The display may include the display of a notebook PC integrated with the main unit. The external output device is not limited to a display; for example, it may be a projector. Furthermore, for the aforementioned touch-enabled device, an input device is also provided.

[0025] In the flowchart-based explanation described later, the display destination will be display 210 unless otherwise specified.

[0026] The video controller 206 can control the video memory (VRAM) used for display control. It can utilize a portion of the RAM 203 as the video memory area, or it can provide a separate, dedicated video memory.

[0027] The memory controller 207 controls access to the external memory 211. The external memory can include an external storage device (hard disk), a flexible disk (FD), or a CompactFlash® memory connected to a PCMCIA card slot via an adapter, which stores boot programs, various applications, font data, user files, editing files, and other data.

[0028] The communication interface controller 208 connects to and communicates with external devices via the network 214 (for example, network 101 shown in Figure 1), and executes communication control processing on the network 214. For example, it can communicate using TCP / IP, telephone lines such as ISDN, and 3G, 4G, and 5G mobile phone lines.

[0029] The CPU 201 enables display on the display 210 by, for example, performing the process of expanding (rasterizing) outline fonts into the display information area in RAM 203. The CPU 201 also enables user input via a mouse cursor (not shown) on the display 210.

[0030] Before explaining the flowchart in Figure 3, we will use Figure 6 to illustrate an example of a diagram illustrating the folder where trained models are stored.

[0031] The trained model storage folder 601 contains trained model storage files 602, which are separated by job ID. Figure 6 shows the trained model storage files for job ID 1 and job ID 2.

[0032] The trained model storage file 602 contains the trained model 603, configuration file 604, dataset information 605, and inference results 606.

[0033] The trained model 603 is an AI model that has been trained on the training dataset stored in the dataset information 605.

[0034] Configuration file 604 stores the parameters set during training as a file.

[0035] Dataset information 605 is a file containing the training dataset and the inference dataset, and also contains the image folder 607 and the annotation folder 609.

[0036] Image folder 607 stores the image data 608 used for training or inference. In Figure 6, the image data is stored in JPG format, but it may also be stored in other formats such as PNG. The file name of image data 608 corresponds to the file name of annotation file 610, and in Figure 6, it is named "IMG_001".

[0037] The annotation folder 609 contains the annotation file 610. The filename of the annotation file 610 corresponds to the filename of the image data 608, and in Figure 6, it is named "IMG_001".

[0038] In this embodiment, the key points are features that are important for understanding the structure and shape of an object to which the present invention applies. For example, in the case of the human body, these would include joints in the skeleton, major points that form the skeleton (such as the head, hips, toes, fingertips, etc., which are not specific joint locations but are necessary for identifying the skeleton), and organs such as the eyes, nose, and ears.

[0039] Annotation refers to the process of adding the locations of key points to training images.

[0040] Therefore, the key points of the ground truth data described in this embodiment are the key points that were added through annotation work when creating the training data, and the key points of the inference results are the key points output as a result of the trained model performing inference processing on the input image. The screen viewed by the user (image data display unit 418) displays the key points and an image in which those key points are connected by lines. The way the lines are connected will be explained in Figure 5 below.

[0041] In this embodiment, we will explain using an example where the joints of the fingers were inferred as feature points from an image of a hand, but the application is not limited to the hand; it can be applied to any part of the body, or even the whole body. Furthermore, it is not limited to the human body, but can be applied to other organisms as well. It can also be applied to techniques for inferring movable parts of tools, robots, and the like.

[0042] The annotation file stores keypoint information of the correct data for the corresponding image data 608. In other words, it stores predetermined position information and label names related to the correct data annotated to the image data 608. For example, for a joint related to the right wrist, it stores coordinate information of the correct data placed at the joint and information such as a label name indicating that the joint is related to the right wrist.

[0043] The inference result 606 stores the results of inference performed using the inference dataset. Specifically, it stores information regarding inference accuracy (information displayed in the inference data accuracy display unit 411), information regarding the reliability of the inference result (information displayed in the score display unit 415), and keypoint information of the inference result.

[0044] In this system's learning function, users train the AI ​​model with the necessary information. The trained AI model (trained model 603) and learning artifacts (configuration file 604, dataset 605) are managed using an identifier called a Job ID. The trained model 603 performs inference for each Job ID based on the learning artifacts and generates inference results 606. The inference results 606 are also managed in association with the Job ID.

[0045] When the inference using the trained model is complete and the client terminal 101 receives a command to start processing, the process shown in Figure 3 begins.

[0046] Next, the processes executed by the client terminal 101 in the embodiment of the present invention will be described using Figures 3 to 5.

[0047] The process performed by the attitude estimation system in the embodiment of the present invention will be explained using the flowchart in Figure 3.

[0048] The flowchart in Figure 3 shows an example of the process by which the CPU 201 of the client terminal 101 reads and executes a predetermined control program, and the process of switching the display of the inference result. Note that the processing of each step is executed by the CPU 201 of each device.

[0049] This pose estimation system is equipped with learning and inference functions using a pre-trained model. Figure 3 is a flowchart showing the process that is executed when the system receives a start command after AI learning and inference have been completed.

[0050] In S301, CPU201 accepts the selection of a job ID from the user. In S302-S304, CPU201 obtains the image, annotations, and inference results 606 corresponding to the selected job ID.

[0051] In S302, the CPU 201 acquires image data 608 from the dataset used by the trained model for inference. The image data 608 is stored in the server 102, associated with the job ID selected in S301. The CPU 201 displays the acquired image data on the display 210. In this embodiment, the CPU 201 previews the acquired image data on the inference data preview display unit 410 of the verification data list 4a.

[0052] In S303, CPU201 retrieves annotations from the dataset used by the trained model for inference. The annotations are stored as annotation files 610 in the annotation folder 609 and are managed in association with the image data used by the trained model for inference. Specifically, CPU201 retrieves the ground truth data and inference results (inferred keypoints) for each image data obtained in S302.

[0053] In S304, CPU201 retrieves the inference results from the dataset used by the trained model for inference. The inference results 606 are managed in association with the job ID and store information regarding inference accuracy (information displayed in the inference data accuracy display unit 411), information regarding the confidence level of the inference results (information displayed in the score display unit 415), and keypoint information of the inference results. CPU201 displays these inference results on the screen in association with the image data acquired in S302.

[0054] In S305, the CPU 201 accepts the selection of image data to be displayed on the image data display unit 418. The CPU 201 then displays the selected image data. Specifically, the CPU 201 displays the image data selected by the user from among the image data previewed on the inference data preview display unit 410 on the image data display unit 418.

[0055] In S306, CPU201 accepts the user's selection of keypoints to be displayed superimposed on the image data acquired in S305. Specifically, the user can choose to display either one of the keypoints from the ground truth data acquired in S303 or the keypoints from the inference results acquired in S304 (first setting), or to display both the keypoints from the inference results and the ground truth data (second setting). If a selection is accepted for both the inference results and the ground truth data, the process proceeds to S307; if a selection is accepted for either the inference results or the ground truth data, the process proceeds to S308.

[0056] In S307, CPU201 connects the inference results and keypoints in the ground truth data with lines. The processing results of S307 will be described later using a specific example in Figure 5.

[0057] In S308, if CPU201 receives a selection of inference results from the user, it connects the key points of the inference results with lines. If it receives a selection of correct data, it connects the key points of the correct data with lines. The processing results of S308 will be described later using a specific example in Figure 5.

[0058] In S309, the CPU 201 displays keypoint information on the screen. Specifically, the result of connecting the keypoints with lines in S307 or S308 is displayed on the image data display unit 418, and the comparison result between the keypoint information related to the inference result acquired in S303 and the keypoint information related to the correct answer data is displayed in keypoint information (4b).

[0059] Details of the S309 process are explained in Figure 4.

[0060] Figure 4 shows an example of a screen displaying inference results, etc. Note that Figure 4 is a cropped portion of the screen for illustrative purposes, and there is data in the verification data list 4a and key point information 4b that is not shown in Figure 4.

[0061] The screen in question consists of a screen switching unit 401, a display switching unit 402, a key point size change unit 403, a line color change unit 404, a line display switching unit 405, a job ID selection unit 406, a job description unit 407, a tag 408, a verification data list 4a, key point information 4b, and an image data display unit 418.

[0062] First, we will explain the screen switching section 401, the display switching section 402, the key point size change section 403, the line color change section 404, and the line display switching section 405 located on the left side of the screen.

[0063] The screen switching unit 401 is a part that can switch to either a screen for executing the learning function or a screen for executing the analysis (inference) function. When a click is received from the pull-down menu, the options "Learning" and "Analysis" are displayed. When either option is clicked, the screen switches according to the selected option. In Figure 4, "Analysis" is selected, and the screen for executing the analysis (inference) function is displayed. If "Learning" is selected, the screen for executing the learning function is displayed. In this embodiment, only the case when "Analysis" is selected will be described.

[0064] The display switching unit 402 is a part that can switch the display of key points superimposed on the image data display unit 418 (S306). Specifically, it can switch between displaying the key points of the correct data, displaying the key points of the inference results, displaying both the key points of the correct data and the inference key points, or not displaying any key points. The display of the lines connecting the key points also switches according to the key point display switch. When the user presses a dropdown menu, the options (correct answer 402a and inference 402b) are displayed. When an option is pressed, the key points displayed on the image data display unit 418 switch according to the selected option. In Figure 4, both the correct answer 402a and inference 402b are selected, and both the key points of the correct data and the key points of the inference results are displayed on the image data display unit 418. To deselect an option, the "×" button on the right end of the option is pressed to deselect each option individually. To deselect all options within the display switching unit 402, pressing the "×" button at the right end of the display switching unit 402 will deselect all options.

[0065] In this way, users can switch the key points displayed according to their purpose. For example, if they want to know the difference between the correct data and the inference result, they can display the key points of both the correct data and the inference result; if they want to check the correct data, they can display only the key points of the correct data; and if they want to check the inference result, they can display only the key points of the inference result.

[0066] Details regarding the lines connecting the keypoints displayed on the image data display unit 418 are explained in Figure 5.

[0067] The keypoint size adjustment unit 403 allows the user to change the size of the object indicating the position of the keypoint displayed in the image data display unit 418. The size can be changed from 5 to 50, with smaller numbers indicating smaller keypoints. The user can intuitively adjust the size of the keypoint by dragging the pointer left or right. In Figure 4, it is set to 25. If the keypoint is large, it is easier to find the keypoint within the image data, but the keypoint and the inference target or the keypoints themselves may overlap, making them difficult to see. On the other hand, if the keypoint is small, it is less likely that the keypoint and the inference target or the keypoints themselves will overlap, but the keypoint itself may be difficult to find. The function to change the size of the keypoint allows the user to adjust the size of the keypoint to a size that is easy to see, according to the distribution of the inference target and keypoints.

[0068] In the line color change section 404, it is possible to change the color of the lines connecting keypoints.

[0069] When the line color change section 404 is pressed, the available colors are displayed, and the user can select any color. The selection method could be either by selecting from a color chart displayed by the user, or by selecting a color adjusted by setting RGB values. In either case, the user can select a line color that is easy to see depending on the color of the inference target or keypoint.

[0070] Furthermore, it may be possible to limit the color of the lines connecting keypoints to a color different from the color of the keypoints themselves. Having different colors for the lines connecting keypoints and the keypoints themselves makes it easier for users to distinguish between the lines and the keypoints.

[0071] The line display switching section 405 allows switching between displaying or not displaying lines connecting keypoints in the image data display section 418. When the user clicks a dropdown menu, the options "Yes" and "No" are displayed. When an option is clicked, the screen switches according to the selected option. In Figure 4, "Yes" is selected, and lines connecting keypoints are displayed in the image data display section 418. The user can switch the display according to their purpose; for example, they can hide the lines connecting keypoints if they only want to check individual keypoints, or display the lines connecting keypoints if they want to check the distance between keypoints.

[0072] Thus, screen switching, key point display switching, key point size changes, line color changes, and line display switching operations can all be performed intuitively by the user.

[0073] Next, we will explain the job ID selection section 406, the job description section 407, and the tag 408.

[0074] The Job ID Selection Unit 406 is the part where the user can select the Job ID to be used for inference (analysis) (S301). The Job ID is an identifier used to manage the trained model 603, configuration file 604, dataset information 605, and inference result 606 used when performing inference. When the pull-down menu is clicked, the selectable Job IDs are displayed. Once the user has selected a Job ID, information is displayed in the Job Description Unit 407, Tag 408, Validation Data List 4a, Keypoint Information 4b, and Image Data Display Unit 418 according to the selected Job ID. In Figure 4, the Job ID named "1" is selected, and the Job Description Unit 407, Tag 408, Validation Data List 4a, Keypoint Information 4b, and Image Data Display Unit 418 display data corresponding to "1".

[0075] The job description section 407 and tag section 408 display a description related to the job ID selected in the job ID selection section 406. For example, the content learned during job training is displayed as a description or tag related to the job ID. The description related to the job ID is set by the user during training.

[0076] As shown in Figure 4, the job description section 407 and the tag 408 may be left blank and can be set as needed.

[0077] Next, the verification data list 4a will be explained. The verification data list 4a is a list consisting of an inference data name display unit 409, an inference data preview display unit 410, an inference data accuracy display unit 411, and an image data selection unit 417.

[0078] The inference data name display unit 409 displays the file name of the image data to be inferred, which is the "Name" column in the verification data list 4a.

[0079] The inference data preview display unit 410 is a display unit that previews the image data to be inferred, and is the "Preview" column in the verification data list 4a. Specifically, it previews the image data acquired in S302. It displays data corresponding to the file name of the inference target displayed in the inference data name 409. The user can select the image data to be displayed in the image data display unit 418 by referring to the image data displayed in the inference data preview display unit 410. Furthermore, even after selecting image data, the user can easily check what image data is being displayed in the image data display unit 418.

[0080] The image data selection unit 417 is the part that selects the image to be displayed in the image data display unit 418, and is the checkbox to the left of the inference data name display unit 409 in Figure 4. When the user clicks a blank checkbox, the image data corresponding to the clicked row is selected and displayed in the image data display unit 418. The entire selected column is highlighted, making it easy for the user to identify which image data they have selected. If the selected checkbox is clicked again while image data is selected, the selection is deselected, and the image displayed in the image data display unit 418 is hidden. If a different checkbox is clicked while image data is selected, the image data corresponding to that other checkbox is displayed in the image data display unit 418.

[0081] The inference data accuracy display unit 411 is the part that displays the inference results acquired in S304, and is the column labeled "PCK," "AUC," and "EPE" in Figure 4. "PCK" and "AUC" are indices that show the accuracy of the inference results, with higher values ​​indicating higher accuracy. "EPE" is an indices that represents the error rate of the inference results, with lower numbers indicating less error (higher accuracy). The "PCK," "AUC," and "EPE" displayed in the inference data accuracy display unit 411 are generally indices used to measure the accuracy of the inference results of the pose estimation AI, but other indices may also be used.

[0082] Next, the keypoint information 4b will be explained. The keypoint information 4b includes a keypoint name display unit 412, a keypoint display selection unit 413, a line length display unit 414, a score display unit 415, and a training image count display unit 416, and displays information regarding the inference accuracy of each keypoint related to the image selected in S305.

[0083] The content displayed in keypoint information 4b is not affected by which keypoints are displayed (or not displayed) by the display switching unit 402. In other words, whether only the keypoints of the correct data are displayed, only the keypoints of the inference results are displayed, or both the keypoints of the correct data and the inference results are displayed, the content displayed in keypoint information 4b remains unchanged. For example, if only the keypoints of the inference results are displayed, the line length display unit 414 will still display nothing because the keypoints of the correct data are not displayed. In this embodiment, an example where the displayed content does not change has been described, but values ​​may be hidden or changed as needed.

[0084] The keypoint name display unit 412 displays the name of each keypoint (keypoint name), which is the "Name" column in the keypoint information 4b. Keypoint names are derived from the joints and skeletons being inferred, such as "wrist" for wrist and "thumb_cmc" (an abbreviation of thumb-carpometacarpal) for carpometacarpal of the thumb. The background color of the keypoint name display unit is different for each keypoint name, i.e., each joint and skeleton that the keypoint represents. The background color of the keypoint name display unit is the same as the color of the keypoint displayed in the image data display unit 418, and the same background color is never used for different keypoint names. Therefore, the user can identify all keypoints in the image data. For example, the background color of the keypoint name "thumb_cmc" is light blue, and the keypoint corresponding to the keypoint name "thumb_cmc" displayed in the image data display unit 418 is also light blue. This makes it easy for the user to see which keypoint name corresponds to which keypoint in the image data.

[0085] Furthermore, the background color of the keypoint name display and the color of the keypoints may be changed to any color the user chooses. This allows the user to select a color that is easy to see depending on the image data and the inference target, making it easier to distinguish between the inference target and the keypoints.

[0086] The keypoint display selection section 413 is a section where it is possible to select whether or not to display a keypoint in the image data display section 418 for each keypoint, and it corresponds to the "Visible" column in the keypoint information 4b. When a blank checkbox is clicked, the keypoint corresponding to the clicked checkbox is selected and displayed in the image data display section 418. When a checkbox that is already clicked is clicked, the selection is deselected, and the keypoint displayed in the image data display section 418 is hidden. Figure 4 shows the state where all checkboxes corresponding to all keypoints are clicked, and all keypoints are displayed.

[0087] The keypoint display selection unit 413 allows the user to adjust the number of keypoints displayed to a number that is easy to view, according to their purpose. For example, if displaying all keypoints makes the inference results difficult to see, only specific keypoints can be displayed, or if the user wants to check the inference results for the entire image, all keypoints can be displayed.

[0088] The distance display unit 414 displays the distance between the keypoints of the inference results and the keypoints of the ground truth data, and is the "Distance" column in the keypoint information 4b. A numerical value greater than or equal to 0 is displayed, and the accuracy of the inference by the trained model can be checked by the value displayed in the distance display unit 414. The lower the value (i.e., the shorter the line), the smaller the difference (hereinafter sometimes referred to as distance) between the ground truth data and the inference results, indicating higher accuracy. The higher the value (i.e., the longer the line), the larger the difference between the ground truth data and the inference results, indicating lower accuracy. The user can check the difference between the ground truth data and the inference results from the value displayed in the distance display unit 414 and check which keypoints are being inferred accurately and which keypoints are not being inferred accurately. By looking at these results, the user can take measures such as increasing the amount of training data or reviewing annotations in areas where the difference between the ground truth data and the inference results is large. In other words, displaying the difference between the ground truth data and the inference results in a way that users can easily recognize contributes to creating a higher-performance AI model.

[0089] Furthermore, if a joint or skeleton is hidden by a shadow or other object being inferred, and the keypoint of the correct data cannot be seen from the image data being inferred, the value displayed in the line length display unit 414 will be "None". For example, in the image displayed in the image data display unit 418, the metacarpophalangeal joint of the little finger is hidden on the underside of the back of the hand, and the keypoint of the correct data corresponding to the metacarpophalangeal joint of the little finger, "little_finger_mcp", is not visible. Therefore, the value displayed in the line length display unit 414 corresponding to little_finger_mcp is None (not shown).

[0090] The score display section 415 displays the confidence level of the inference result and is the "Score" column in the keypoint information 4b. The numerical value displayed in the score display section 415 is between 0 and 1, with a higher value indicating higher confidence in the inference result from the trained model, and a lower value indicating lower confidence in the inference result from the trained model.

[0091] The distance between the correct data and the inference result displayed in the line length display unit 414, and the confidence level of the inference result displayed in the score display unit 415, are correlated when the training data is normal. A smaller value indicating the distance between the correct data and the inference result indicates higher inference accuracy. Therefore, it is expected that a correlation will be observed where, for example, the smaller the value indicating the distance between the correct data and the inference result (higher inference accuracy), the higher the confidence level of the inference result. However, if no correlation is observed, it is expected that there is some problem with the training data, and the user can use this as an opportunity to review the training data.

[0092] By checking the correlation, it is possible to confirm whether there are any deficiencies in the annotations applied to the images selected by the image data selection unit 417, whether the amount of training data is sufficient, and so on.

[0093] If there is a correlation, that is, if the value of the line length display unit 414 is small and the value of the score display unit 415 is large (high accuracy and high reliability of the inference result), it can be confirmed that there is no discrepancy in the annotation during training and that the amount of training data is sufficient. If the value of the line length display unit 414 is large and the value of the score display unit 415 is small (low accuracy and low reliability of the inference result), it indicates that although a correlation is observed, the accuracy and reliability of the inference result are low due to some reason such as insufficient training data, and improvement is needed.

[0094] If there is no correlation, the following improvement measures can be considered. For example, if the value of the line length display section 414 is large and the value of the score display section 415 is large (low accuracy and high confidence in the inference result), it is possible that the annotations applied to the selected image during training are misaligned. Therefore, by applying annotations to the correct positions and retraining, it may be possible to improve to a state where accuracy is high and the confidence in the inference result is high. Also, if the value of the line length display section 414 is small and the value of the score display section 415 is small (high accuracy and low confidence in the inference result), it is possible that the amount of training data is insufficient. Therefore, in order to further improve confidence, the user needs to increase the number of images to which annotations related to low-confidence keypoints are applied during training.

[0095] In the example shown in Figure 4, the keypoint named "thumb-tip" has a value of 0.02 displayed in the line length display section 414, and a value of 0.79 displayed in the score display section 415 corresponding to "thumb-tip" (confidence level of the inference result). A correlation is observed between distance and the confidence level of the inference result, indicating that the user is learning correctly. On the other hand, one could also view the low value of 0.02 as indicating a lower confidence level than expected. In that case, measures can be taken to increase the confidence level, such as by increasing the amount of training data.

[0096] In this way, by comparing the distance between the ground truth data and the inference result (the accuracy of the inference by the trained model) and the confidence level of the inference result, it is possible to check for anomalies in the training data and use this information to add or modify the training data as needed.

[0097] It should be noted that the countermeasures after confirming the correlation are merely examples. In this invention, the distance between the correct data and the inference result, and the reliability of the inference result displayed in the score display unit 415 are shown, but the presence or absence of correlation and the countermeasures are left to the user to consider.

[0098] Furthermore, a notification may be output if no correlation is found. For example, a correlation coefficient can be calculated between the value in the line length display unit 414 and the value in the score display unit 415 using a known method. If the calculated correlation coefficient is less than a pre-set threshold, a notification is output indicating that the training data should be reviewed. The notification may, for example, identify and display the rows related to keypoint names for which no correlation is found, and notify the user to increase the training data or review the dataset information 605. The method of notification is not limited to this, and any method that suggests to the user that they should review the training data is acceptable.

[0099] In this embodiment, when displaying only the inference result keypoints for the keypoint name "wrist," two keypoints are marked, one on the right hand and one on the left hand. In this case, the accuracy of the inference by the trained model displayed on the line length display unit 414 and the reliability of the inference result displayed on the score display unit 415 are the average values ​​of the left and right hands. Thus, even when displaying only the inference result or the correct answer data, if two or more keypoints are displayed on the image data display unit 418, the accuracy of the inference by the trained model displayed on the line length display unit 414 and the reliability of the inference result displayed on the score display unit 415 are the average values ​​of those two or more keypoints.

[0100] Furthermore, the accuracy of the inference by the trained model displayed in the line length display unit 414 and the confidence level of the inference result displayed in the score display unit 415 may be displayed separately for each of the two or more marks. For example, in Figure 4, the values ​​for the right hand and the left hand are displayed separately. This means that the confidence level of the inference result and the accuracy of the inference by the trained model may differ between the right and left hands, allowing for the confirmation of more accurate data. In addition, the results for the right and left hands can be compared and checked to determine which hand has higher confidence in the inference result, higher accuracy in the inference, etc.

[0101] The training image count display unit 416 displays the number of images to which annotations have been added for each keypoint, and is the AnnTotal column in the keypoint information 4b. An integer greater than or equal to 0 is displayed, and a higher number indicates a larger number of images to which annotations have been added. Specifically, it counts the number of labels assigned to each keypoint in the annotation folder 609. For example, for the keypoint "wrist", in some image data 608 the wrist may be cut off and therefore cannot be annotated. The number of images in the image folder 607 that have been annotated for the wrist (i.e., the number of images in the annotation folder 609 with the label name "wrist") is counted and displayed on the training image count display unit 416.

[0102] This allows us to examine the distribution of the number of images annotated with each keypoint. Users can identify annotations with low values ​​(i.e., few images annotated with them) and take measures such as increasing the number of images containing those annotations used for training.

[0103] Alternatively, a threshold for the number of training images can be set, and if the value is below the threshold, the training image display unit 416 that includes the value below the threshold may be identified and displayed as indicating that the number of annotated images is insufficient.

[0104] The image data display unit 418 displays the image data selected by pressing the image data selection unit 417 from the list of verification data (S305). The upper left of the image data display unit 418 displays "Image name: egocentric_0030," showing the file name of the selected image data. This allows the user to view the image data while confirming which image data has been selected. Also, at the bottom of the image data display unit 418, it says "Correct answers are displayed with circles, inferences with squares," clearly indicating that key points of correct data are displayed with circles, and key points of inference results are displayed with squares. The color of the displayed key points is the same as the background color of each key point name displayed in the key point name display unit 412. Therefore, it is easy for the user to see which key point name corresponds to which key point in the image data.

[0105] The contents displayed on the image data display unit 418 are explained in detail in Figure 5.

[0106] Figure 5 shows an example of a screen displaying image data. Figure 5 is a cropped view of the image data display unit 418 from Figure 4.

[0107] Screen 5a displays the keypoints of the correct data and the keypoints of the inference results (S307). The keypoints of the correct data are displayed as circles, and the keypoints of the inference results are displayed as squares. By using different symbols for the keypoints of the correct data and the keypoints of the inference results, it becomes easier for the user to distinguish between them. The keypoints of the correct data and the keypoints of the inference results with the same keypoint name are the same color, making it easier to recognize the correspondence between keypoints.

[0108] In screen 5a, keypoint 501 in the correct data, which corresponds to the metacarpophalangeal joint of the middle finger and has the keypoint name middle_finger_mcp in the right hand, and keypoint 503 in the inference result, which also corresponds to the keypoint name middle_finger_mcp in the right hand, are connected by line 502. Similarly, keypoint 504 in the correct data and which corresponds to the keypoint name middle_finger_mcp in the left hand, and keypoint 506 in the inference result, which also corresponds to the keypoint name middle_finger_mcp in the left hand, are connected by line 505. In this way, keypoints in the correct data and the corresponding keypoints in the inference result are displayed by connecting them with lines.

[0109] Furthermore, if the object of inference is divided into two or more parts, that is, if there are a total of four or more key points for the correct answer and key points for inference, the key points for each divided object of inference will be connected by lines. For example, if the object of inference is divided into a right hand and a left hand, the key points within the right hand will be connected by lines, and the key points within the left hand will be connected by lines.

[0110] By displaying the results in this way, users can more easily understand the distance between the inference result and the correct data (the difference between the inference result and the correct data), making it easier to evaluate the accuracy of the inference process. In other words, where the length of the line connecting the key points of the inference result and the correct data is long, it can be said that the accuracy is low, and it becomes possible to make the user aware that measures need to be taken to improve the accuracy.

[0111] This invention is useful not only for comparing keypoints within image data, but also for comparing inference results across multiple image data sets. For example, it can compare the inference results of image data A with those of image data B to determine which image has higher inference accuracy, or whether there are differences in inference accuracy even for the same keypoint names.

[0112] When displaying the keypoints of the correct data and the keypoints of the inference results on screen 5a, any display format is acceptable as long as it clearly shows that the keypoints are related to each other. In addition to displaying the keypoints connected by lines as in this embodiment, other methods are possible, for example, when the user requests that only the keypoints corresponding to a specific keypoint name (e.g., middle_finger_mcp) be displayed, the keypoints of the inference results and the keypoints of the correct data corresponding to that keypoint name may be displayed blinking or highlighted.

[0113] Screen 5b displays only the keypoints of the inference results (S308). In screen 5b, keypoint 501 of the inference results corresponding to the keypoint name middle_finger_mcp is connected to keypoint 507 of the inference results corresponding to the keypoint name wrist by line 508. In this way, in screens that display only specific keypoints (unlike screens that display both the ground truth data and inference results as in Figure 5a), keypoints are displayed by connecting them according to the shape of the skeleton. This display format makes it easier to understand whether the positions of the keypoints have been identified in the appropriate locations based on the shape of the skeleton.

[0114] Screen 5b may display only the keypoints of the correct data. This allows the user to see where the keypoints are displayed when the reasoning is accurate.

[0115] Thus, the present invention has the effect of visualizing the accuracy of the inference results and facilitating user verification by having the CPU 201 associate key points and display them superimposed on the image data.

[0116] As described above, this embodiment allows for easy confirmation of evaluation results. Specifically, when displaying both the correct data and the inference results, the differences between the correct data and the inference results are clearly displayed by connecting the corresponding key points of the correct data and the key points of the inference results with lines. When displaying only the inference results, the results are clearly displayed to show whether annotations have been placed in the appropriate positions along the inference target, enabling appropriate evaluation of the inference results.

[0117] The present invention can take the form of, for example, a system, apparatus, method, program, or recording medium. Specifically, it may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device.

[0118] Furthermore, the various controls described above, which are performed by CPU201, may be performed by a single piece of hardware, or multiple pieces of hardware (for example, multiple processors or circuits) may share the processing to control the entire device.

[0119] Furthermore, the program in this invention is a program that allows a computer to execute the processing method shown in the flowchart in Figure 3, and the storage medium of this invention stores a program that allows a computer to execute the processing method shown in Figure 3. Note that the program in this invention may also be a program for each processing method of each device shown in Figure 1.

[0120] As described above, it goes without saying that the objectives of the present invention can also be achieved by supplying a recording medium containing a program that realizes the functions of the embodiments described above to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0121] In this case, the program read from the recording medium itself realizes the novel function of the present invention, and the recording medium on which that program is recorded constitutes the present invention.

[0122] For recording media used to supply programs, examples include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon disks, and the like.

[0123] Furthermore, it goes without saying that the functions of the aforementioned embodiments are realized not only by the computer executing the program it has read, but also by the operating system (OS) running on the computer performing some or all of the actual processing based on the instructions of that program, thereby realizing the functions of the aforementioned embodiments.

[0124] Furthermore, it goes without saying that this also includes cases where, after a program read from a recording medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer, the CPU or other components of the function expansion board or function expansion unit perform some or all of the actual processing based on the instructions of the program code, and the functions of the aforementioned embodiments are realized through that processing.

[0125] Furthermore, the present invention may be applied to a system composed of multiple devices or to a device consisting of a single device. It goes without saying that the present invention can also be applied when the results are achieved by supplying a program to a system or device. In this case, by reading a recording medium containing a program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

[0126] Furthermore, by downloading and reading the program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. It should be noted that configurations combining the above-described embodiments and their variations are all included in the present invention. [Explanation of symbols]

[0127] 101 Client terminals 102 Servers

Claims

1. An input means for inputting an input image to be used for inference processing by a trained model, A display control means that controls the display of the results of inference processing on an object related to the input image, using a trained model that has learned the feature points of the object. Equipped with, In the first setting, the display control means displays the feature points of objects related to the input image, which have been identified by the inference processing by the trained model, by associating them with each other. In the second setting, the information processing device is characterized by controlling the display of feature points relating to an object in the input image, which have been identified by inference processing using the trained model, in association with feature points relating to the correct data for that object.

2. The first setting indicates that the feature points identified by the inference process will be displayed, and the feature points related to the correct data pertaining to the object will not be displayed. The information processing apparatus according to claim 1, characterized in that the second setting is a setting that indicates the display of feature points identified by the inference process and feature points related to the correct data relating to the object.

3. The display control means is characterized by controlling the display of an object at the position of the feature point, The information processing apparatus according to claim 1, characterized in that the display control means controls the display of feature points in association with each other by connecting the objects related to the feature points with lines.

4. The information processing apparatus according to claim 3, characterized in that objects relating to feature points at the same location on the object are controlled to be displayed in the same color.

5. The information processing apparatus according to claim 3, characterized in that it is possible to change at least one of the following: the size of the object relating to the feature point, the thickness of the line connecting the objects relating to the feature point, and the color of the line connecting the objects relating to the feature point.

6. The information processing apparatus according to claim 3, characterized in that the display control means controls the display of the color of the object relating to the feature point and the color of the line connecting the objects relating to the feature point in different colors.

7. The information processing device according to claim 1, characterized in that the aforementioned feature point is a point indicating the position of a joint of the object.

8. The input means of the information processing device inputs an input image that is the subject of inference processing by a trained model, The display control means of the information processing device controls the display to display the results of inference processing on an object relating to an input image by a trained model that has learned the feature points of the object, Equipped with, In the first setting, the display control means displays the feature points of objects related to the input image, which have been identified by the inference processing by the trained model, by associating them with each other. The second setting is an information processing method characterized by controlling the display of feature points relating to an object in the input image, which have been identified by inference processing using the trained model, in association with feature points relating to the correct data for that object.

9. A program for causing at least one computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 7.

Images