Information processing system, information processing method and program

Abstract

The present invention improves the accuracy of a machine learning model while facilitating organization of training data.　This information processing system refers to an output, which is produced by a machine learning model trained by training data in response to input of input data to the machine learning model, so as to output the reliability of the output for the input data, generates new training data on the basis of the input data if the reliability satisfies a predetermined condition, and trains the machine learning model by the new training data.

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] A typical machine learning model is trained using pre-prepared training data.

[0003] Sida Peng et al. published a paper titled "PVNet: Pixel-Wise Voting Network for 6DoF Pose Estimation" at the 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). This paper discloses that a machine learning model is trained using training data including input images and correct output images, and that the positions of key points on the image used for pose estimation are calculated based on the output when a captured image is input to the machine learning model. Summary of the Invention [Problem to be solved by the invention]

[0004] A large amount of training data is required to train a machine learning model, but preparing that data takes a lot of effort. On the other hand, reducing the amount of training data can lead to a risk of not being able to ensure the accuracy of the machine learning model.

[0005] The present invention has been made in consideration of the above-mentioned situation, and its purpose is to provide a technology that improves the accuracy of machine learning models while reducing the effort required to prepare training data. [Means for solving the problem]

[0006] In order to solve the above problem, the information processing system of the present invention includes a machine learning model trained using training data, a reliability output means for outputting the reliability of the output for input data based on the output of the machine learning model when the input data is input, a generation means for generating new training data based on the input data if the reliability satisfies a predetermined condition, and a learning control means for training the machine learning model using the new training data.

[0007] In one aspect of the present invention, the information processing system includes a trained estimation model that outputs an estimation result based on the output of the machine learning model, and the reliability output means may output the reliability of the output of the machine learning model for the input data based on the output of the estimation model.

[0008] In one aspect of the present invention, the input data includes an image of a target object, the estimation model outputs an image showing key points for estimating the posture of the target object based on the output of the machine learning model, and the confidence output means outputs the confidence based on the image.

[0009] In one aspect of the present invention, the estimation model outputs an image in which each point indicates its positional relationship with a keypoint, and the reliability output means may output the reliability based on the variability of a plurality of candidate keypoint positions, each candidate keypoint position being generated from different points included in the image output by the estimation model.

[0010] In one aspect of the present invention, the reliability output means may output the reliability based on information indicating the difference between the output of the estimation model when an input image of a target object is input to the estimation model and the output when a processed image obtained by processing the input image using a predetermined processing process is input to the estimation model.

[0011] In one aspect of the present invention, the machine learning model may output information indicating whether the input data includes the target object.

[0012] In one aspect of the present invention, the estimation model is trained using estimation training data, and the generation means generates new estimation training data based on the input data when the reliability satisfies the specified condition, and the learning control means trains the machine learning model using the new training data.

[0013] In one aspect of the present invention, the input data includes an image of a target object, the machine learning model outputs an image showing key points for estimating the posture of the target object based on the input data, and the confidence output means outputs the confidence based on the image.

[0014] In one aspect of the present invention, the training data may include a plurality of learning images rendered from a three-dimensional shape model and ground truth images, each of which is ground truth data for the learning images.

[0015] In one aspect of the present invention, the generation means generates new training data including a first additional image obtained by processing the input data using a first processing process and a second additional image obtained by processing the input data using a second processing process different from the first processing process, and the learning control means may train the machine learning model based on the difference between the output when the first additional image is input into the machine learning model and the output when the second additional image is added to the machine learning model.

[0016] In addition, the information processing method of the present invention includes a step of outputting the reliability of the output for input data based on the output of a machine learning model trained using training data when input data is input to the machine learning model, a step of generating new training data based on the input data if the reliability satisfies a predetermined condition, and a step of training the machine learning model using the new training data.

[0017] In addition, the program of the present invention causes a computer to execute the following process: based on the output of a machine learning model trained using training data when input data is input to the machine learning model, output the reliability of the output for the input data; if the reliability satisfies a predetermined condition, generate new training data based on the input data, and train the machine learning model using the new training data. [Effects of the Invention]

[0018] According to the present invention, it is possible to improve the accuracy of a machine learning model while reducing the effort required to prepare training data. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a functional block diagram showing an example of functions implemented in an information processing system according to an embodiment of the present invention. [Figure 3] FIG. 10 is a diagram illustrating an example of an input image. [Figure 4] FIG. 2 is a diagram illustrating an example of key points of a target object. [Figure 5] FIG. 10 is a diagram schematically illustrating an example of a position image in a target area. [Figure 6] FIG. 10 is a flowchart mainly showing an example of processing by a target region acquisition unit and a posture estimation unit. [Figure 7] FIG. 10 is a diagram illustrating the posture of a detected target object. [Figure 8] FIG. 1 is a flow diagram that illustrates the training of a discriminative model and an estimation model. [Figure 9] FIG. 10 is a flow diagram illustrating an example of a process for generating initial training data. [Figure 10] FIG. 10 is a diagram illustrating photographing of a target object. [Figure 11] FIG. 10 is a flowchart illustrating an example of a process for re-learning an estimation model. DETAILED DESCRIPTION OF THE INVENTION

[0020] An embodiment of the present invention will be described in detail below with reference to the accompanying drawings. In this embodiment, a case will be described in which the invention is applied to an information processing system that receives a photographed image of an object and estimates its posture.

[0021] This information processing system includes a machine learning model that determines whether at least a portion of a captured image includes an object, and outputs information indicating an estimated pose of an object from an image that includes the object. The information processing system is configured to complete the learning process in a short period of time. It is expected that the required time will be, for example, several tens of seconds to grasp and rotate the object and several minutes for the machine learning process.

[0022] FIG. 1 is a diagram showing an example of the configuration of an information processing system according to one embodiment of the present invention. The information processing system according to this embodiment includes an information processing device 10. The information processing device 10 is, for example, a computer such as a game console or a personal computer. As shown in FIG. 1, the information processing device 10 includes, for example, a processor 11, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an imaging unit 20. The information processing system may be configured by one information processing device 10, or may be configured by multiple devices including the information processing device 10.

[0023] The processor 11 is a program-controlled device such as a CPU that operates according to a program installed in the information processing device 10, for example.

[0024] The storage unit 12 is composed of at least a part of a storage element such as a ROM or a RAM, or an external storage device such as a solid state drive. The storage unit 12 stores programs executed by the processor 11, etc.

[0025] The communication unit 14 is a communication interface for wired or wireless communication, such as a network interface card, and transmits and receives data to and from other computers or terminals via a computer network such as the Internet.

[0026] The operation unit 16 is an input device such as a keyboard, a mouse, a touch panel, or a game console controller, and receives operation input from the user and outputs a signal indicating the content of the input to the processor 11.

[0027] The display unit 18 is a display device such as a liquid crystal display, and displays various images according to instructions from the processor 11. The display unit 18 may be a device that outputs a video signal to an external display device.

[0028] The imaging unit 20 is a photographing device such as a digital camera. The imaging unit 20 according to this embodiment is, for example, a camera capable of capturing moving images. The imaging unit 20 may be a camera capable of acquiring a visible RGB image. The imaging unit 20 may be a camera capable of acquiring a visible RGB image and depth information synchronized with the RGB image. The imaging unit 20 may be located outside the information processing device 10, in which case the information processing device 10 and the imaging unit 20 may be connected via the communication unit 14 or an input / output unit described below.

[0029] The information processing device 10 may include audio input / output devices such as a microphone, a speaker, etc. The information processing device 10 may also include, for example, a communication interface such as a network board, an optical disc drive that reads optical discs such as DVD-ROMs and Blu-ray (registered trademark) discs, and an input / output unit (a USB (Universal Serial Bus) port) for inputting and outputting data to and from external devices.

[0030] 2 is a functional block diagram showing an example of functions implemented in an information processing system according to an embodiment of the present invention. As shown in FIG. 2, the information processing system functionally includes a target region acquisition unit 21, a posture estimation unit 25, a captured image acquisition unit 33, a discriminative training data generation unit 34, a discriminative learning unit 35, a shape model acquisition unit 36, an estimated training data generation unit 37, an estimated learning unit 38, and a reliability acquisition unit 39. The target region acquisition unit 21 functionally includes a region extraction unit 22, a feature extraction unit 23, and a discriminative model 24. The posture estimation unit 25 functionally includes an estimation model 26, a keypoint determination unit 27, and a posture calculation unit 28. The discriminative model 24 and the estimation model 26 are both types of machine learning models.

[0031] These functions are implemented mainly by the processor 11 and the storage unit 12. More specifically, these functions may be implemented by having the processor 11 execute a program that is installed in the information processing device 10, which is a computer, and that includes execution instructions corresponding to the above functions. Furthermore, this program may be supplied to the information processing device 10 via a computer-readable information storage medium such as an optical disk, a magnetic disk, or a flash memory, or via the Internet, for example.

[0032] Note that the information processing system according to this embodiment does not necessarily have to be equipped with all of the functions shown in FIG. 2, and may also be equipped with functions other than those shown in FIG.

[0033] The target area acquisition unit 21 acquires an input image captured by the imaging unit 20, and determines whether or not each of one or more candidate areas 56 (see FIG. 3 ) included in the acquired input image includes an image of the target object 51. The area extraction unit 22 extracts the one or more candidate areas 56, and the feature extraction unit 23 extracts features indicating image characteristics from each of the candidate areas 56. The features are input to the identification model 24 as the image of the candidate area 56, and the identification model 24 outputs information indicating whether or not the candidate area 56 includes an image of the target object 51.

[0034] When the candidate region 56 includes the target object 51, the target region acquisition unit 21 acquires a target region 55 including an image of the target object 51, which is extracted from the input image. The target object 51 is an object whose posture is to be estimated in the information processing device 10. The target object 51 is the subject of prior learning.

[0035] Fig. 3 is a diagram showing an example of an input image. In the example of Fig. 3, the target object 51 is a power tool, and in the subsequent figures, unless otherwise specified, the example of the target object 51 will be a power tool. The input image is captured by the image capture unit 20, and the target area 55 is a rectangular area that includes the target object 51 and its vicinity. In the process of acquiring the target area 55, one or more candidate areas 56 that include areas that do not include the target object 51 are also extracted as candidates for areas that include the target object 51.

[0036] The region extraction unit 22 extracts from the input image images of candidate regions 56 that are to be subjected to judgment by the discriminative model 24. More specifically, the region extraction unit 22 uses a known Region Proposal technique to identify one or more candidate regions 56 in which some object is photographed from the input image, and extracts each of the one or more candidate regions 56.

[0037] The discrimination model 24 is a machine learning model that is trained using training data. When input data is input, the trained discrimination model 24 outputs data as a discrimination result. The input data input to the discrimination model 24 is information indicating an image of the candidate region 56, such as features extracted from the image by the feature extraction unit 23. When input data is input, the discrimination model 24 outputs information indicating whether the image of the candidate region 56 includes an image of the target object 51.

[0038] The training data for the discriminative model 24 includes data representing training images including a plurality of positive example images including images of the target object 51, and a plurality of negative example images not including the target object 51. Details of the discriminative model 24 and its training will be described later. Each of the training images may be an image of an area in a captured image where the target object 51 exists. Extraction of this area may be performed using a method similar to that used by the area extraction unit 22. The discriminative model 24 is trained not only using the above training data, but also using additional training data.

[0039] Note that the image of the candidate region 56 may be input directly to the discriminative model 24 without going through the feature extraction unit 23. Although there is a risk of a decrease in accuracy, the region extraction unit 22 may not be present. In this case, the feature extraction unit 23 may extract features from the input image itself, and the discriminative model 24 may determine whether the target object 51 is present in the input image, or the input image may be input directly to the discriminative model 24.

[0040] The posture estimation unit 25 estimates the posture of the target object 51 based on information output when a target region 55 is input to the estimation model 26. The estimation model 26 is a machine learning model that is trained using training data, and when input data is input, the trained estimation model 26 outputs data as an estimation result. The training data includes a plurality of training images rendered using a 3D shape model of the target object 51 and ground truth data that is information regarding the posture of the target object 51 in the training images.

[0041] The trained estimation model 26 receives input of information indicating an image of a target region 55, and outputs information indicating the positions of keypoints for estimating the posture of the target object. The target region 55 is an image based on a candidate region 56 selected based on the output of the discriminative model 24. The training data for the estimation model 26 includes multiple training images rendered using a 3D shape model of the target object 51 and ground truth data indicating the positions of keypoints of the target object 51 in the training images. Keypoints are virtual points within the target object 51 that are used to calculate the posture. The estimation model 26 is trained not only using the training data described above but also using additional training data. The additional training data includes images generated based on the input image, and whether to add training data based on the input image is determined based on the output of the estimation model 26.

[0042] FIG. 4 is a diagram showing an example of key points of a target object 51. The three-dimensional positions of the key points of the target object 51 are determined from a three-dimensional shape model of the target object 51 (more specifically, information on vertices included in the three-dimensional shape model) using, for example, a well-known farthest point algorithm. For ease of explanation, three key points K1 to K3 are shown in FIG. 4, but the actual number of key points may be greater. For example, in this embodiment, the actual number of key points of the target object 51 is eight.

[0043] When a target region 55 is input, the trained estimation model 26 outputs information indicating the two-dimensional positions of the keypoints of the target object 51 in the target region 55. The two-dimensional positions of the keypoints in the input image are determined from the two-dimensional positions of the keypoints in the target region 55 and the position of the target region 55 in the input image. The data indicating the positions of the keypoints may be a position image in which each point indicates the positional relationship (e.g., direction) between that point and the keypoint.

[0044] FIG. 5 is a diagram schematically illustrating an example of a position image in a target region 55. A position image may be generated for each type of key point. The position image indicates the relative direction of each point between that point and the key point. In the position image shown in FIG. 5, a pattern according to the value of each point is depicted, and the value of each point indicates the direction between the coordinates of that point and the coordinates of the key point. FIG. 5 is merely a schematic diagram, and the actual value of each point changes continuously. Although not explicitly shown in the figure, the position image is a Vector Field image that indicates the relative direction of each point between that point and the key point based on that point.

[0045] The keypoint determination unit 27 determines the two-dimensional positions of keypoints in the target region 55 and the input image based on the output of the estimation model 26. More specifically, for example, the keypoint determination unit 27 calculates candidates for the two-dimensional positions of keypoints in the target region 55 based on the position image output from the estimation model 26, and determines the two-dimensional positions of keypoints in the input image from the calculated candidate two-dimensional positions. The keypoint determination unit 27, for example, calculates candidate keypoints from each combination of any two points in the position image, and generates scores indicating whether the candidate points match the directions indicated by each point in the position image. The keypoint determination unit 27 may estimate the candidate point with the highest score as the position of the keypoint. The keypoint determination unit 27 also repeats the above process for each keypoint.

[0046] The orientation calculation unit 28 estimates the orientation of the target object 51 based on information indicating the two-dimensional positions of keypoints in the input image and information indicating the three-dimensional positions of keypoints in a three-dimensional shape model of the target object 51, and outputs orientation data indicating the estimated orientation. The orientation of the target object 51 is estimated using a known algorithm. For example, it may be estimated using a solution to the Perspective-n-Point (PNP) problem for orientation estimation (e.g., EPnP). Furthermore, the orientation calculation unit 28 may estimate not only the orientation of the target object 51 but also the position of the target object 51 in the input image, and the orientation data may include information indicating the position.

[0047] Details of the estimation model 26, the keypoint determination unit 27, and the pose calculation unit 28 may be those described in the paper PVNet: Pixel-Wise Voting Network for 6DoF Pose Estimation.

[0048] The captured image acquisition unit 33, the discriminative training data generation unit 34, the discriminative learning unit 35, the shape model acquisition unit 36, the estimated training data generation unit 37, the estimated learning unit 38, and the reliability acquisition unit 39 constitute a configuration related to the learning of the discriminative model 24 and the estimated model. In this embodiment, the discriminative model 24 and the estimated model 26 are first learned in a short period of time, such as a few seconds or a few minutes, based on a captured image of the target object 51, and after the target region acquisition unit 21 and the pose estimation unit 25 operate based on the learned discriminative model 24 and the estimated model 26, the discriminative model 24 and the estimated model 26 are re-learned.

[0049] The captured image acquisition unit 33 acquires captured images of the target object 51 captured by the imaging unit 20 in order to train the estimation model 26 included in the posture estimation unit 25 and / or the discrimination model 24 included in the target region acquisition unit 21. The imaging unit 20 is assumed to have its internal camera parameters acquired in advance by calibration. These parameters are used when solving the PnP problem.

[0050] The discrimination training data generation unit 34 generates positive example training data based on images including the target object 51 and negative example training data based on images not including the target object 51. The images including the target object 51 may be acquired by the captured image acquisition unit 33.

[0051] The discriminative learning unit 35 trains the discriminative model 24 included in the target region acquisition unit 21 based on the training data generated by the discriminative training data generation unit 34.

[0052] The shape model acquisition unit 36 extracts a plurality of feature vectors indicating local features from each of a plurality of captured images of the target object 51 acquired by the captured image acquisition unit 33, determines the three-dimensional position of the point from which the feature vector was extracted based on the corresponding plurality of feature vectors extracted from the plurality of captured images and the position from which the feature vector was extracted in the captured image, and acquires a three-dimensional shape model of the target object 51 based on the three-dimensional position. This method is a well-known method that is also used in software that realizes so-called SfM and Visual SLAM, and therefore a detailed description will be omitted.

[0053] The estimated training data generation unit 37 generates training data for training the estimated model 26. More specifically, the estimated training data generation unit 37 generates, as initial training data, training data including a rendered training image and ground truth data indicating the positions of key points from a three-dimensional shape model of the target object 51.

[0054] Estimation learning unit 38 trains estimation model 26 included in posture estimation unit 25 using the training data generated by estimation training data generation unit 37.

[0055] The reliability acquisition unit 39 acquires the reliability of the output of the machine learning model for input data based on the output of the machine learning model when the input data is input. Acquiring the reliability based on the output of the machine learning model means, for example, calculating the reliability based on the output of the identification model 24, which is a machine learning model, more specifically, the result of subsequent processing that receives the output, and calculating the reliability based on the position image output by the estimation model 26.

[0056] Next, a process related to posture estimation will be described. Fig. 6 is a flow diagram mainly showing an example of the process of target region acquisition section 21 and posture estimation section 25. The process shown in Fig. 6 may be periodically repeated.

[0057] First, the area extraction unit 22 included in the target area acquisition unit 21 acquires an input image captured by the imaging unit 20 (S101). The area extraction unit 22 may acquire the input image by receiving the input image directly from the imaging unit 20, or may acquire the input image received from the imaging unit 20 and stored in the storage unit 12.

[0058] The region extraction unit 22 extracts one or more candidate regions 56 containing some kind of object from the input image (S102). The region extraction unit 22 may include a pre-trained RPN (Regional Proposal Network). The RPN may be trained using training data that is not related to images in which the target object 51 is captured. This process reduces unnecessary calculations and ensures a certain degree of robustness against the environment.

[0059] Here, the region extraction unit 22 may further perform processing such as background removal (masking) and size adjustment on the image of the extracted candidate region 56. The processed image of the candidate region 56 may then be used for subsequent processing. This processing reduces the domain gap due to the background and lighting conditions, making it possible to train the discriminative model 24 with a small amount of training data.

[0060] The target region acquisition unit 21 determines whether each of the candidate regions 56 includes an image of the target object 51 (S103). This process includes a process in which the feature extraction unit 23 extracts features from the images of the candidate regions 56, and a process in which the discrimination model 24 outputs information indicating whether the candidate region 56 includes the target object 51 from the features.

[0061] The feature extraction unit 23 outputs features corresponding to the image of the candidate region 56 from the image. The feature extraction unit 23 includes a trained convolutional neural network (CNN). In response to an input image, the CNN outputs feature data (input feature data) indicating features corresponding to the image. The feature extraction unit 23 may extract features from the image of the candidate region 56 extracted by an RPN, or may acquire features extracted in RPN processing, such as Faster R-CNN.

[0062] The discriminant model 24 is a type of machine learning model, such as an SVM (Support Vector Machine). In response to input feature amount data indicating features corresponding to the image of the candidate area 56, the discriminant model 24 outputs a discrimination score indicating the probability that the object appearing in the candidate area 56 belongs to a positive class in the discriminant model 24. The discriminant model 24 is trained using a plurality of positive example training data for positive examples and a plurality of negative example training data for negative examples. The positive example training data is generated from training images including images of the target object 51, and the negative example training data is generated from images of an object different from the target object 51 and is prepared in advance. The negative example training data may be generated by capturing an image of the environment of the image capturing unit 20 captured by the image capturing unit 20.

[0063] In this embodiment, this CNN is used to generate feature data indicating feature amounts corresponding to the image on which the normalization process has been performed. Note that the feature extraction unit 23 may output feature data indicating feature amounts corresponding to the image in response to an input image using another known algorithm that calculates feature amounts indicating the features of the image.

[0064] For example, if the identification score is greater than a threshold value, the target region acquisition unit 21 determines that the candidate region 56 includes an image of the target object 51.

[0065] Once it has been determined whether each of the candidate regions 56 includes an image of the target object 51, the target region acquisition unit 21 determines the target region 55 based on the determination result (S104). More specifically, the target region acquisition unit 21 acquires a rectangular region including an area near the target object 51 as the target region 55 based on the candidate region 56 determined to include the target object 51. The target region acquisition unit 21 may acquire a square region including an area near the target object 51 as the target region 55, or may simply acquire the candidate region 56 as the target region 55. Note that the target region acquisition unit 21 does not always need to acquire the target region 55 by the processes of S102 and S103. For example, the target region acquisition unit 21 may acquire the target region 55 by performing a known time-series tracking process on an input image acquired after acquiring the target region 55.

[0066] The posture estimation unit 25 inputs an image of the target region 55 to the trained estimation model 26 (S105). The image of the target region 55 input here may be an image whose size has been adjusted (enlarged or reduced) to match the size of the input image of the estimation model 26. Adjusting the size (normalizing) improves the efficiency of training the estimation model 26. Note that the posture estimation unit 25 may mask the background of the image of the target region 55 and input the image of the target region 55 with the background masked to the estimation model 26.

[0067] The keypoint determination unit 27 included in the posture estimation unit 25 determines the two-dimensional positions of the keypoints in the target region 55 and the input image based on the output of the estimation model 26 (S106). If the output of the estimation model 26 is a position image, the keypoint determination unit 27 calculates candidate positions of the keypoints from each point in the position image and determines the positions of the keypoints based on the candidates. If the output of the estimation model 26 is the position of the keypoint in the target region 55, the position of the keypoint in the input image may be calculated from the position. Note that the processes of S105 and S106 are performed for each type of keypoint.

[0068] The posture calculation unit 28 included in the posture estimation unit 25 calculates an estimated posture of the target object 51 based on the two-dimensional positions of the determined key points (S107). The posture calculation unit 28 may calculate the position of the target object 51 along with the posture. The posture and position may be calculated by solving the PNP problem described above.

[0069] Fig. 7 is a diagram illustrating the posture of the detected target object 51. For ease of explanation, Fig. 7 shows the posture of the target object 51 using local coordinate axes 59 that indicate the local coordinate system of the target object 51. The position of the origin of the local coordinate axes 59 indicates the position of the target object 51, and the direction of the lines of the local coordinate axes 59 indicates the posture.

[0070] Here, the reliability acquisition unit 39 calculates the reliability of the output of the estimation model 26 for the target region 55 (S108). Then, if the reliability satisfies a predetermined condition, the discriminative training data generation unit 34 and the estimation training data generation unit 37 generate additional training data for the discriminative model 24 and the estimation model 26, respectively, based on the target region (S109). The processing of S109 generates additional training data based on data input to the machine learning model after learning of the machine learning model (during inference). The processing of S108 and S109 will be described in detail later.

[0071] The estimated posture and position of the target object 51 may be used in various ways. For example, they may be input into application software for a game or the like instead of operation information input by a controller. The processor 11, which executes the execution code of the application software, may then generate image data based on the posture (and position) and cause the display unit 18 to output the image. The processor 11 may also cause the information processing device 10 or an audio output device connected to the information processing device 10 to output a sound based on the posture (and position). The processor 11 may also control the behavior of an AI agent, such as a robot, by notifying the AI agent of the position and posture of the object, causing the AI agent to, for example, grasp an object.

[0072] Next, an overview of the training of the discriminative model 24 and the estimation model 26 will be described. Fig. 8 is a flow diagram that schematically explains the training of the discriminative model 24 and the estimation model 26.

[0073] First, the discriminative training data generation unit 34 acquires initial training data for the discriminative model 24, and the estimation training data generation unit 37 acquires initial training data for the estimation model 26 (S201).

[0074] The process of step S201 will now be described in more detail. Fig. 9 is a flow diagram showing an example of a process for generating initial training data.

[0075] The photographed image acquisition unit 33 acquires a plurality of photographed images of the target object 51 (S301).

[0076] FIG. 10 is a diagram illustrating capturing an image of a target object 51. The target object 51 is held by, for example, a hand 53, and is captured by the image capturing unit 20. In this embodiment, it is desirable to capture images of the target object 51 from various directions. Therefore, the image capturing unit 20 periodically captures images as in video capture, while changing the image capturing direction of the target object 51. For example, the image capturing direction of the target object 51 may be changed by changing the posture of the target object 51 with the hand 53. Alternatively, the target object 51 may be placed on an AR marker, and the image capturing direction may be changed by moving the image capturing unit 20. The interval at which captured images used in the processing described below are acquired may be wider than the interval at which the video is captured.

[0077] When the photographed images are acquired, the photographed image acquisition unit 33 masks the image of the hand 53 from the photographed images (S302). Masking of the image of the hand 53 may be performed by a known method. For example, the photographed image acquisition unit 33 may mask the image of the hand 53 by detecting a skin-colored area included in the photographed images.

[0078] The shape model acquisition unit 36 then calculates a 3D shape model of the target object 51 and the orientation of each captured image from the multiple captured images (S303). This process may be performed by the aforementioned known method that is also used in software that realizes so-called SfM or Visual SLAM. The shape model acquisition unit 36 may calculate the orientation of the target object 51 based on the calculation logic of the camera's shooting direction according to this method.

[0079] Once the 3D shape model of the target object 51 is calculated, the shape model acquisition unit 36 determines the 3D positions of multiple key points used to estimate the posture of the 3D shape model (S304). The shape model acquisition unit 36 may determine the 3D positions of multiple key points using, for example, a known farthest point algorithm.

[0080] Once the three-dimensional positions of the keypoints have been calculated, the estimated training data generation unit 37 generates training data for the estimation model 26, including a plurality of training images and a plurality of position images (S305). More specifically, the estimated training data generation unit 37 generates a plurality of training images rendered from the three-dimensional shape model, and generates position images indicating the positions of the keypoints in the plurality of training images. The plurality of training images are rendered images of the target object 51 viewed from a plurality of different directions, and a position image is generated for each combination of a training image and a keypoint.

[0081] The estimation training data generator 37 virtually projects the positions of the keypoints onto the rendered training image, and generates a position image based on the relative positions of the projected keypoints and each point in the image. The training data used to learn the estimation model 26 includes the training image and the position image.

[0082] The training images included in the initial training data are rendered images. This is because it is difficult to obtain images taken from various shooting directions in a short time, while using a 3D shape model makes it easy to generate images viewed from various shooting directions. Note that the initial training data may also include real-life training images.

[0083] The discrimination training data generation unit 34 generates positive example training data from the multiple captured images acquired by the captured image acquisition unit 33, more specifically, from images including the target object 51, and acquires negative example training data from images not including the target object stored in the storage unit 12, for example (S306). The positive example training data and negative example training data are training data for the discrimination model 24.

[0084] The discriminative training data generation unit 34 may generate positive example training data from captured images by processing the images input to the discriminative model 24, such as by cutting out a region including the target object 51, normalizing the size, masking the background, and extracting features. The discriminative training data generation unit 34 generates multiple negative example training data by inputting negative example sample images stored in advance in the storage unit 12 to the feature extraction unit 23 and acquiring the output feature data. The features are extracted by the same process as the feature extraction unit 23 included in the discriminative model 24. The negative example sample images may be, for example, images captured in advance by the capture unit 20, images collected from the web, or positive example images of other objects. The negative example training data may be generated in advance and stored in the storage unit 12.

[0085] The identification model 24 is not limited to the one described above, and may be one that determines whether or not the target object 51 exists directly from the image.

[0086] Once the initial training data for the discriminative model 24 and the estimation model 26 are acquired, the discriminative learning unit 35 trains the discriminative model 24 using the initial training data for the discriminative model, and the estimation learning unit 38 trains the estimation model 26 using the initial training data for the discriminative model (S202). The discriminative model 24 may be, for example, an SVM, and the discriminative learning unit 35 may train the SVM using positive example training data and negative example training data.

[0087] Once the identification model 24 and the estimation model 26 have been trained, in steps S203 to S207, the information processing system performs so-called inference processing using these models, while acquiring additional training data (additional training data) for each of the identification model 24 and the estimation model 26 according to their reliability.

[0088] In S203, the information processing system inputs the captured image as an input image to the target area acquisition unit 21, and the target area acquisition unit 21 and the posture estimation unit 25 execute processing to extract the target area 55 and estimate the posture of the target object 51 included in the target area 55. The processing of S203 corresponds to the processing from S101 to S107 in FIG. 6.

[0089] Next, reliability acquisition unit 39 calculates the reliability of the output based on the output of estimation model 26 included in posture estimation unit 25 (S204). This process corresponds to the process of S108 in FIG.

[0090] More specifically, the reliability acquisition unit 39 calculates the reliability, for example, using the following procedure. The reliability acquisition unit 39 selects multiple groups, each containing two points, from the position image output by the estimation model 26. For each group, the reliability acquisition unit 39 calculates a candidate position of the keypoint based on the direction of the keypoint indicated by each point included in the group. A candidate position corresponds to the intersection of a line extending from a certain point in the direction indicated by that point and a line extending from another point in the direction indicated by that point. Once the reliability for each group is calculated, the reliability acquisition unit 39 calculates a value indicating the variability of the candidate positions as the reliability. For example, the reliability acquisition unit 39 may take the average value of the distances from the center of gravity of the candidate positions as the reliability, or may calculate the standard deviation of the candidate positions in any direction as the reliability.

[0091] The reliability obtaining unit 39 may calculate the reliability from a value other than the value indicating the variation of the candidate positions. For example, the reliability obtaining unit 39 may calculate the reliability based on information indicating the difference between the output of the estimation model 26 when an input image, such as an image of the target area, is input to the estimation model 26 and the output when a processed image obtained by processing the input image through a predetermined processing is input to the estimation model 26.

[0092] More specifically, the reliability acquisition unit 39 first performs a predetermined augmentation on the image of the target region. This augmentation may be, for example, a change in brightness or the addition of noise. Next, the reliability acquisition unit 39 inputs the augmented image to the estimation model 26 and acquires the output position image. The reliability acquisition unit 39 then calculates, as the reliability, a value indicating the difference between the position image output for the original image of the target region (original output) and the position image output for the augmented image. This value may be a statistic of the difference in values at each point between the original output and the output for the augmented image, or it may be the distance between the position of a keypoint calculated from the original output and the position of a keypoint calculated from the output for the augmented image. Alternatively, instead of the output for the original image of the target region, the output when an image of the image of the target region that has been processed differently from the predetermined processing is input to the estimation model 26 may be used. Note that the augmentation performed here may be performed using a different method from the processing performed by the estimation training data generation unit 37, which will be described later. The difference in the method reduces the accuracy of the reliability that occurs when calculating the reliability using 26 on an estimation model trained with additional training data.

[0093] The reliability acquisition unit 39 may output a final reliability by combining (an element of) the reliability calculated from the value indicating the variation of the candidate positions with a value indicating the difference between the initial output and the output for the processed image. For example, the reliability acquisition unit 39 may output a value obtained by weighting and adding the former and the latter as the reliability.

[0094] After the reliability is calculated, the reliability obtaining unit 39 determines whether the calculated reliability satisfies an addition condition for adding training data (S205). The addition condition may be, for example, that the value of the variability calculated as the reliability is smaller than a threshold value.

[0095] If the reliability satisfies the addition condition (Y in S205), the discriminative training data generation unit 34 and the estimated training data generation unit 37 generate additional training data to be added to the training data of the discriminative model 24 and the estimated model 26, respectively (S206). S205 and S206 correspond to the processing of S109 in FIG. 6.

[0096] More specifically, the discrimination training data generation unit 34 determines an image corresponding to the target region 55 that was the source of the position image (for example, an image of the corresponding candidate region 56) as a positive example image, and adds the data of the positive example image to the training data of the discrimination model. The discrimination training data generation unit 34 may generate positive example training data from the captured image by processing the image determined as the positive example image in accordance with the image to be input to the discrimination model 24, for example, by extracting features.

[0097] Furthermore, the estimated training data generation unit 37 generates a set of a first additional image and a second additional image based on the image of the target area that is the basis of the position image, and adds the set to the additional training data for the estimated model.

[0098] More specifically, the estimated training data generation unit 37 performs a first augmentation on the image of the target region and acquires the augmented image as a first additional image. The estimated training data generation unit 37 also performs a second augmentation on the image of the target region and acquires the augmented image as a second additional image. The first and second augmentations are different from each other, and may each involve, for example, changing brightness or adding noise. Alternatively, one of the first and second augmentations may not involve substantial augmentation. A method for training an estimated model using a set of the first and second additional images (consistency loss) will be described later.

[0099] The estimated training data generation unit 37 may add to the additional training data a set of an image of the target area that is the source of the position image and ground truth data indicating the orientation of the image calculated by the orientation estimation unit 25. For this, the estimation model 26 may be trained using the same method as in the initial training.

[0100] In this embodiment, a portion of the input data used for inference using a trained machine learning model is added to the training data. On the other hand, the input data used for inference is not usually added to the training data. This is because, for example, if the output for the input data is erroneous, adding the input data may result in a decrease in the quality of the training data. In this embodiment, the reliability of the output of the machine learning model is calculated, and the reliability is used to filter whether or not to add the data to the training data. This ensures the quality of the added data, making it possible to improve the accuracy of the machine learning model while reducing the effort required to generate training data.

[0101] Here, the reliability calculated in this embodiment can be considered as the reliability of the identification model 24 and the estimation model 26. From the perspective of the reliability of the output of the identification model 24, it can be said that the reliability calculated indicates whether the position image output by the estimation model 26, which is located downstream of the identification model 24, is in a state in which keypoints can be accurately determined. In this way, by using the appropriateness of processing including the downstream machine learning model as an indicator of reliability, it is possible to calculate the reliability simply and effectively. Furthermore, from the perspective of the reliability of the output of the estimation model 26, it can be said that the appropriateness of downstream processing for determining keypoints from the output position image is an indicator of reliability.

[0102] When additional training data is acquired, the process from S203 onward is repeatedly executed unless the condition for starting re-learning is met (N in S207). The condition for starting re-learning may be that the number of acquired additional training data reaches a threshold value, or that an operation to end the so-called repeated estimation process is input.

[0103] When the conditions for starting re-learning are met (Y in S207), the discriminative learning unit 35 and the estimation learning unit 38 re-learn the discriminative model 24 and the estimation model 26, respectively (S208).

[0104] Here, relearning refers to training a machine learning model using training data including additional training data. The machine learning models to be learned (the discriminative model 24 and the estimation model 26) may be instances different from the discriminative model 24 and the estimation model 26 that are the machine learning models performing inference, or may be the same instances as the machine learning models performing inference. In the former case, the instances of the discriminative model 24 and the estimation model 26 used for inference may be switched after learning is completed. Alternatively, instead of switching the instances, parameters of the newly trained machine learning model may be copied to the instances of the discriminative model 24 and the estimation model 26 used for inference.

[0105] For the discriminative model 24, the discriminative training unit 35 may add additional training data to the initial training data and train the discriminative model 24 using the training data after the addition. The training data used to train the discriminative model 24 may be all of the initial training data and the additional training data, or may be a portion of them. The portion of the training data used to train the discriminative model 24 may be selected so that the number of samples is equal to or less than the maximum total number of samples, or may be sample data that has been determined to be of low quality by some method.

[0106] On the other hand, the method for retraining the estimation model 26 having additional training data for the first additional image and the second additional image is different. Fig. 11 is a flow diagram showing an example of the retraining process of the estimation model, and in the retraining, the process shown in Fig. 11 is repeatedly executed multiple times.

[0107] First, the estimation learning unit 38 trains the estimation model 26 using initial training data for the estimation model 26 (S501). This training is performed using the same method as the training of the estimation model 26 in step S202. More specifically, the estimation learning unit 38 adjusts the parameters of the estimation model 26 using the difference (L1 loss) between the position image output by the estimation model 26 and the correct data as a teacher signal.

[0108] Next, the estimation learning unit 38 acquires one of the unacquired sets included in the additional training data for the estimation model 26 (S502). The estimation learning unit 38 inputs a first additional image included in that set to the estimation model 26 and acquires the output (first output) of the estimation model 26 (S503). The estimation learning unit 38 also inputs a second additional image included in that set to the estimation model 26 and acquires the output (second output) of the estimation model 26 (S504).

[0109] The estimation learning unit 38 calculates information indicating the difference (consistency loss) between the first output and the second output (S505), and adjusts the parameters of the estimation model 26 based on the information indicating the difference (S506). The information indicating the difference may be a statistic (e.g., an average) of the difference between the values of the first output and the second output at each point.

[0110] Here, since the additional training data is learned according to the difference between the first output and the second output, if this method is used primarily for learning, there is a risk that the parameters of the estimation model 26 will converge to output the same position image regardless of the input. To avoid this situation, it is desirable to keep the ratio of the number of additional training data to the number of all training data including the initial training data within a predetermined value (for example, 20%).

[0111] By using additional training data and retraining the system based on whether two images based on the same image match, it becomes possible to train using training data that does not have a correct label, thereby improving accuracy.

[0112] In this embodiment, the processing of the target area acquisition unit 21 limits the images input to the estimation model 26 to images of the area in the captured image where the target object 51 exists, and to images where there is a sufficiently high probability that the target object 51 exists at the center. Furthermore, the estimation model 26 of the pose estimation unit 25 is trained using training data generated by a 3D shape model, while the identification model 24 of the target area acquisition unit 21 is trained based on captured images of the target object 51.

[0113] By appropriately limiting the images input to the estimation model 26, the accuracy of the output of the estimation model 26 is improved, and the accuracy of the estimated posture of the target object 51 is also improved. Furthermore, by training the identification model 24 based on captured images rather than images based on a 3D shape model, it becomes possible to select the target region 55 more accurately, and ultimately the accuracy of the estimation model 26 can be improved.

[0114] In this embodiment, the captured images used to generate a three-dimensional shape model for training the estimation model 26 of the posture estimation unit 25 are also used when training the identification model 24. This reduces the effort required to photograph the target object 51 and shortens the time required to train the estimation model 26 and the identification model 24.

[0115] The present invention is not limited to the above-described embodiment.

[0116] For example, the discrimination model 24 may be an SVM with an arbitrary kernel. Alternatively, the discrimination model 24 may be a classifier using a method such as a K-nearest neighbor method, logistic regression, or a boosting method such as Adaboost. Alternatively, the discrimination model 24 may be implemented by a neural network, a naive Bayes classifier, a random forest, a decision tree, or the like.

[0117] The output of the estimation model 26 may be a position image such as a heat map showing the positions of keypoints. In this case, for example, the reliability acquisition unit 39 may calculate the number of peaks in the position image output by the estimation model 26 as the reliability. If the number of peaks is smaller than a threshold, the input data may be added to the training data.

[0118] Furthermore, the specific character strings and numerical values described above and the specific character strings and numerical values in the drawings are merely examples, and the present invention is not limited to these character strings and numerical values and may be modified as necessary.

Claims

1. a machine learning model trained by the training data; a trained estimation model that outputs an estimation result based on the output of the machine learning model when input data is input; and a reliability output means for outputting the reliability of the output for the input data based on the estimation result output by the estimation model; generating means for generating new training data based on the input data when the reliability satisfies a predetermined condition; a learning control means for learning a machine learning model using the new training data; An information processing system including:

2. 2. The information processing system according to claim 1, the input data includes a captured image of a target object; The estimation model outputs an image indicating key points for pose estimation of the target object based on an output of the machine learning model; the reliability output means outputs the reliability based on the image. Information processing system.

3. 3. The information processing system according to claim 2, The estimation model outputs an image in which each point indicates a positional relationship with a keypoint; the reliability output means outputs the reliability based on a variance of a plurality of keypoint position candidates, each of which is generated from a different point included in the image output by the estimation model. Information processing system.

4. 4. The information processing system according to claim 1, the reliability output means outputs the reliability based on information indicating a difference between an output of the estimation model when an input image is input to the estimation model and an output when a processed image obtained by processing the input image through a predetermined processing is input to the estimation model. Information processing system.

5. 4. The information processing system according to claim 2, the machine learning model outputs information indicating whether the input data includes the target object; Information processing system.

6. 6. The information processing system according to claim 1, the estimation model is trained using estimation training data; the generating means generates new estimated training data based on the input data when the reliability satisfies the predetermined condition; the learning control means causes a machine learning model to learn using the new training data. Information processing system.

7. 2. The information processing system according to claim 1, the input data includes a captured image of a target object; The machine learning model outputs an image showing key points for pose estimation of the target object based on the input data; the reliability output means outputs the reliability based on the image. Information processing system.

8. 8. The information processing system according to claim 7, the training data includes a plurality of training images rendered from a three-dimensional shape model and ground truth images, each of which is ground truth data for the training images; Information processing system.

9. 9. The information processing system according to claim 1, the generating means generates new training data including a first additional image obtained by processing the input data through a first processing process and a second additional image obtained by processing the input data through a second processing process different from the first processing process; the learning control means causes the machine learning model to learn based on a difference between an output when the first additional image is input to the machine learning model and an output when the second additional image is added to the machine learning model; Information processing system.

10. A computer comprising: a step of outputting a reliability of the output for the input data based on an estimation result output by a trained estimation model based on an output of the machine learning model when input data is input to the machine learning model trained using training data; generating new training data based on the input data if the reliability satisfies a predetermined condition; training a machine learning model with the new training data; An information processing method including:

11. outputting a reliability of the output for the input data based on an estimation result output by a trained estimation model based on an output of the machine learning model when input data is input to the machine learning model trained using training data; generating new training data based on the input data when the reliability satisfies a predetermined condition; training a machine learning model using the new training data; A program that causes a computer to execute a process.

Images