Information processing device, information processing method, and program

Abstract

An information processing system for improving the accuracy of attitude recognition using a key point comprises: determining a plurality of key points for recognizing the attitude of an object on the basis of a three-dimensional model of the object; determining one or a plurality of candidate key points for at least some of the plurality of key points; determining, from information outputted by inputting a captured image to a machine learning model which outputs, upon input of a captured image, information that indicates the position of a key point included in a set and information that indicates the position of the candidate key points, said information from the machine learning model indicating the positions of the key point and the candidate key points, the level of confidence regarding each of the key point and the candidate key points; and replacing at least some of the key points included in the set with at least some of the candidate key points, on the basis of the determined level of confidence.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] There is a method for estimating the positions of keypoints of an object from a captured image, and then estimating the pose of the object from the estimated keypoints. In this method, the three-dimensional positions of keypoints in a 3D model of the object are determined in advance, and the pose is estimated by performing a predetermined process using the three-dimensional positions and the estimated positions of keypoints in the image. For example, the farthest point method is known as a method for determining keypoints in a three-dimensional model of an object.

[0003] Sida Peng et al. published a paper titled "PVNet: Pixel-Wise Voting Network for 6DoF Pose Estimation" at the 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). This paper discloses that a machine learning model is trained using training data that includes input images generated from a 3D model and correct output images, and that the positions of key points on the image used for pose estimation are calculated based on the output when a captured image is input to the machine learning model. Summary of the Invention [Problem to be solved by the invention]

[0004] When keypoints are determined from a 3D model using known methods such as the farthest point method, it can be difficult to recognize the position of the keypoints from captured images of the object using a trained machine learning model. For example, if the edge of a 3D model that is inaccurate with the actual object is selected as a keypoint, or if the bottom of a depression is selected as a keypoint, it can be difficult to accurately recognize the edge from the captured image. In such cases, the accuracy of keypoint estimation may decrease, which could lead to a decrease in the accuracy of pose estimation.

[0005] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technique for improving the accuracy of posture recognition using key points. [Means for solving the problem]

[0006] In order to solve the above problem, the information processing device of the present invention includes: a set determination means for determining a set including a plurality of keypoints for recognizing the posture of an object based on a three-dimensional model of the object; a candidate determination means for determining one or more candidate keypoints that are candidates for replacing at least some of the plurality of keypoints included in the set; a reliability determination means for determining a reliability of each of the keypoints included in the set and the candidate keypoints from the information indicating the positions of the keypoints included in the set and the candidate keypoints, which is information that is output by inputting a captured image to a trained machine learning model that receives the captured image and outputs information indicating the positions of the keypoints included in the set and information indicating the positions of the candidate keypoints; and an exchange means for exchanging at least some of the keypoints included in the set with at least some of the candidate keypoints based on the determined reliability.

[0007] In one aspect of the present invention, the machine learning model, to which a captured image is input, may output a plurality of images each showing the positions of the keypoints and candidate keypoints included in the set.

[0008] In one aspect of the present invention, each of a plurality of images output by the machine learning model to which a captured image is input indicates a positional relationship between each point and any of the key points included in the set and the candidate key points, The reliability determination means may determine the reliability of any of the plurality of key points and candidate key points corresponding to any of the plurality of output images, based on the variability of a plurality of candidate position positions determined from mutually different points included in any of the images.

[0009] In one aspect of the present invention, the information processing device further includes a posture determination means for determining the posture of the object from information output by inputting a captured image into the machine learning model, the information indicating the positions of some of the key points included in the set and any of the candidate key points, and the reliability determination means may determine the reliability of the key points and the candidate key points based on the positions of the key points and the candidate key points reprojected based on the determined posture and the positions of the key points and the candidate key points indicated by the output information.

[0010] In one aspect of the present invention, the information processing device further includes a posture determination means for determining the posture of the object from information output by inputting a captured image into the machine learning model, the information indicating the positions of some of the key points included in the set and any of the candidate key points, and the reliability determination means may determine an estimated reliability for each of the key points included in the set and the candidate key points based on the determined posture and ground truth data for the posture of the object in the captured image.

[0011] Further, an information processing method according to the present invention includes the steps of: determining a set including a plurality of keypoints for recognizing the posture of an object based on a three-dimensional model of the object; determining one or more candidate keypoints that are candidates for replacing at least some of the plurality of keypoints included in the set; determining a reliability for each of the keypoints included in the set and the candidate keypoints from the information indicating the positions of the keypoints included in the set and the candidate keypoints, the information being output by inputting a captured image into a trained machine learning model that receives the captured image and outputs information indicating the positions of the keypoints included in the set and information indicating the positions of the candidate keypoints; and replacing at least some of the keypoints included in the set with at least some of the candidate keypoints based on the determined reliability.

[0012] The program of the present invention also causes a computer to execute the following process: determine a set including a plurality of keypoints for recognizing the posture of an object based on a three-dimensional model of the object; determine one or more candidate keypoints that are candidates for replacing at least some of the keypoints included in the set; determine reliability for each of the keypoints included in the set and the candidate keypoints from the information indicating the positions of the keypoints included in the set and the candidate keypoints, which information is output by inputting a captured image to a trained machine learning model that receives the captured image and outputs information indicating the positions of the keypoints included in the set and the candidate keypoints; and replace at least some of the keypoints included in the set with at least some of the candidate keypoints based on the determined reliability. [Effects of the Invention]

[0013] According to the present invention, it is possible to improve the accuracy of posture recognition using keypoints. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a functional block diagram showing an example of functions implemented in an information processing system according to an embodiment of the present invention. [Figure 3] FIG. 2 is a flow chart schematically illustrating processing of the information processing system. [Figure 4] FIG. 10 is a flow diagram showing an example of a process for photographing an object and generating a three-dimensional model. [Figure 5] FIG. 10 is a diagram illustrating photographing an object. [Figure 6] FIG. 10 is a flowchart illustrating an example of a process for detecting a rotation axis. [Figure 7] 10A and 10B are diagrams illustrating an example of an axis of rotation and instructions for additional imaging. [Figure 8] FIG. 10 is a flow diagram illustrating an example of a process for determining keypoints and training an estimation model. [Figure 9] FIG. 1 is a diagram illustrating primary and sub-key points generated from an object. [Figure 10] FIG. 10 is a flow diagram illustrating an example of a process for generating training data and learning an estimation model. [Figure 11] FIG. 10 is a diagram illustrating an example of correct answer data. DETAILED DESCRIPTION OF THE INVENTION

[0015] An embodiment of the present invention will be described in detail below with reference to the accompanying drawings. In this embodiment, a case will be described in which the invention is applied to an information processing system that receives a photographed image of an object and estimates its posture.

[0016] This information processing system includes a machine learning model that outputs information indicating an estimated pose of an object based on a captured image of the object. The information processing system is also configured to complete the learning of the machine learning model in a short period of time. The expected time required is, for example, several tens of seconds to grasp and rotate the object and several minutes for the machine learning process.

[0017] FIG. 1 is a diagram showing an example of the configuration of an information processing system according to one embodiment of the present invention. The information processing system according to this embodiment includes an information processing device 10. The information processing device 10 is, for example, a computer such as a game console or a personal computer. As shown in FIG. 1, the information processing device 10 includes, for example, a processor 11, a storage unit 12, a communication unit 14, an operation unit 16, a display unit 18, and an imaging unit 20. The information processing system may be configured by one information processing device 10, or may be configured by multiple devices including the information processing device 10.

[0018] The processor 11 is a program-controlled device such as a CPU that operates according to a program installed in the information processing device 10, for example.

[0019] The storage unit 12 is composed of at least a part of a storage element such as a ROM or a RAM, or an external storage device such as a solid state drive. The storage unit 12 stores programs executed by the processor 11, etc.

[0020] The communication unit 14 is a communication interface for wired or wireless communication, such as a network interface card, and transmits and receives data to and from other computers or terminals via a computer network such as the Internet.

[0021] The operation unit 16 is an input device such as a keyboard, a mouse, a touch panel, or a game console controller, and receives operation input from the user and outputs a signal indicating the content of the input to the processor 11.

[0022] The display unit 18 is a display device such as a liquid crystal display, and displays various images according to instructions from the processor 11. The display unit 18 may be a device that outputs a video signal to an external display device.

[0023] The imaging unit 20 is a photographing device such as a digital camera. The imaging unit 20 according to this embodiment is, for example, a camera capable of capturing moving images. The imaging unit 20 may be a camera capable of acquiring a visible RGB image. The imaging unit 20 may be a camera capable of acquiring a visible RGB image and depth information synchronized with the RGB image. The imaging unit 20 may be located outside the information processing device 10, in which case the information processing device 10 and the imaging unit 20 may be connected via the communication unit 14 or an input / output unit described below.

[0024] The information processing device 10 may include audio input / output devices such as a microphone, a speaker, etc. The information processing device 10 may also include, for example, a communication interface such as a network board, an optical disc drive that reads optical discs such as DVD-ROMs and Blu-ray (registered trademark) discs, and an input / output unit (a USB (Universal Serial Bus) port) for inputting and outputting data to and from external devices.

[0025] 2 is a functional block diagram showing an example of functions implemented in an information processing system according to an embodiment of the present invention. As shown in FIG. 2, the information processing system functionally includes a posture estimation unit 25, a captured image acquisition unit 31, a shape model acquisition unit 32, a symmetry detection unit 33, and a learning control unit 34. The posture estimation unit 25 functionally includes an estimation model 26, a keypoint determination unit 27, and a posture determination unit 28. The learning control unit 34 functionally includes an initial generation unit 35, a replacement candidate determination unit 36, an estimation learning unit 37, a reliability determination unit 38, and an exchange unit 39. The estimation model 26 is a type of machine learning model.

[0026] These functions are implemented mainly by the processor 11 and the storage unit 12. More specifically, these functions may be implemented by having the processor 11 execute a program that is installed in the information processing device 10, which is a computer, and that includes execution instructions corresponding to the above functions. Furthermore, this program may be supplied to the information processing device 10 via a computer-readable information storage medium such as an optical disk, a magnetic disk, or a flash memory, or via the Internet, for example.

[0027] Note that the information processing system according to this embodiment does not necessarily have to be equipped with all of the functions shown in FIG. 2, and may also be equipped with functions other than those shown in FIG.

[0028] The posture estimation unit 25 estimates the posture of the target object 51 based on information output when an input image is input to the estimation model 26. The input image is an image of the object photographed by the photographing unit 20, and is acquired by the photographed image acquisition unit 31. The estimation model 26 is a machine learning model that is trained using training data, and when input data is input, the trained estimation model 26 outputs data as an estimation result.

[0029] Information about an image of a target object is input to the trained estimation model 26, which outputs information indicating the positions of keypoints for estimating the pose of the object. The estimation model 26 receives the captured image and outputs an image indicating the positions of primary keypoints and subkeypoints included in the set. Primary keypoints and subkeypoints will be described later.

[0030] The training data for the estimation model 26 includes multiple training images rendered using a 3D shape model of the target object, and ground truth data indicating the positions of keypoints of the object in the training images. Keypoints are virtual points within the object and are used to calculate the posture. The data indicating the positions of the keypoints may be a position image in which each point indicates the positional relationship (e.g., relative direction) between that point and the keypoint, or may be a position image that is a heat map in which each point indicates the probability that the keypoint exists. Details of the training of the estimation model 26 will be described later.

[0031] The input image may be an image obtained by processing an image of an object photographed by the photographing unit 20. For example, it may be an image in which an area excluding the target object is masked, or an image in which the size of the object in the image is enlarged or reduced to a predetermined size.

[0032] The keypoint determination unit 27 determines the two-dimensional position of the keypoint in the input image based on the output of the estimation model 26. More specifically, for example, the keypoint determination unit 27 determines candidates for the two-dimensional position of the keypoint in the input image based on the position image output from the estimation model 26. For example, the keypoint determination unit 27 calculates candidate keypoints from each combination of any two points in the position image, and generates scores indicating whether the candidate points match the directions indicated by each point in the position image. The keypoint determination unit 27 may estimate the candidate point with the largest score as the position of the keypoint. The keypoint determination unit 27 also repeats the above process for each keypoint.

[0033] The posture determination unit 28 estimates the posture of the target object 51 based on information indicating the two-dimensional positions of keypoints in the input image and information indicating the three-dimensional positions of keypoints in a three-dimensional shape model of the target object 51, and outputs posture data indicating the estimated posture. The posture of the target object 51 is estimated using a known algorithm. For example, it may be estimated using a solution to the Perspective-n-Point (PNP) problem for posture estimation (e.g., EPnP). Furthermore, the posture determination unit 28 may estimate not only the posture of the target object 51 but also the position of the target object 51 in the input image, and the posture data may include information indicating the position.

[0034] Details of the estimation model 26, the keypoint determination unit 27, and the pose determination unit 28 may be those described in the paper PVNet: Pixel-Wise Voting Network for 6DoF Pose Estimation.

[0035] The captured image acquisition unit 31 acquires a captured image of a target object captured by the image capture unit 20. The image capture unit 20 is assumed to have its internal camera parameters acquired in advance through calibration. These parameters are used when solving the PnP problem.

[0036] The shape model acquisition unit 32 generates and acquires a 3D model of the object from multiple captured images of the object acquired by the captured image acquisition unit 31. More specifically, the shape model acquisition unit 32 extracts multiple feature vectors indicating local features from each of the multiple captured images, and determines the 3D position of the point from which the feature vector was extracted based on the multiple corresponding feature vectors extracted from the multiple captured images and the position from which the feature vector was extracted in the captured image. The shape model acquisition unit 32 then acquires a 3D shape model of the target object 51 based on the 3D position. This method is a well-known method also used in software that realizes so-called SfM and Visual SLAM, so a detailed description will be omitted.

[0037] The symmetry detection unit 33 detects the symmetry of an object from the three-dimensional model. More specifically, the symmetry detection unit 33 detects the mirror symmetry or rotational symmetry of the object from the three-dimensional model.

[0038] The learning control unit 34 determines key points of the target object based on the three-dimensional model and causes the estimation model 26 to learn.

[0039] The initial generation unit 35 generates an initial set of multiple primary keypoints based on the 3D model. The initial generation unit 35 may generate a set of multiple keypoints (primary keypoints) using, for example, a known farthest point algorithm. The initial generation unit 35 also generates multiple alternative keypoints (subkeypoints) that can be candidates for replacing keypoints based on the 3D model. The initial generation unit 35 may generate multiple subkeypoints using, for example, a known farthest point algorithm. In this embodiment, the number N of primary keypoints is 8, but it may be any integer greater than or equal to 4. The number M of subkeypoints is 20 to 50, but it may be any integer greater than the number N of primary keypoints.

[0040] The exchange candidate determination unit 36 ​​determines one or more subkeypoints (exchange candidates) that are candidates for exchanging at least some of the multiple primary keypoints (target keypoints) included in the set. The exchange candidate determination unit 36 ​​may determine N subkeypoints (N is an integer greater than or equal to 1 and less than M) that are close to the target keypoint as exchange candidates from among the multiple subkeypoints. "Close" may mean that the distance between the multiple subkeypoints and the target keypoint is the first to Nth closest. The number of target keypoints may be one or more and less than the number of primary keypoints. Below, an example will be described in which the number of target keypoints in one processing is one.

[0041] The estimation learning unit 37 generates training data used to train the estimation model 26 and trains the estimation model 26 using the training data. The training data includes multiple training images rendered using a 3D shape model of the target object and supervised data indicating the positions of keypoints of the object in the training images. The keypoints for which the estimation learning unit 37 generates supervised data include at least a set of primary keypoints and subkeypoints that are candidates for replacement. The estimation learning unit 37 may generate supervised data for multiple primary keypoints and all subkeypoints included in the initial set.

[0042] More specifically, the estimation learning unit 37 may determine the positions of primary keypoints and subkeypoints in the learning images based on the pose of the rendered object, and generate correct position images for each of the primary keypoints and subkeypoints according to their positions. Note that the training data may include learning images in which the object is photographed, and position images generated from the pose of the object in the learning images estimated by so-called SfM or Visual SLAM.

[0043] The reliability determination unit 38 determines the reliability of each of the primary keypoints and the subkeypoints that are candidates for replacement from information that is output by inputting a captured image into the trained estimation model and that indicates the positions of the primary keypoints and the subkeypoints that are candidates for replacement.

[0044] The exchange unit 39 exchanges the target keypoint with at least some of the subkeypoints that are candidates for exchange based on the reliability. Note that the exchange unit 39 does not need to exchange the target keypoint if the reliability of the target keypoint is higher than that of the subkeypoints. Note that after the exchange by the exchange unit 39, the set of primary keypoints is used for pose estimation based on the output of the estimation model 26. If there are multiple target keypoints, the exchange unit 39 exchanges each target keypoint with one of the subkeypoints that are candidates for exchange according to the reliability.

[0045] The processing of the information processing system will be described below with reference to a flow chart shown in FIG.

[0046] First, the information processing system generates a 3D shape model of a target object based on an image of the object (S101). Then, a learning control unit 34 included in the information processing system determines the 3D positions of keypoints based on the 3D shape model and trains an estimation model 26 for pose estimation (S102). Here, keypoints refer to primary keypoints, and the same applies to S103 to S105.

[0047] Once estimation model 26 has been trained, posture estimation unit 25 inputs an input image of an object into trained estimation model 26 (S103) and acquires data output by estimation model 26. Then, based on the output of estimation model 26, it determines the two-dimensional positions of key points in the image (S104).

[0048] More specifically, if the output of the estimation model 26 is a position image in which each point indicates a relative direction to a keypoint, the keypoint determination unit 27 included in the posture estimation unit 25 calculates candidates for the position of the keypoint from each point in the position image and determines the position of the keypoint based on the candidates.If the output of the estimation model 26 is a position image of a heat map, the keypoint determination unit 27 may determine the position of the point with the highest probability as the position of the keypoint using a known method.

[0049] The posture estimation unit 25 estimates the posture of the object based on the two-dimensional positions of the determined key points and the three-dimensional positions of the key points in the three-dimensional shape model (S105). Although Fig. 3 shows that the processes from S103 to S105 are performed once, in practice, the processes from S103 to S105 may be repeatedly performed until an instruction is received from the user.

[0050] FIG. 4 is a flow diagram showing an example of the process of photographing a target object and generating a three-dimensional model, and describes the process of S101 in more detail.

[0051] First, the photographed image acquisition unit 31 acquires a plurality of photographed images of a target object (S201).

[0052] FIG. 5 is a diagram illustrating capturing an image of a target object. The target object 51 shown in FIG. 5 is held by, for example, a hand 53 and captured by the image capturing unit 20. In this embodiment, it is desirable to capture images of the target object 51 from various directions. Therefore, the image capturing unit 20 periodically captures images, as in video capture, while changing the image capturing direction of the target object 51. For example, the image capturing direction of the target object 51 may be changed by changing the posture of the target object 51 using the hand 53. Alternatively, the target object 51 may be placed on an AR marker, and the image capturing direction may be changed by moving the image capturing unit 20. The interval between captures of captured images used in the processing described below may be longer than the interval between captures of the video. The captured image capture unit 31 may mask the image of the hand 53 from the captured images using a known method (for example, skin color detection).

[0053] Next, the shape model acquisition unit 32 generates a three-dimensional shape model of the object from the acquired multiple photographed images (S202). The details of the method for generating the three-dimensional shape model may be the same as those previously described.

[0054] Once the three-dimensional shape model is generated, the symmetry detection unit 33 detects the symmetry of the object (S203). Here, the symmetry detection unit 33 may detect whether the object is rotationally symmetric and the rotation axis as the symmetry of the object, or may detect whether the object is mirror-symmetric and the symmetry plane as the symmetry of the object.

[0055] Further explanation of detecting symmetry of an object will be given in Figure 6. Figure 6 is a flow diagram showing an example of a process for detecting a rotation axis.

[0056] First, the symmetry detection unit 33 sets a vertically upward axis having the center of the object's model coordinate system as the origin as the first axis (y-axis) (S221). Next, it acquires multiple vertices of the 3D shape model that are in a plane PL perpendicular to the y-axis (S222).

[0057] 7 is a diagram showing an example of the relationship between an object and an axis. The plane PL is, for example, the xz plane passing through the origin. The rotation direction display R will be described later.

[0058] The symmetry detection unit 33 sets multiple different axes that pass through the origin within the plane PL, and generates a score indicating mirror symmetry for each of the multiple axes (S223). The score is the sum of the distances between a point on the 3D shape model rotated 180 degrees about that axis and the vertex closest to that point.

[0059] After the scores are calculated, the symmetry detection unit 33 determines the axis with the smallest score as the second axis (e.g., the x-axis) based on the scores calculated for each of the axes (S225). Note that once the first and second axes are determined, the third axis is automatically determined. The first axis and the second axis may be rotationally symmetric axes.

[0060] The symmetry detection unit 33 determines the axis of rotational symmetry from the first axis and the second axis as the symmetry of the object (S227). The symmetry detection unit 33 may finely divide the coordinates along the axis and determine the axis with the smallest variation in the distance between each of the vertices within the divided range and the origin of the axis as the axis of symmetry. Note that the axis of symmetry detected by the symmetry detection unit 33 is merely a candidate axis of rotational symmetry, and does not need to be strictly rotationally symmetric.

[0061] The symmetry detection unit 33 may determine a mirror symmetry plane instead of a rotational symmetry axis. The symmetry detection unit 33 may also prompt the user to input the symmetry axis.

[0062] When symmetry of the object is detected in S203, the shape model acquisition unit 32 determines whether there are insufficient images in the rotation direction (S205). The determination of the image capture direction may be made based on whether the rotation direction component along the symmetry axis of the difference between the image capture direction determined when creating the 3D model and the image capture direction of the adjacent image is within a threshold. If it is determined that there are sufficient images in the rotation direction (N in S205), the processing of FIG. 4 ends.

[0063] On the other hand, if it is determined that there are insufficient images in the rotation direction (Y in S205), the shape model acquisition unit 32 outputs an instruction to take additional images (S206). The instruction to take additional images may be given by displaying an image including a rendering image of the object and a rotation direction indication R. The captured image acquisition unit 31 also acquires additional captured images and repeats the processes from S202 onwards.

[0064] A three-dimensional shape model of an object is obtained by the process shown in Fig. 4. Furthermore, by the processes of S203 to 207, it becomes possible to obtain a three-dimensional shape model with a certain degree of accuracy even for an object with symmetry.

[0065] 8 is a flow diagram showing an example of the process of determining primary and sub-keypoints and training the estimation model 26. FIG. 8 is a diagram illustrating the process of S102 in FIG. 3 in more detail.

[0066] First, the initial generation unit 35 generates a set of initial primary keypoints and multiple alternate keypoints (subkeypoints) (S301). More specifically, the initial generation unit 35 may generate the 3D positions of the initial keypoints and multiple alternate keypoints from a 3D shape model of the object (more specifically, information on vertices included in the 3D shape model) using, for example, the well-known Farthest Point algorithm.

[0067] Figure 9 is a diagram illustrating primary and sub-keypoints generated from an object. For ease of explanation, the number of primary keypoints K1 to K4 is smaller than the actual number in Figure 9. Also, Figure 3 shows only sub-keypoints S1 to S3 near primary keypoint K4.

[0068] Once the primary and sub-keypoints have been generated, the estimation learning unit 37 generates training data for the estimation model 26 (S302). The training data includes training images rendered based on the 3D shape model and ground truth data indicating the positions of the primary and sub-keypoints in the training images.

[0069] FIG. 10 is a flow diagram showing an example of a process for generating training data. FIG. 10 is a diagram explaining the process of S302 in more detail. First, the estimation learning unit 37 acquires data of a 3D shape model of an object (S321). Then, the estimation learning unit 37 acquires multiple viewpoints for rendering (S322). More precisely, the estimation learning unit 37 acquires multiple camera viewpoints for rendering and shooting directions corresponding to the camera viewpoints. The multiple camera viewpoints may be provided at positions at a constant distance from the origin of the 3D shape model, and the shooting direction is a direction from the camera viewpoint toward the origin of the 3D shape model.

[0070] Furthermore, if a symmetry axis is set as the symmetry, estimation learning unit 37 adds a camera viewpoint in a direction rotated 180 degrees along the symmetry axis. Adding a camera viewpoint in the rotation direction makes it possible to concentrate learning on angles that are prone to errors, and to suppress a decrease in the accuracy of pose estimation caused by similar appearances due to symmetry.

[0071] Once the viewpoints are acquired, the estimation learning unit 37 renders an image of the object for each viewpoint based on the 3D shape model (S325). The images may be rendered using a known method.

[0072] Once the image is rendered, the estimation and learning unit 37 transforms the rendered image using a modulation filter and acquires the transformed image as a training image (S326). The modulation filter intentionally changes the brightness of each pixel in the rendered image to prevent degradation of inference performance due to differences in the colors of the captured image from the actual colors. The estimation and learning unit 37 transforms the rendered image by calculating the product of the element value of each pixel in the rendered image and the value of the corresponding pixel in the modulation filter. The modulation filter is one of several data augmentation techniques for the rendered training image, and the estimation and learning unit 37 may also apply other data augmentation techniques in S326. For example, the estimation and learning unit 37 may perform general data augmentation on the rendered image, such as perturbing at least some of the image's brightness, saturation, and hue, or cropping and resizing a portion of the image to the same size as the original, in addition to the transformation using the modulation filter.

[0073] The modulation filter is generated by the following method. First, the estimation learning unit 37 sets the value of each pixel of an original image with a lower resolution (e.g., 8x8) than the resolution of the rendered image (e.g., 96x96) so that it is a random value between 0.5 and 1.5. The value of each pixel is set so that the average value of the pixel values ​​is 1.0.

[0074] Next, the estimation / learning unit 37 enlarges the size of the original image to the size of the resolution of the rendered image. When enlarging, the estimation / learning unit 37 may determine the value of each pixel by linear interpolation. Once the size is enlarged, the estimation / learning unit 37 further applies a 3x3 Gaussian filter multiple times (e.g., three times) to make the spatial variation of each pixel value more gradual.

[0075] This causes variations in the brightness of the images included in the training data, preventing the estimation model 26 from over-learning about brightness and suppressing a decrease in accuracy when the estimation model 26 processes real-life images. Note that the estimation learning unit 37 may convert only a portion of the rendered image and use that portion as the training image. Converting only a portion can achieve a higher effect. Alternatively, instead of converting the image itself, the texture map of the 3D shape model may be converted.

[0076] After the process of S326 is completed, the estimation learning unit 37 adds the photographed image of the object with the viewpoint to the training image (S327). This photographed image may be the photographed image used to generate the 3D shape model. The camera viewpoint of the photographed image may be the camera viewpoint acquired when generating the 3D shape model.

[0077] Once the training images are prepared, the estimation learning unit 37 generates ground truth data indicating the positions of keypoints in each training image based on the three-dimensional positions of the primary and sub-keypoints and the viewpoint of the training image (S328). The estimation learning unit 37 generates ground truth data for each primary and sub-keypoint for each training image.

[0078] 11 is a diagram showing an example of the correct answer data. The correct answer data is information indicating the two-dimensional positions of key points of an object in a training image, and may be a position image in which each point indicates the positional relationship (e.g., direction) between that point and the key point.

[0079] A position image may be generated for each type of keypoint. The position image indicates the relative direction of each point between that point and a keypoint. In the position image shown in FIG. 11, a pattern according to the value of each point is depicted, and the value of each point indicates the direction between the coordinates of that point and the coordinates of the keypoint. FIG. 11 is merely a schematic diagram, and the actual value of each point changes continuously. Although not explicitly shown in FIG. 11, the position image is a Vector Field image that indicates the relative direction of each point between that point and the keypoint based on that point.

[0080] Through the process shown in FIG. 10, training data including training images and correct answer data is generated.

[0081] Once the training data is generated, the estimation learning unit 37 trains the estimation model 26 of the primary and sub-keypoints using the training data (S303).

[0082] In training the estimation model 26, the estimation learning unit 37 first trains a neural network that outputs primary keypoints from the estimation model 26 using training data on primary keypoints. The neural network may be the one described in the paper PVNet.

[0083] Next, a network for subkeypoints is added and connected to some of the previous layers of the trained neural network, and the parameters of the previous layers are fixed and the neural network is trained using training data for the subkeypoints. By using the parameters learned from the primary keypoints when training the subkeypoints in this way, the time required for training can be shortened.

[0084] Once the estimation model 26 has been trained, the replacement candidate determination unit 36 ​​selects one of the unselected initial primary keypoints as a target keypoint, and selects N subkeypoints in the vicinity of the selected primary keypoint as replacement candidates (S304). Note that the replacement candidate determination unit 36 ​​may select the subkeypoints with the smallest distance from the target keypoint, from 1 to N, as the nearby subkeypoints.

[0085] The reliability determination unit 38 acquires information indicating the positions of the primary key points and replacement candidates, which is output from the estimation model 26 when the captured image for reliability calculation is input to the estimation model 26 (S305). Note that the input of the captured image to the estimation model 26 may be performed in this step or before S304. The captured image for reliability calculation may be a part of the image used when generating the 3D shape model.

[0086] The reliability determination unit 38 calculates the reliability of the positions of the target keypoint and the replacement candidate based on the acquired information (S306). If the acquired information is a Vector Field image for each of the primary keypoint and the replacement candidate, the reliability determination unit 38 may calculate the reliability for each of the target keypoint and the replacement candidate, for example, using the following method.

[0087] The reliability determination unit 38 selects multiple groups, each containing two points, from the Vector Field image output by the estimation model 26. For each group, the reliability determination unit 38 calculates a candidate position of the keypoint based on the direction of the keypoint indicated by each point included in the group. A candidate position corresponds to the intersection of a line extending from a certain point in the direction indicated by that point and a line extending from another point in the direction indicated by that point. After calculating the reliability for each group, the reliability determination unit 38 calculates a value indicating the variability of the candidate positions as the reliability. For example, the reliability determination unit 38 may use the average value of the distances from the center of gravity of the candidate positions as the reliability value, or may calculate the standard deviation of the candidate positions in any direction as the reliability value.

[0088] When the reliability is calculated using the above method, the smaller the reliability value (higher the reliability), the more accurately the keypoint position is estimated. Of course, the reliability may be the average value of the reliability factors calculated for each of the multiple captured images. The captured images may be taken in different directions.

[0089] The reliability may be calculated using other methods. For example, the reliability determination unit 38 may determine the reliability based on the pose of the object estimated by the pose determination unit 28 and its correct pose. More specifically, the reliability determination unit 38 selects one of the target keypoint and the replacement candidate, and estimates the pose of the object using the selected keypoint and the unselected primary keypoints. The reliability determination unit 38 estimates the pose for each of the target keypoint and the replacement candidate using the above method. For each of the target keypoint and the replacement candidate, the reliability determination unit 38 reprojects the positions of the target keypoint and the replacement candidate in the captured image based on the estimated pose and the three-dimensional positions of the target keypoint and the unselected keypoints among the replacement candidate, and stores the reprojected positions in the storage unit 12. Then, for each of the target keypoint and the replacement candidate, the reliability determination unit 38 calculates the average distance between the position estimated by the output of the estimation model 26 and the reprojected position.

[0090] For example, the reliability determination unit 38 may calculate the reliability based on the correct positions of keypoints in the image obtained from the correct pose of the captured image. If the captured image is an image used to generate a 3D shape model, the pose obtained by SLAM technology or the like can be used as the correct answer. In this case, the reliability determination unit 38 calculates the reliability based on the difference between the positions of keypoints obtained by the output of the estimation model 26 and the correct positions of keypoints.

[0091] The exchange unit 39 determines the most reliable keypoint among the target keypoint and the replacement candidates as the new primary keypoint (S307). In other words, if the reliability of any of the replacement candidates is higher than that of the target keypoint, the exchange unit 39 exchanges the target keypoint with the most reliable replacement candidate.

[0092] If an unselected and initial primary key point exists (Y in S308), the process from S304 onwards is repeated. On the other hand, if an unselected and initial primary key point does not exist (N in S308), the process in FIG. 8 ends.

[0093] 8, the exchange unit 39 may remove parts of the neural networks included in the estimation model 26 that are used only for estimating initial primary keypoints and subkeypoints that are not included in the final set of primary keypoints. In other words, the exchange unit 39 may leave only the neural networks related to the primary keypoints used for pose estimation in the estimation model 26, and remove the other neural networks. This can prevent an increase in the amount of calculations required for the estimation model 26 during inference.

[0094] For example, if primary keypoints are determined solely using a method such as the Farthest Point algorithm, the determined locations may not be appropriate for pose estimation. When generating a 3D shape model from a real-life image, the shape of the protruding parts is likely to be inaccurate, while the Farthest Point algorithm tends to select edges as keypoints (see K4 in Figure 9). This results in keypoints being estimated using an estimation model 26 trained on a rendering image that reflects inaccurate edges, which raises concerns about a decrease in keypoint estimation accuracy. Furthermore, even if a complete 3D shape model is used, if a depression is selected as a keypoint, it is likely to be hidden by other parts of the object, making it difficult to accurately estimate the keypoint's position. In this embodiment, the accuracy of pose estimation can be improved by replacing the selected keypoint with a keypoint whose position can be estimated more accurately as needed.

[0095] Furthermore, by replacing the initial primary keypoint with a nearby subkeypoint, the possibility of the primary keypoints being close to each other is reduced, and by replacing the primary keypoints, the accuracy of pose estimation can be improved more reliably while reducing the amount of calculation.

[0096] The present invention is not limited to the above-described embodiment.

[0097] For example, subkeypoints that are not close to the primary keypoint may be used as replacement candidates, although this may result in a decrease in pose estimation accuracy.Also, confidence scores may be calculated for each set of target keypoints and each set of replacement subkeypoints, and the sets may be replaced based on their confidence scores.

[0098] When the output of the estimation model 26 is a position image such as a heat map, the reliability determining unit 38 may determine the number of peaks in the position image output by the estimation model 26 as the reliability.

[0099] Furthermore, the specific character strings and numerical values ​​described above and the specific character strings and numerical values ​​in the drawings are merely examples, and the present invention is not limited to these character strings and numerical values ​​and may be modified as necessary.

Claims

1. a set determination means for determining a set including a plurality of key points for recognizing the pose of an object based on a three-dimensional model of the object; a candidate determiner for determining one or more candidate keypoints that are candidates for replacement with at least some of the plurality of keypoints included in the set; a reliability determination means for determining the reliability of each of the key points included in the set and the candidate key points from the information indicating the positions of the key points included in the set and the candidate key points, the information being output by inputting the captured image into a trained machine learning model that receives the captured image as input and outputs information indicating the positions of the key points included in the set and the candidate key points; a replacement means for replacing at least some of the keypoints included in the set with at least some of the candidate keypoints based on the determined confidence levels; An information processing device comprising:

2. 2. The information processing device according to claim 1, The machine learning model receives the captured image and outputs a plurality of images that respectively indicate the positions of the keypoints included in the set and the candidate keypoints. Information processing device.

3. 3. The information processing device according to claim 2, Each of the plurality of images output by the machine learning model to which the captured image is input indicates a positional relationship between each point and one of the key points included in the set and the candidate key points; the reliability determination means determines the reliability of any of the plurality of key points and candidate key points corresponding to any of the output images, based on the variance of a plurality of position candidates determined from mutually different points included in any of the output images; Information processing device.

4. 3. The information processing device according to claim 1, The machine learning model further includes a posture determination unit for determining a posture of the object from information output by inputting a captured image into the machine learning model, the information indicating the positions of some of the key points included in the set and any of the candidate key points; the reliability determination means determines reliability of the keypoints and the candidate keypoints based on positions of the keypoints and the candidate keypoints reprojected based on the determined pose and positions of the keypoints and the candidate keypoints indicated by the output information. Information processing device.

5. 3. The information processing device according to claim 1, The machine learning model further includes a posture determination unit for determining a posture of the object from information output by inputting a captured image into the machine learning model, the information indicating the positions of some of the key points included in the set and any of the candidate key points; the reliability determination means determines an estimated reliability for each of the key points included in the set and the candidate key points based on the determined pose and ground truth data for the pose of the object in the captured image; Information processing device.

6. A computer comprising: determining a set of key points for recognizing the pose of the object based on a three-dimensional model of the object; determining one or more candidate keypoints that are candidates for replacing at least some of the keypoints in the set; A step of inputting a captured image into a trained machine learning model that receives the captured image and outputs information indicating the positions of keypoints included in the set and information indicating the positions of the candidate keypoints, the model outputting the captured image and information indicating the positions of the keypoints included in the set and the candidate keypoints, the model determining the reliability of each of the keypoints included in the set and the candidate keypoints from the information indicating the positions of the keypoints included in the set and the candidate keypoints; replacing at least some of the keypoints in the set with at least some of the candidate keypoints based on the determined confidence levels; An information processing method including:

7. determining a set of key points for recognizing a pose of the object based on a three-dimensional model of the object; determining one or more candidate keypoints that are candidates for replacement with at least some of the keypoints in the set; The captured image is input to a trained machine learning model that receives the captured image and outputs information indicating the positions of keypoints included in the set and information indicating the positions of the candidate keypoints, and the model outputs information indicating the positions of the keypoints included in the set and the candidate keypoints. The model determines the reliability of each of the keypoints included in the set and the candidate keypoints from the information indicating the positions of the keypoints included in the set and the candidate keypoints. replacing at least some of the keypoints included in the set with at least some of the candidate keypoints based on the determined confidence levels; A program that causes a computer to execute a process.

Images