Input image processing method, input image processing device and program
The method enhances 3D human model rendering accuracy by calculating confidence scores and using consistency loss to filter out inaccuracies, addressing issues in crowded scenes and partial visibility.
Patent Information
- Application Number
- JP2023567046
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-05-05
- Filing Date
- 2022-03-23
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-03-23
AI Technical Summary
Existing regression-based 3D human model rendering techniques struggle with accuracy, particularly in crowded scenes or partial visibility, and lack effective methods to filter out inaccurate results.
A method involving confidence score calculation based on projected and direct keypoints, and a training system using consistency loss to enhance the accuracy of 3D human model rendering by integrating 3D and 2D estimation techniques.
Improves the reliability and accuracy of 3D human model rendering by filtering out inaccurate results and enhancing the training process through consistency loss and confidence scoring.
Smart Images

Figure 0007586344000004 
Figure 0007586344000005 
Figure 0007586344000006
Abstract
Description
[Technical field]
[0001] The present invention relates broadly to image processing methods and apparatus, but is not limited thereto. [Background technology]
[0002] Image processing, which renders 3D objects from 2D images, is attracting attention not only in academic research but also in the enterprise market. For example, it can generate 3D human avatars from photos of people for clothing design purposes. This technology is useful in many application fields, such as analyzing sports scenes and suspicious behavior.
[0003] Regression-based 3D person and shape estimation, such as human mesh recovery (HMR), is one method to estimate and render a human body model from an input image (see Non-Patent Document 1). In this method, an image is analyzed to identify the shape of a human body present in the image. 3D coordinates of the vertices and surfaces of the identified human body shape are generated, and the camera field of view and angle in 3D coordinates are determined for the identified human body shape. 2D projected body keypoints (KPTs) can then be computed from these outputs. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] Kanazawa, Angjoo, et al. "End-to-end recovery of human shape and pose. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018 Summary of the Invention [Problem to be solved by the invention]
[0005] In existing techniques for regression-based person model fitting, it is difficult to guarantee the reliability of the results, i.e. the confidence that the results are accurate. In images of crowded places with multiple people, HMR may not be able to identify all the people, but may only identify one human body rendering and generate an output. If a person in the image is only partially visible, the result output may also be inaccurate. However, there is no way to filter the inaccurate results that may occur due to the above scenario.
[0006] Unlike existing 2D human body KPT estimation techniques, in HMR, it is difficult to learn accurate 2D projection KPTs from training data. HMR can be trained by a direct regression combination loss of 2D projection KPTs. However, compared to other 2D human body KPT training techniques such as KPT heatmap learning, this loss cannot take much supervised signal from the training data.
[0007] SUMMARY Disclosed herein are embodiments of an image processing apparatus and method that address one or more of the above problems.
[0008] Furthermore, other desirable features and characteristics will become apparent from the following detailed description and the appended claims, taken in conjunction with the accompanying drawings and the background of this disclosure. [Means for solving the problem]
[0009] In a first aspect, the present disclosure provides a method for processing an input image, comprising: obtaining, by a processor, projected and direct keypoints of features of an input image, the projected keypoints comprising a first set of coordinates of the features projected from a 3D rendering of the input image and the direct keypoints comprising a second set of coordinates of the features based on a 2D rendering of the features; and calculating, by the processor, a confidence score based on the projected and direct keypoints, a higher confidence score indicating greater accuracy of the projected and direct keypoints.
[0010] In a second aspect, the present disclosure provides an apparatus for processing an input image comprising a memory in communication with a processor and storing a recorded computer program executable by the processor, the apparatus executing the computer program to obtain at least projected and direct keypoints of features of an input image, the projected keypoints comprising a first set of coordinates of the features projected from a 3D rendering of the input image and the direct keypoints comprising a second set of coordinates of the features based on a 2D rendering of the features, and to calculate a confidence score based on the projected and direct keypoints, a higher confidence score indicating greater accuracy of the projected and direct keypoints.
[0011] In a third aspect, the present disclosure provides a system for processing an input image comprising the apparatus according to the second aspect and at least one imaging device.
[0012] In a fourth aspect, the present disclosure provides a method for training a model rendering of an input image, comprising: obtaining, by a processor, projected keypoints and direct keypoints for each feature of an input image, the projected keypoints including a first set of coordinates for each feature projected from a 3D rendering of the input image and the direct keypoints including a second set of coordinates for each feature based on a 2D rendering of each feature; calculating, by the processor, a consistency loss value for each feature based on each of the projected keypoints and each of the direct keypoints; calculating, by the processor, a total loss based on the consistency loss value for each feature and ground truth data of the input image; deriving a total loss error based on the total loss; and propagating, by the processor, the total loss error to a model rendering.
[0013] In a fifth aspect, the present disclosure provides an apparatus for training a model rendering of an input image, comprising a memory in communication with a processor and storing a recorded computer program executable by the processor, the apparatus comprising, upon execution of the computer program, at least: obtaining projected keypoints and direct keypoints for each feature of an input image, the projected keypoints including a first set of coordinates for each feature projected from a 3D rendering of the input image and the direct keypoints including a second set of coordinates for each feature based on a 2D rendering of each feature; calculating a consistency loss value based on each of the projected keypoints and each of the direct keypoints; calculating a total loss based on the consistency loss value for each feature and ground truth data of the input image; deriving a total loss error based on the total loss; and propagating the total loss error to a model rendering.
[0014] In a sixth aspect, the present disclosure provides a system for training a model rendering of an input image comprising the apparatus according to the fifth aspect and at least one imaging device. [Brief description of the drawings]
[0015] The accompanying drawings, in which like reference numbers may refer to identical or functionally similar elements throughout the individual views, are incorporated in and form a part of this specification, together with the following detailed description, and are useful for illustrating various embodiments by way of non-limiting examples and for explaining various principles and advantages.
[0016] BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the present invention will be better understood and readily apparent to those skilled in the art from the following written description, given by way of example only, in conjunction with the drawings in which: FIG. [Figure 1] FIG. 1 illustrates a process for generating a 3D mesh from a 2D image according to various embodiments of the present disclosure. [Figure 2A] 1A-1D illustrate respective inaccurate 3D human models that may be generated due to occluded persons or lack of visibility of key points of the human body. [Figure 2B] 1A-1D are diagrams illustrating inaccurate 3D human models that may be generated due to occluded persons and lack of visibility of key points on the human body, respectively. [Diagram 3] FIG. 1 illustrates a keypoint heatmap estimation according to various embodiments of the present disclosure. [Figure 4] FIG. 11 is a flow diagram of confidence score calculation based on visibility and consistency of 2D poses, according to various embodiments of the present disclosure. [Diagram 5] FIG. 11 illustrates how a obtained confidence score may be compared to a threshold, according to various embodiments of the present disclosure. [Figure 6] FIG. 2 is a block diagram for reliability score calculation according to various embodiments of the present disclosure. [Figure 7] 1 is an example flowchart of a reliability score calculation according to various embodiments of the present disclosure. [Figure 8] FIG. 1 illustrates an extended HMR network architecture with consistency loss computation according to various embodiments of the present disclosure. [Figure 9] FIG. 2 is a block diagram for a model training computation according to various embodiments of the present disclosure. [Figure 10] 1 illustrates an example flowchart for training a model rendering of training images according to various embodiments of the present disclosure. [Figure 11] FIG. 1 is a block diagram illustrating a system for processing an input image according to various embodiments of the present disclosure. [Figure 12] FIG. 2 illustrates an exemplary computing device that can be used to perform the methods of the preceding figures. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0017] Explanation of terms Key points (KPTs) refer to points on body parts such as the top of the head, shoulders, elbows, and other similar body parts or joints. Available human body key points also include body parts such as the nose, inner eye parts, outer eye parts, ears, right and left sides of the mouth, wrists, knuckles, left and right hips, knees, ankles, heels, feet, toes, and other similar body parts or joints.
[0018] The 3D pose and shape regressor refers to a module or process that estimates the 3D position of a human mesh (vertices and surfaces) and camera parameters including one or both of the 3D position and angle of the camera to render a 3D human mesh that matches the shape and pose of a 2D human body identified in an input image. The module may be, for example, a trainable neural network model. An example of a 3D person pose and shape estimation process is shown in FIG. 1. In FIG. 1, an input image 102 is processed to obtain keypoints that are representative of the person shown in the image 102. These keypoints are used to estimate a 3D mesh 104 that matches the shape and pose of the person shown in the image 102. A texture overlay may then be applied to the 3D mesh 104 to form a 3D model 106. It can be seen that further texture refinements are possible from the 3D model 106. The generated 3D model 106 (or an improved version thereof) can be used as an avatar for various applications, such as a designer tool for clothing design, surveillance applications, suspicious activity analysis, games, and other similar applications.
[0019] Feature - As commonly used in the art, a feature may be any vector of values that may not be human readable / interpretable. Features may be extracted, for example, from a 2D image. The process of extracting features from an image refers to the process of quantifying features of objects in an image. Keypoints may be generated from the extracted features for person pose estimation and / or human mesh recovery. The extracted features may be used, for example, in a trainable neural network model to improve person pose estimation.
[0020] Error propagation, or back propagation, refers to an algorithm for fitting neural network models. Unlike the naive direct computation of gradients for each weight individually, back propagation computes the gradient of a loss function for a single input-output example with respect to the weights of the network model, making it more efficient to compute the gradient.
[0021] Ground truth data refers to the input and output examples of a machine learning model. In the case of HMR model training, the input and output are usually images and 3D human meshes, respectively. A loss function is usually calculated by the input and output examples. A machine learning model tends to output similar results from the input by minimizing the loss function through error propagation.
[0022] Embodiment When steps and / or features that may have the same numerals are referenced in any one or more of the accompanying drawings, these steps and / or features are intended to be of the same function or operation, unless otherwise indicated for convenience of description.
[0023] It should be noted that the discussion in the "Background" section and the above discussion of prior art configurations relate to a discussion of devices that form part of the public knowledge through their use, and such discussion should not be construed as a representation by the inventors or applicants of the present application that the devices in question in any way form part of the common general knowledge in the art.
[0024] End-to-end HMR is a form of regression-based 3D human pose and shape estimation, based on processing images to generate a human body model that best fits the human body shape identified in the image. The input is an image in which a human body is expected to appear. The output is the 3D coordinates of the vertices and surfaces of the identified human body, and the position and angle of the camera in 3D coordinates relative to the identified human body. From these outputs, 2D projected keypoints (i.e., keypoints defined on a 2D plane, e.g., in XY coordinates) can then be computed. In an example process for end-to-end human shape and pose recovery, input images can be recursively processed to estimate and determine outputs such as the 3D coordinates of the vertices and surfaces of the identified human body, and the position and angle of the camera in 3D coordinates relative to the identified human body in the input image. A 3D human body model may be generated based on these outputs. The keypoints may then be projected from the 3D human body model onto a 2D projection plane to form the 2D projected keypoints.
[0025] However, existing techniques of regression-based human model fitting as described above generally have difficulty outputting accurate human body models, especially in difficult situations where the human body is occluded or there is a lack of visibility, such as in images with multiple people. For example, FIG. 2A shows an inaccurate 3D human body model 202 that may be generated due to the presence of an occluded person captured in an image 200. FIG. 2B shows an inaccurate 3D human body model 206 that may be generated due to the lack of visibility of the human body keypoints of the person captured in an image 204. It is generally difficult to ensure a confidence level that the results obtained from regression-based human model fitting are accurate. Also, there is no way to filter out the inaccurate results due to the lack of visibility of the human body keypoints.
[0026] The accuracy of the projected 2D keypoints from regression-based human model fitting is also an issue. Unlike existing 2D human body keypoint estimation techniques, HMR has difficulty learning accurate 2D projected keypoints from training data. HMR can be trained by a direct regression combination loss of 2D projected KPT. However, compared to other 2D human body KPT training techniques such as KPT heatmap learning, this loss does not take much supervised signal from the training data.
[0027] FIG. 3 shows a flow diagram 300 for KPT heatmap estimation. An input image 302 is processed through a 2D keypoint estimation process 304 to identify a heatmap, e.g., heatmap 306, for each keypoint in the input image 302. A heatmap can be generalized as a map with regions of varying probability, and the location of the keypoint associated with the heatmap can be estimated by the coordinate with the highest probability in the probability map. A visibility or confidence value can also be derived from the heatmap. The visibility or confidence value may be the highest probability value of the map. Currently, there is no way to utilize 2D training methods such as heatmap learning techniques with HMR.
[0028] To address the above problem, FIG. 4 shows a flow diagram 400 of a confidence score calculation based on 2D pose visibility and consistency. The confidence score calculation is based on the premise that visible keypoints tend to have higher confidence values in terms of accuracy, and that if the results are consistent between different methods, the results are accurate and have higher confidence. In the flow diagram 400, projected keypoints are obtained for features of an input image 402. Projected keypoints can be obtained by applying a 3D pose and shape estimation process 404 (e.g., by 3D pose and shape regression techniques as described above or 3D estimation techniques known in the art) to the features of the input image 402 used to generate a 3D rendering 406, and then applying a 3D-to-2D keypoint projection process 408 to the 3D rendering 406 to obtain coordinates of projected keypoints associated with the features. A projected keypoint consists of a set of coordinates of a feature projected from the 3D rendering 406 of the input image 402. Multiple projected keypoints 410 can be obtained from multiple features of the input image 402.
[0029] Further, direct keypoints are obtained based on a 2D rendering of the features of the input image 402. Specifically, a 2D keypoint estimation process 412 can be applied to the features of the input image 402 to obtain direct keypoints (e.g., by using heatmap estimation as shown in FIG. 3 or other 2D estimation techniques known in the art). A direct keypoint is comprised of a set of coordinates of a feature based on the 2D rendering of the input image 402. If the 2D rendering is a heatmap rendering, the heatmap rendering is comprised of one or more sets of coordinates of the feature, each of the one or more sets of coordinates having a probability value, and the set of coordinates of the direct keypoint has the highest probability value among the one or more sets of coordinates. A plurality of direct keypoints 414 can be obtained from a plurality of features of the input image 402.
[0030] Then, at 416, a confidence score is calculated based on the projected and direct keypoints, with a higher confidence score indicating a higher accuracy of the projected and direct keypoints. The confidence score can be calculated, for example, by applying a formula such as:
number
[0031] The above formula is based on the positions or coordinates obtained for the projected and direct keypoints and the visibility value v of the direct keypoint. In the above formula, a consistency score is first calculated based on the projected and direct keypoints, and then the consistency score and the visibility value v are multiplied to calculate the reliability score. The tuning parameter α in the consistency score calculation may be a pre-fixed value that can be manually tuned based on experiments to obtain a more accurate score in this calculation. The visibility value v may be obtained during the 2D keypoint estimation process 412. For example, if heat map estimation is used for the process 412, the visibility value may be the highest probability value of the heat map of the associated features. It goes without saying that other 2D estimation techniques known in the art can be used with corresponding methods to obtain the visibility value. By applying the above formula, a reliability score is obtained that preferably indicates the accuracy of the projected and direct keypoints.
[0032] 5, the resulting confidence score may also be compared to a threshold. For example, after completing confidence score calculation process 502, the confidence score is obtained and compared to a confidence score threshold in confidence score threshold process 504. If the confidence score is lower than the threshold for input image feature 506, the projected and direct keypoints associated with feature 506 may be determined to be inaccurate. If the confidence score is equal to or greater than the threshold for input image feature 508, the projected and direct keypoints associated with feature 508 may be determined to be accurate. This allows inaccurate results to be identified and eliminated.
[0033] FIG. 6 shows a block diagram 600 for the above mentioned confidence score calculation. A feature extractor 602 is used to extract features from the input image. A 3D pose estimator 604 applies 3D estimation techniques known in the art to obtain projected keypoints of the extracted features. A 2D pose estimator 606 applies 2D estimation techniques known in the art to obtain direct keypoints of the extracted features. A consistency score calculator 608 calculates and obtains a consistency score based on the projected and direct keypoints, such as by applying the consistency calculation part of the above equation to process 416. A confidence score calculator 610 then calculates and obtains a confidence score based on the consistency score and the visibility value of the direct keypoints. The visibility value is calculated by the 2D pose estimator 606 and input to the confidence score calculator 610, which applies a formula based on the visibility value and the consistency score. The formula has already been described for process 416, and the consistency score is multiplied by the visibility value to obtain the confidence score.
[0034] FIG. 7 shows an example flow chart 700 for confidence score calculation. The process starts at step 702. At step 704, an image showing a human body is input. At step 706, features are extracted from the input image by a feature extractor. At step 708, a 3D pose and shape regressor estimates 3D pose and camera position from the extracted features. At step 710, 2D projected keypoints are calculated from the estimated 3D pose and camera position. At step 712, a 2D keypoint estimator estimates a 2D keypoint heatmap. At step 714, 2D direct keypoints and associated visibility values are obtained from the keypoint heatmap. At step 716, a consistency score is calculated from the 2D projected and 2D direct keypoints. At step 718, a confidence score is calculated from the consistency score and the visibility value. The process then ends at step 720.
[0035] As shown in Figure 8, the above-mentioned approach of comparing projected keypoints from 3D pose and shape regression and direct keypoints from 2D estimation can be further extended to HMR network architecture by computing consistency loss. In this architecture, one feature extractor is shared between 2D keypoint estimator and 3D pose and shape estimator. The basic premise of this architecture is that results that are consistent with each other despite being obtained from different techniques (i.e., 3D estimation and 2D estimation) are considered to be more accurate. This architecture may be utilized to train a model rendering of input images to favorably improve image processing results to obtain accurate keypoints through deep learning.
[0036] In FIG. 8, an input image 802 undergoes a feature extraction process 804 by a feature extractor to obtain extracted features. The input image 802 is comprised of ground truth data including both 2D and 3D keypoint data. The ground truth data may be used to minimize the total loss when training a model rendering of the image 802. Projected keypoints are obtained for the features of the input image 802. The projected keypoints may be obtained by applying a 3D pose and shape estimation process 806 (e.g., by 3D pose and shape regression techniques as described above or 3D estimation techniques known in the art) to the features of the input image 802 used to generate a 3D rendering 808, and then applying a 3D-to-2D keypoint projection process 810 to the 3D rendering 808 to obtain the coordinates of the projected keypoints associated with the features. The projected keypoints consist of a set of coordinates of the features projected from the 3D rendering 808 of the input image 802. A plurality of projected keypoints 812 may be obtained from a plurality of extracted features of the input image 802. A 3D pose and shape loss computation process 814 is also applied to the 3D renderings to obtain a 3D keypoint loss L3D for each extracted feature, and a 2D projected keypoint loss Lproj is computed for each feature through a 2D projected keypoint loss computation process 816. The 3D keypoint loss corresponds to the error between the estimated 3D keypoint locations of the associated extracted feature and the ground truth 3D keypoint locations, and the 2D projected keypoint loss corresponds to the error between the projected keypoint locations and the ground truth 2D keypoint locations.
[0037] Direct keypoints are also obtained based on a 2D rendering of the extracted features of the input image 802. Specifically, the direct keypoints can be obtained by applying a 2D keypoint estimation process 818 to the extracted features of the input image 802 (e.g., by using heatmap estimation as shown in FIG. 3 or other 2D estimation techniques known in the art). The direct keypoints consist of a set of coordinates of the features based on the 2D rendering of the input image 802. If the 2D rendering is a heatmap rendering, the heatmap rendering consists of one or more sets of coordinates of the features, each of the one or more sets of coordinates having a probability value, and the set of coordinates of the direct keypoint has the highest probability value among the one or more sets of coordinates. A number of direct keypoints 820 can be obtained from the extracted features of the input image 802. A 2D direct keypoint loss calculation process 822 is also applied to the 2D rendering to obtain a 2D direct keypoint loss L2D for each extracted feature. The 2D direct keypoint loss corresponds to the error between the location of the direct keypoint and the location of the ground truth 2D keypoint.
[0038] Then, at 824, a consistency loss value is calculated based on the projected and direct keypoints, with a lower consistency loss value indicating higher accuracy of the projected and direct keypoints. integrity The score Lc can be calculated, for example, by applying the following formula:
number
[0039] The above formula is based on the obtained positions or coordinates of the projected and direct keypoints and the visibility value v of the direct keypoint. The above formula first calculates the consistency loss based on the projected and direct keypoints and the associated visibility value v, and then adds up all the consistency losses obtained for multiple extracted features. The visibility value v may be obtained during the 2D keypoint estimation process 818. For example, if heat map estimation is used for the process 818, the visibility value may be the highest probability value of the heat map of the associated features. It goes without saying that other 2D estimation techniques known in the art can be used with corresponding methods to obtain the visibility value. By applying the above formula, a consistency loss value is obtained that preferably indicates the accuracy of the projected and direct keypoints.
[0040] Furthermore, we define the consistency loss value as the total loss L Total The training process using an architecture such as that shown in Figure 8 can be used to calculate the total loss L Total For example, by applying the following formula, L Total may be obtained.
number
[0041] In the above formula, we apply weights w to the 3D pose and shape loss L3D, the 2D projection keypoint loss Lproj, the 2D direct loss L2D, and the consistency loss Lc, and sum them to obtain the total loss L Total The weights w may be fixed in advance in the model training, or the weight values for the 3D pose and shape loss L3D, the 2D projection keypoint loss Lproj, the 2D direct loss L2D, and the consistency loss Lc may be manually adjusted based on experiments to train and obtain a more accurate model. The total loss L Total Minimizing θ can advantageously improve the overall accuracy of the model rendering of the input image 802.
[0042] FIG. 9 shows a block diagram 900 for such model training computation. A feature extractor 902 may be used to extract features from an input image. A 3D pose estimator 904 applies 3D estimation techniques known in the art to obtain 3D keypoints and 2D projected keypoints of the extracted features. A 3D loss calculator 906 calculates a 3D keypoint loss based on the 3D keypoints and 3D ground truth data of the input image, and calculates a 2D projected keypoint loss based on the 2D projected keypoints and ground truth data of the input image. A 2D pose estimator 908 applies 2D estimation techniques known in the art to obtain direct keypoints of the extracted features. A 2D loss calculator 910 calculates and obtains a 2D direct keypoint loss based on the 2D direct keypoints and 2D ground truth data of the input image. A consistency loss calculator 912 calculates and obtains a consistency loss for each extracted feature based on the 2D projected keypoints, the 2D direct keypoints, and the visibility values of the direct keypoints, such as by applying the above formula in process 824. The visibility values are calculated in the 2D pose estimator 908 and input to the consistency loss calculator 912, where the above formula is applied to obtain a consistency loss. Finally, a total loss calculator 914 calculates and obtains a total loss value, such as by applying the above total loss formula to the inputs of the weighted sum of the 3D keypoint loss, the 2D projected keypoint loss, the 2D direct keypoint loss, and the consistency loss. The weights may be fixed in advance in training the model, or the weight values of the 3D keypoint loss, the 2D projected keypoint loss, the 2D direct keypoint loss, and the consistency loss may be manually adjusted based on experiments to train and obtain a more accurate model.
[0043] FIG. 10 shows an example flowchart 1000 for training a model rendering of a training image. The process starts at step 1002. At step 1004, a training image with ground truth data including 2D and 3D keypoint locations is input. At step 1006, features are extracted from the input image by a feature extractor. At step 1008, a 3D pose and shape regressor estimates 3D pose and camera position from the extracted features and obtains 3D keypoints for the extracted features. At step 1010, 2D projected keypoints are calculated from the estimated 3D pose and camera position. At step 1012, a 2D body keypoint estimator estimates a 2D body keypoint heatmap from the extracted features. At step 1014, visibility values associated with 2D direct keypoints are obtained from the keypoint heatmap. At step 1016, a consistency loss is calculated from the 2D projected keypoints, 2D direct keypoints and associated visibility values for all extracted features. In step 1018, the weighted values of the 3D keypoint loss, the 2D projected keypoint loss, the 2D direct keypoint loss, and the consistency loss are taken as inputs and summed to obtain a total loss. In step 1020, the error of the total loss is propagated to the entire model. Then, the process ends in step 1022.
[0044] FIG. 11 is a block diagram illustrating an input image processing system 1100 according to various embodiments. In one example, management of image input is performed by at least an imaging device 1102 and a device 1104. The system 1100 includes an imaging device 1102 in communication with the device 1104. In an implementation, the device 1104 may be generally described as a physical device including at least one processor 1106 and at least one memory 1108 having computer program code. The at least one memory 1108 and the computer program code, together with the at least one processor 1106, configure the physical device to perform the operations illustrated in one or both of FIG. 7 and FIG. 10. The processor 1106 is configured to receive images from the imaging device 1102 or retrieve images from a database 1110.
[0045] The imaging device 1102 may be a device that can input an image. For example, a digital image may be input, or a physical copy of an image may be input, such as scanning an image and using the scanned image as input. The imaging device 1102 may be configured to receive training images that include ground truth data with 2D and 3D keypoint information. The imaging device may be a camera that can capture an image and use the image as an input image for the device 1104.
[0046] The device 1104 may be configured to communicate with the imaging device 1102 and the database 1110. In one example, the device 1104 may receive an input image from the imaging device 1102 or obtain the input image from the database 1110, and may calculate a confidence score based on the projected and direct keypoints of the extracted features of the input image after processing by the processor 1106 of the device 1104. The higher the confidence score, the higher the accuracy of the projected and direct keypoints. The device 1104 may be configured to calculate a total loss based on the consistency loss value of each extracted feature and the ground truth data of the input image, derive a total loss error based on the total loss, and propagate the total loss error to the model rendering.
[0047] Figure 12 illustrates an exemplary computing device 1200, hereinafter referred to interchangeably as computer system 1200 or device 1200. One or more of the computing devices 1200 described above may be used to implement the methods of system 1100 illustrated in Figure 11 or the previous figures. The following description of computing device 1200 is by way of example only and not by way of limitation.
[0048] As shown in Figure 12, the exemplary computing device 1200 includes a processor 1204 for executing software routines. Although a single processor is shown for clarity, the computing device 1200 may include a multi-processor system. The processor 1204 is connected to a communications infrastructure 1206 for communicating with other components of the computing device 1200. The communications infrastructure 1206 may include, for example, a communications bus, crossbar, or network.
[0049] The computing device 1200 further includes a primary memory 1208, such as a random access memory (RAM), and a secondary memory 1210. The secondary memory 1210 may include a storage drive 1212, which may be, for example, a hard disk drive, a solid state drive, or a hybrid drive, and / or a removable storage drive 1214, which may include, for example, a magnetic tape drive, an optical disk drive, a solid state storage drive (such as a USB flash drive, a flash memory device, a solid state drive, or a memory card), etc. The removable storage drive 1214 reads from and / or writes to the removable storage medium 1218 in a known manner. The removable storage medium 1218 may include a magnetic tape, an optical disk, a non-volatile memory storage medium, etc., and is read from and written to by the removable storage drive 1214. As will be appreciated by those skilled in the relevant art, the removable storage medium 1218 includes a computer readable storage medium having computer executable program code instructions and / or data stored thereon.
[0050] In other implementations, the secondary memory 1210 may additionally or alternatively include other similar means for allowing computer programs or other instructions to be loaded into the computing device 1200. Such means may include, for example, a removable storage unit 1222 and an interface 1220. Examples of removable storage units 1222 and interfaces 1220 may include program cartridges and cartridge interfaces (such as those found in video game console devices), removable memory chips (e.g., EPROM or PROM) and associated sockets, removable solid-state storage devices (e.g., USB flash drives, flash memory devices, solid-state drives, or memory cards), and other removable storage units 1222 and interfaces 1220 by which software and data may be transferred from the removable storage unit 1222 to the computer system 1200.
[0051] The computing device 1200 also includes at least one communication interface 1224. The communication interface 1224 allows software and data to be transferred between the computing device 1200 and external devices via a communication path 1226. In various embodiments of the present invention, the communication interface 1224 allows data to be transferred between the computing device 1200 and a data communication network, such as a public or private data communication network. The communication interface 1224 may be used to exchange data between different computing devices 1200 that form part of an interconnected computer network as described above. Examples of the communication interface 1224 may include a modem, a network interface (such as an Ethernet card), a communication port (such as serial, parallel, printer, GPIB, IEEE 1394, RJ45, USB, etc.), an antenna with associated circuitry, etc. The communication interface 1224 may be wired or wireless. The software and data transferred via the communication interface 1224 is in the form of electronic, electromagnetic, optical, or other signals that can be received by the communication interface 1224. These signals are transmitted over the communication path 1226. 1226 The communication interface is provided via
[0052] As shown in FIG. 12, the computing device 1200 may further include a display interface 1202 that performs operations to render images on an associated display 1230, and an audio interface 1232 that performs operations to play audio content via associated speakers 1234.
[0053] The term "program product" (or computer readable medium, which may be a non-transitory computer readable medium) as used herein may refer, in part, to removable storage medium 1218, removable storage unit 1222, hard disk attached to storage drive 1212, or a carrier wave that carries the software to communication interface 1224 via communication path 1226 (wireless link or cable). Computer readable storage medium (or computer readable medium) refers to any non-transitory, non-volatile, tangible storage medium that provides recorded instructions and / or data to computing device 1200 for execution and / or processing. Examples of such storage media include magnetic tape, CD-ROM, DVD, Blu-ray disk, hard disk drive, ROM or integrated circuit, solid state storage drive (USB flash drive, flash memory device, solid state drive, memory card, etc.), hybrid drive, magneto-optical disk, or computer readable card such as PCMCIA card, whether these devices are internal or external to computing device 1200. Examples of transitory or intangible computer-readable transmission media that may be involved in providing software, application programs, instructions and / or data to computing device 1200 include wireless or infrared transmission channels, as well as network connections to other computers or networked devices, the Internet or intranets, including email transmissions and information stored on websites and the like.
[0054] Computer programs (also referred to as computer program code) are stored in the primary memory 1208 and / or the secondary memory 1210. Computer programs may also be received via communications interface 1224. Such computer programs, when executed, can cause computing device 1200 to implement one or more features of the embodiments described herein. In various embodiments, the computer programs, when executed, can cause processor 1204 to implement the features of the embodiments described above. Thus, such computer programs act as controllers for computer system 1200.
[0055] The software is stored in a computer program product and loaded into the computing device 1200 using the removable storage drive 1214, the storage drive 1212, or the interface 1220. The computer program product may be a non-transitory computer readable medium. Alternatively, the computer program product may be downloaded to the computer system 1200 via the communications path 1226. The software is executed by the processor 1204 to cause the computing device 1200 to perform the functions of the embodiments described herein.
[0056] It should be understood that the embodiment of Figure 12 is merely exemplary. Thus, in some embodiments, one or more features of computing device 1200 may be omitted. Also, in some embodiments, one or more features of computing device 1200 may be combined. Additionally, in some embodiments, one or more features of computing device 1200 may be split into one or more components.
[0057] It will be apparent to those skilled in the art that numerous variations and / or modifications may be made to the present invention as illustrated in the specific embodiments without departing from the spirit or scope of the invention as broadly described. For example, the above description focuses primarily on presenting alerts on a visual interface. However, it will be appreciated that alternative embodiments may use other types of alert presentations, such as audio alerts, to implement similar methods. Several modifications are contemplated and may be incorporated, such as adding access points, modifying login routines, etc. The present embodiments are therefore considered in all respects to be illustrative and not restrictive.
[0058] For example, all or part of the above-described embodiments can be described as an addendum as follows, but is not limited to this.
[0059] (Supplementary Note 1) A method for processing an input image, comprising: a processor obtaining projected and direct keypoints of features of an input image, the projected keypoints including a first set of coordinates of the features projected from a 3D rendering of the input image and the direct keypoints including a second set of coordinates of the features based on a 2D rendering of the features; and the processor calculating a confidence score based on the projected and direct keypoints, a higher confidence score indicating greater accuracy of the projected and direct keypoints.
[0060] (Supplementary Note 2) The method for processing an input image described in Supplementary Note 1, further comprising obtaining visibility values of the direct keypoints, the visibility values being calculated by the 2D rendering, and the calculation of the confidence score further comprising applying a mathematical formula to the visibility values, the projected keypoints and the direct keypoints.
[0061] (Appendix 3) The method of processing an input image described in Appendix 2, wherein the 2D rendering is a heat map rendering, the heat map rendering of the feature having one or more coordinate sets of the feature, each of the one or more coordinate sets having a probability value, and the second coordinate set having the highest probability value among the one or more coordinate sets.
[0062] (Supplementary Note 4) The method for processing an input image according to Supplementary Note 1, further comprising comparing the confidence score with a threshold, and rejecting the projected keypoints and the direct keypoints if the confidence score is lower than the threshold.
[0063] (Supplementary Note 5) A method for training a model rendering of an input image, comprising: a processor obtaining projected keypoints and direct keypoints for each feature of an input image, the projected keypoints including a first set of coordinates for each feature projected from a 3D rendering of the input image and the direct keypoints including a second set of coordinates for each feature based on a 2D rendering of each feature; the processor calculating a consistency loss value for each feature based on each of the projected keypoints and each of the direct keypoints; the processor calculating a total loss based on the consistency loss value for each feature and ground truth data of the input image; deriving a total loss error based on the total loss; and the processor propagating the total loss error to a model rendering.
[0064] (Supplementary Note 6) The method for training a model rendering of an input image described in Supplementary Note 5, further obtaining visibility values of the direct keypoints, the visibility values being calculated by the 2D rendering, and the calculation of the consistency loss further includes applying a mathematical formula to the visibility values, the projected keypoints and the direct keypoints.
[0065] (Appendix 7) The method of training a model rendering of an input image described in Appendix 5, wherein the 2D rendering is a heat map rendering, the heat map rendering of the features includes one or more coordinate sets for each of the features, each of the one or more coordinate sets having a probability value, and the second coordinate set has the highest probability value among the one or more coordinate sets.
[0066] (Supplementary Note 8) The method for training a model rendering of an input image described in Supplementary Note 6, wherein the processor further obtains 3D keypoints for each feature from the 3D rendering of the input image.
[0067] (Supplementary Note 9) The method for training a model rendering of an input image described in Supplementary Note 8, wherein the ground truth data includes ground truth 2D keypoints and ground truth 3D keypoints, and the calculation of the total loss further applies a formula having: a 2D projected keypoint loss corresponding to an error between a position of the projected keypoint and a position of the ground truth 2D keypoint; a 3D keypoint loss corresponding to an error between a position of the 3D keypoint and a position of the ground truth 3D keypoint; and a 2D keypoint loss corresponding to an error between a position of the direct keypoint and a position of the ground truth 2D keypoint.
[0068] (Supplementary Note 10) The method for training a model rendering of an input image described in Supplementary Note 9, wherein the calculation of the total loss further comprises applying a weight to at least one of the 2D projected keypoint loss, the 3D keypoint loss, the 2D keypoint loss, and the consistency loss value.
[0069] (Supplementary Note 11) An apparatus for processing an input image, comprising: a memory in communication with a processor and storing a recorded computer program executable by the processor, wherein execution of the computer program obtains at least projected and direct keypoints of features of an input image, the projected keypoints including a first set of coordinates of the features projected from a 3D rendering of the input image and the direct keypoints including a second set of coordinates of the features based on a 2D rendering of the features, and calculates a confidence score based on the projected and direct keypoints, a higher confidence score indicating greater accuracy of the projected and direct keypoints.
[0070] (Supplementary Note 12) When the memory and the computer program are executed by the processor, the device further obtains visibility values of the direct keypoints, the visibility values being calculated by the 2D rendering, and the calculation of the confidence score further applies a mathematical formula to the visibility values, the projected keypoints and the direct keypoints. The processing device for an input image described in Supplementary Note 11.
[0071] (Supplementary Note 13) The processing device for an input image described in Supplementary Note 12, wherein the 2D rendering is a heat map rendering, and the heat map rendering of the feature includes one or more coordinate sets of the feature, each of the one or more coordinate sets having a probability value, and the second coordinate set has the highest probability value among the one or more coordinate sets.
[0072] (Supplementary Note 14) When the memory and the computer program are executed by the processor, the device further compares the reliability score with a threshold, and if the reliability score is lower than the threshold, the projected keypoints and the direct keypoints are rejected.
[0073] (Supplementary Note 15) An apparatus for training a model rendering of an input image, comprising: a memory in communication with a processor and storing a recorded computer program executable by the processor, wherein execution of the computer program includes obtaining at least projected keypoints and direct keypoints for each feature of an input image, the projected keypoints including a first set of coordinates for each feature projected from a 3D rendering of the input image and the direct keypoints including a second set of coordinates for each feature based on a 2D rendering of each feature, calculating a consistency loss value based on each of the projected keypoints and each of the direct keypoints, calculating a total loss based on the consistency loss value for each feature and ground truth data of the input image, deriving a total loss error based on the total loss, and propagating the total loss error to a model rendering.
[0074] (Supplementary Note 16) When the memory and the computer program are executed by the processor, the apparatus further obtains visibility values of the direct keypoints, the visibility values being calculated by 2D rendering, and the calculation of the consistency loss value further includes applying a mathematical formula to the visibility values, the projected keypoints and the direct keypoints.
[0075] (Supplementary Note 17) The training apparatus for a model rendering of an input image as described in Supplementary Note 16, wherein the 2D rendering is a heat map rendering, and the heat map rendering of the feature includes one or more coordinate sets of the feature, each of the one or more coordinate sets having a probability value, and the second coordinate set has the highest probability value among the one or more coordinate sets.
[0076] (Supplementary Note 18) When the memory and the computer program are executed by the processor, the device further obtains 3D keypoints for each feature from the 3D rendering of the input image. The training device for model rendering of an input image described in Supplementary Note 15.
[0077] (Supplementary Note 19) The training apparatus for model rendering of an input image according to Supplementary Note 18, wherein the ground truth data includes ground truth 2D keypoints and ground truth 3D keypoints, and when the memory and the computer program are executed by the processor, the apparatus further calculates the total loss by applying a formula including: a 2D projected keypoint loss corresponding to an error between a position of the projected keypoint and a position of the ground truth 2D keypoint; a 3D keypoint loss corresponding to an error between a position of the 3D keypoint and a position of the ground truth 3D keypoint; and a 2D keypoint loss corresponding to an error between a position of the direct keypoint and a position of the ground truth 2D keypoint.
[0078] (Supplementary Note 20) When the memory and the computer program are executed by the processor, the apparatus further applies a weight to at least one of the 2D projected keypoint loss, the 3D keypoint loss, the 2D keypoint loss, and the consistency loss value. The training apparatus for model rendering of an input image described in Supplementary Note 19.
[0079] (Supplementary Note 21) An input image processing system comprising the device according to any one of Supplementary Notes 11 to 14 and at least one imaging device.
[0080] (Supplementary Note 22) A training system for model rendering of an input image, comprising the device according to any one of Supplementary Notes 15 to 20 and at least one imaging device.
[0081] While the present invention has been particularly shown and described with reference to exemplary embodiments, the present invention is not limited to these exemplary embodiments, and it will be understood by those skilled in the art that various changes in form and detail can be made therein without departing from the spirit and scope of the invention.
[0082] This application is based on and claims the benefit of priority to Singapore patent application No. 10202104691X, filed on May 5, 2021, the disclosure of which is incorporated herein by reference in its entirety.
Claims
1. obtaining projected and direct keypoints of features of the input image by a processor; the projected keypoints include a first set of coordinates of the feature projected from a 3D rendering of the input image, and the direct keypoints include a second set of coordinates of the feature based on a 2D rendering of the feature; calculating, by the processor, a confidence score based on the projected keypoints and the direct keypoints; A higher confidence score indicates a higher accuracy of the projected keypoints and the direct keypoints. How to process the input image.
2. Further obtaining visibility values of the direct keypoints; the visibility value is calculated by the 2D rendering; The calculation of the confidence score further comprises applying a mathematical formula to the visibility values, the projected keypoints and the direct keypoints. The method of processing an input image according to claim 1.
3. the 2D rendering is a heat map rendering; the heat map rendering of the feature comprises one or more sets of coordinates of the feature, each of the one or more sets of coordinates having a probability value; the second set of coordinates has a highest probability value among the one or more sets of coordinates. The method of processing an input image according to claim 2.
4. comparing the confidence score to a threshold; If the confidence score is lower than the threshold, the projected keypoints and the direct keypoints are rejected. The method of processing an input image according to claim 1.
5. a memory in communication with a processor and storing a recorded computer program executable by said processor; Execution of the computer program performs at least Obtain the projected and direct keypoints of the input image features; the projected keypoints include a first set of coordinates of features projected from a 3D rendering of the input image, and the direct keypoints include a second set of coordinates of the features based on a 2D rendering of the features; Calculate a confidence score based on the projected keypoints and the direct keypoints; A higher confidence score indicates a higher accuracy of the projected keypoints and the direct keypoints. A processing device for input images.
6. The memory and the computer program executed by the processor further include obtaining visibility values of the direct keypoints; the visibility value is calculated by the 2D rendering; The calculation of the confidence score further comprises applying a mathematical formula to the visibility values, the projected keypoints and the direct keypoints. An input image processing device according to claim 5 .
7. the 2D rendering is a heat map rendering; the heat map rendering of the feature includes one or more sets of coordinates of the feature, each of the one or more sets of coordinates having a probability value; the second coordinate set has a highest probability value among the one or more coordinate sets; An input image processing device according to claim 6.
8. The memory and the computer program executed by the processor further include: comparing the reliability score to a threshold; If the confidence score is below the threshold, the projected keypoints and the direct keypoints are rejected. An input image processing device according to claim 5 .
9. Obtaining projected and direct keypoints of features of an input image; calculating a confidence score based on the projected keypoints and the direct keypoints; the projected keypoints include a first set of coordinates of the feature projected from a 3D rendering of the input image, and the direct keypoints include a second set of coordinates of the feature based on a 2D rendering of the feature; A higher confidence score indicates a higher accuracy of the projected keypoints and the direct keypoints. program.
Citation Information
Patent Citations
Facial feature point detecting system and method thereof
JP2007233997A
Facial pose analysis method, apparatus, device, storage medium, and program
JP2020535568A
