Key point prediction model training method, three-dimensional key point prediction method and related equipment

Abstract

The present application relates to the field of computer vision technology, and discloses a three-dimensional key point prediction method, a training method and related equipment. The training method of the key point prediction model comprises: obtaining a sample image pair, the sample image pair comprising a first sample image and a second sample image; constructing a three-dimensional human body according to the first sample image to obtain a three-dimensional human body model; sampling key points in the three-dimensional human body model to obtain the three-dimensional coordinates of the key points, the three-dimensional coordinates of the key points being used to determine the posture presented by the human body pixel area; predicting the key points according to the second sample image by the key point prediction model to obtain the predicted three-dimensional coordinates of the key points; calculating a first prediction loss according to the three-dimensional coordinates of the key points and the predicted three-dimensional coordinates of the key points; reversely training the key point prediction model according to the first prediction loss; the key point prediction model of this scheme can automatically complete the key points that are blocked in the image.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a training method for a key point prediction model, a three-dimensional key point prediction method, and related equipment. Background Art

[0002] Human posture recognition is an important application scenario in the field of computer vision technology. Human posture recognition generally involves first determining multiple key points that represent human posture based on an image, and then performing posture recognition based on the determined multiple key points.

[0003] In actual application scenarios, there may be situations where key points in the image are occluded. In this case, the occluded key points cannot be determined, resulting in incomplete key points representing the posture, which may cause inaccurate posture recognition. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application propose a key point prediction model training method, a three-dimensional key point prediction method and related equipment to improve the above problems.

[0005] According to one aspect of an embodiment of the present application, a training method for a key point prediction model is provided, comprising: obtaining a sample image pair, the sample image pair comprising a first sample image and a second sample image, the second sample image being obtained after partial occlusion processing or cropping processing is performed on a human body pixel area in the first sample image; constructing a three-dimensional human body according to the first sample image to obtain a three-dimensional human body model; sampling key points in the three-dimensional human body model to obtain three-dimensional coordinates of the key points, the three-dimensional coordinates of the key points being used to determine the posture presented by the human body pixel area; predicting key points according to the second sample image by a key point prediction model to obtain predicted three-dimensional coordinates of the key points; the key point prediction model is used to predict the three-dimensional coordinates of key points having a posture recognition function in the input image according to an input image; calculating a first prediction loss according to the three-dimensional coordinates of the key points and the predicted three-dimensional coordinates of the key points; and reversely training the key point prediction model according to the first prediction loss.

[0006] According to one aspect of an embodiment of the present application, a three-dimensional key point prediction method is provided, including: acquiring a target image, the target image including a human body pixel area; performing three-dimensional key point prediction based on the target image by a key point prediction model to obtain target three-dimensional key point information, wherein the key point prediction model is trained according to the training method of the key point prediction model as described above.

[0007] According to one aspect of an embodiment of the present application, a training device for a key point prediction model is provided, comprising: a sample image pair acquisition module, used to acquire a sample image pair, the sample image pair comprising a first sample image and a second sample image, the second sample image being obtained after partial occlusion processing or cropping processing is performed on a human body pixel area in the first sample image; a first three-dimensional human body construction module, used to construct a three-dimensional human body according to the first sample image to obtain a three-dimensional human body model; a first sampling module, used to sample key points in the three-dimensional human body model to obtain three-dimensional coordinates of the key points, the three-dimensional coordinates of the key points being used to determine the posture presented by the human body pixel area; a first prediction module, used to predict key points according to the second sample image by a key point prediction model to obtain predicted three-dimensional coordinates of the key points; the key point prediction model is used to predict the three-dimensional coordinates of key points in the input image that have a posture recognition function according to an input image; a first prediction loss calculation module, used to calculate a first prediction loss according to the three-dimensional coordinates of the key points and the predicted three-dimensional coordinates of the key points; a first reverse training module, used to reversely train the key point prediction model according to the first prediction loss.

[0008] In some embodiments of the present application, the three-dimensional human body model is constructed by a multi-person linear skinning SMPL model; the training device of the key point prediction model also includes: a second acquisition module, used to acquire a third sample image and annotation information of the third sample image; a model construction module, used to construct a three-dimensional human body model according to the third sample image by the SMPL model to obtain a predicted three-dimensional human body model; a second prediction loss calculation module, used to calculate a second prediction loss according to the predicted three-dimensional human body model and the annotation information of the third sample image; a second reverse training module, used to reverse train the SMPL model according to the second prediction loss.

[0009] In some embodiments of the present application, the annotation information includes the annotated two-dimensional coordinates of the annotated key points and the annotated three-dimensional coordinates of the annotated key points; the second prediction loss calculation module includes: a sampling unit, which is used to sample the annotated key points in the predicted three-dimensional human body model to obtain the predicted three-dimensional coordinates corresponding to the annotated key points; a predicted two-dimensional coordinate determination unit, which is used to project the predicted three-dimensional coordinates corresponding to the annotated key points to the image plane where the third sample image is located according to the internal parameter matrix of the target image acquisition device, to obtain the predicted two-dimensional coordinates of the annotated key points on the image plane; the target image acquisition device refers to an image acquisition device that acquires the third sample image; a three-dimensional prediction loss calculation unit, which is used to calculate the three-dimensional prediction loss according to the annotated three-dimensional coordinates of the annotated key points and the predicted three-dimensional coordinates of the annotated key points; a two-dimensional prediction loss calculation unit, which is used to calculate the two-dimensional prediction loss according to the predicted two-dimensional coordinates of the annotated key points and the annotated two-dimensional coordinates of the annotated key points; a second prediction loss calculation unit, which is used to calculate the second prediction loss according to the three-dimensional prediction loss and the two-dimensional prediction loss.

[0010] In some embodiments of the present application, the annotation information includes the annotated two-dimensional coordinates of the annotated key points and the annotated three-dimensional coordinates of the annotated key points; the training device of the key point prediction model also includes: a depth image acquisition module, used to acquire the depth image corresponding to the third sample image; a depth value acquisition module, used to acquire the depth value of the annotated key points according to the depth image; a three-dimensional coordinate determination module, used to determine the annotated three-dimensional coordinates of the annotated key points according to the depth value of the annotated key points, the internal parameter matrix of the target image acquisition device and the two-dimensional coordinates of the annotated key points; the target image acquisition device refers to an image acquisition device that acquires the third sample image.

[0011] In some embodiments of the present application, a three-dimensional coordinate determination module includes: a target transformation matrix determination unit, which is used to determine the target transformation matrix corresponding to the labeled key point based on the intrinsic parameter matrix of the target image acquisition device and the depth value of the labeled key point; a labeled three-dimensional coordinate determination unit, which is used to transform the two-dimensional coordinates of the labeled key point into a three-dimensional space according to the target transformation matrix to obtain the labeled three-dimensional coordinates of the labeled key point.

[0012] In some embodiments of the present application, the key point prediction model includes a feature extraction network and three branch networks; wherein, one branch network is used to predict the coordinates of the key point on a coordinate axis; the predicted three-dimensional coordinates include the predicted coordinates of the key point on three coordinate axes; the first prediction module includes: a feature extraction unit, used to extract features of the second sample image through the feature extraction network to obtain a feature map of the second sample image; an input unit, used to input the feature map into the three branch networks respectively; a coordinate output unit, used to perform coordinate prediction by each branch network according to the feature map to obtain the coordinates of the predicted key point on the coordinate axis corresponding to each branch network.

[0013] In some embodiments of the present application, the branch network includes a mean processing layer, an attention-based residual network and a fully connected layer; in this embodiment, the coordinate output unit includes: a mean processing unit, which is used to take the average of the feature map along the direction of the coordinate axis corresponding to the branch network by the mean processing layer to obtain a first intermediate feature map; a convolution processing unit, which is used to perform convolution processing on the first intermediate feature map by the attention-based residual network to obtain a second intermediate feature map; a fully connected output unit, which is used to output the coordinates on the coordinate axis corresponding to the branch network by the fully connected layer according to the second intermediate feature map.

[0014] According to one aspect of an embodiment of the present application, a three-dimensional key point prediction device is provided, including: a target image acquisition module, used to acquire a target image, the target image including a human body pixel area; a second prediction module, used to perform three-dimensional key point prediction based on the target image by a key point prediction model to obtain target three-dimensional key point information, wherein the key point prediction model is trained according to the training method of the key point prediction model as described above.

[0015] In some embodiments of the present application, the three-dimensional key point prediction device further includes: a posture determination module, which is used to determine the posture presented by the human body pixel area in the target image according to the target three-dimensional key point information.

[0016] In some embodiments of the present application, the three-dimensional key point prediction device also includes: a virtual object control module, which is used to control the virtual object to simulate the posture presented by the human body pixel area in the target image according to the target three-dimensional key point information.

[0017] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the training method of the key point prediction model as described above, or the three-dimensional key point prediction method as described above is implemented.

[0018] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the training method of the key point prediction model as described above, or the three-dimensional key point prediction method as described above is implemented.

[0019] According to one aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the training method of the key point prediction model, or the three-dimensional key point prediction method as described above.

[0020] In the scheme of the present application, the human body pixel area in the first sample image is occluded or cropped to obtain a second sample image, so that compared with the key points included in the portrait pixel area of ​​the first sample image, the key points included in the human body pixel area of ​​the second sample image are not complete; then the second sample image and the three-dimensional coordinates of the key points sampled in the three-dimensional human body model constructed by the first sample image are used to train the key point prediction model, so that the key point prediction model can learn the ability to automatically complete the occluded key points in the image, so that even if the image input to the key point prediction model is an image in which multiple key points expressing posture are partially occluded, the key point prediction model can also accurately output the three-dimensional coordinates of the occluded key points, thereby ensuring the accuracy of the posture identified based on the output three-dimensional coordinates. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a schematic diagram of an application scenario shown according to an embodiment of the present application.

[0023] Figure 2 It is a flowchart of a method for training a key point prediction model according to an embodiment of the present application.

[0024] Figure 3 1 is a schematic diagram of the position of each key point in a key point set in an SMPL model according to an embodiment of the present application.

[0025] Figure 4 It is a flowchart of the steps before step 220 according to an embodiment of the present application.

[0026] Figure 5 A shows a schematic diagram of three-dimensional key points output for an input target image according to a key point prediction model in the related art.

[0027] Figure 5 B shows a schematic diagram showing the three-dimensional key points output by the trained key point prediction model for the input target image according to an embodiment of the present application.

[0028] Figure 6 It is a flowchart of the steps before step 120 according to an embodiment of the present application.

[0029] Figure 7 It is a flow chart of determining the three-dimensional coordinates of the annotation key points according to an embodiment of the present application.

[0030] Figure 8 It is a flowchart of step 630 according to an embodiment of the present application.

[0031] Fig. 9 It is a schematic diagram of training the SMPL model according to an embodiment of the present application.

[0032] Fig.10 is a flowchart of step 240 according to an embodiment of the present application.

[0033] Fig.11 It is a schematic diagram of predicting the three-dimensional coordinates of key points using a key point prediction model according to an embodiment of the present application.

[0034] Fig.12 It is a flowchart of step 1030 according to an embodiment of the present application.

[0035] Fig.13 It is a schematic diagram of the branch network predicting the coordinates in the x direction according to an embodiment of the present application.

[0036] Fig.14 It is a flowchart of a three-dimensional key point prediction method according to an embodiment of the present application.

[0037] Fig.15 It is a flowchart of performing three-dimensional key point prediction according to a specific implementation of the present application.

[0038] Fig.16 A-16C is a schematic diagram showing controlling a virtual object based on a real user's gesture according to an embodiment of the present application.

[0039] Fig.16 D shows the SMPL model based on Fig.16 Schematic diagram of a three-dimensional human body model constructed from the target image shown in A.

[0040] Fig.17 It is a block diagram of a training device for a key point prediction model according to an embodiment of the present application.

[0041] Fig.18 It is a block diagram of a three-dimensional key point prediction device according to an embodiment of the present application.

[0042] Fig.19 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0044] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0045] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0046] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0047] It should be noted that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0048] Before going into detail, the terms used in this application are explained as follows:

[0049] SMPL (Skinned Multi-Person Linear) model: a parameterized human skin model that generates human skin through the rotation of 23 joints and 10 shape parameters. The mesh of the SMPL model consists of N = 6890 vertices and K = 23 joints. The human body can be understood as the sum of a basic model and deformations based on the model. PCA (Principal Component Analysis) is performed on the deformation to obtain low-dimensional parameters that describe the shape, namely shape parameters; at the same time, a motion tree is used to represent the posture of the human body, that is, the rotation relationship between each joint and the parent node of the motion tree. This relationship can be expressed as a three-dimensional vector. Finally, the local rotation vector of each joint constitutes the pose parameter of the SMPL model. The posture training data of SMPL covers a variety of body shapes, so that the position of the joint can be well predicted through it.

[0050] Depth image: Each pixel on it records the distance from the viewpoint to the object, represented by a grayscale image. The closer the distance from the viewpoint to the object, the darker it is in the depth image.

[0051] Kinect: A 3D motion-sensing camera that can be used to capture depth images.

[0052] Key points: Also known as human key points, they refer to the points on the human body that are representative of posture recognition. In practical applications, key points can be joints of the human body, such as head key points, neck key points, shoulder key points, elbow key points, knee key points, hip key points, etc.

[0053] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0054] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present application, such as Figure 1 As shown, the application scenario includes a terminal 110 and a server 120 that is connected to the terminal 110 through a network. The network can be a wide area network or a local area network, or a combination of the two.

[0055] The terminal 110 may send a first sample image including a human pixel region to the server 120. The terminal 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a vehicle-mounted terminal, or other device.

[0056] The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0057] After receiving the first sample image, the server 120 may perform occlusion processing or cropping processing on the human body pixel area in the first sample image to obtain a second sample image, and associate the first sample image with the second sample image to obtain a sample image pair, and store the sample image pair in the first training database 121 to achieve the construction of the first sample training database 121. The server 120 and the first training database 121 may be set separately or integrated together.

[0058] In other embodiments, the terminal 110 may first perform partial occlusion processing or cropping processing on the first sample image to obtain a second sample image, and then associate the first sample image with the second sample image and send it to the server 120, so that the server 120 stores the sample image pair including the first sample image and the second sample image in the first training database 121.

[0059] The server 120 may also deploy an SMPL model and a key point prediction model to be trained. The server 120 may use the sample image pairs in the first sample training database 121 to train the key point prediction model according to the scheme provided in the embodiment of the present application.

[0060] After the key point prediction model is completed, the key point prediction model can perform three-dimensional key point prediction based on the image including the human body pixel area and output the three-dimensional coordinates of the key points. Moreover, since the key point prediction model is trained in combination with the second sample image obtained by occluding or cropping the first sample image, the key point prediction model can learn the ability to complete the key points that are occluded, so that even if the key points presented in the input image are incomplete, the key point prediction model can predict the three-dimensional coordinates of all key points in the posture represented by the image based on the image.

[0061] The terminal 110 may also send a target image to be predicted for key points to the server 120, the target image including a human pixel area. The server 120 may output target three-dimensional key point information according to the target image based on the trained key point prediction model, and the target three-dimensional key point information indicates the three-dimensional coordinates of the predicted key points. In the display interface of the terminal 110, the position of the key points may be determined in a preset three-dimensional coordinate system according to the predicted three-dimensional coordinates and the key points may be displayed. Thus, the user may determine the posture presented by the human pixel area in the target image according to the position of the key points in the three-dimensional coordinate system.

[0062] In this solution, the server 120 uses computer vision in artificial intelligence technology to predict the three-dimensional coordinates of key points based on two-dimensional images, thereby facilitating posture recognition.

[0063] Computer Vision (CV) is a science that studies how to make machines "see". It uses cameras and computers to identify and measure targets instead of human eyes, and further processes images so that the computer processes them into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, video processing, behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, autonomous driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0064] Figure 2 is a flowchart of a method for training a key point prediction model according to an embodiment of the present application. The method can be executed by an electronic device with processing capabilities, and the electronic device can be Figure 1 The server or terminal shown, that is, the method can be executed by the server or the terminal, or by the server and the terminal together, which is not specifically limited here. Figure 2 As shown, the method at least includes steps 210 to 260, which are described in detail as follows:

[0065] Step 210: Acquire a sample image pair, the sample image pair comprising a first sample image and a second sample image, the second sample image being obtained by partially blocking or cropping a human body pixel region in the first sample image.

[0066] The first sample image includes a human body pixel region, wherein the human body pixel region in the first sample image may present a part of a human body (e.g., the upper body of a human body) or a whole body of a human body, which is not specifically limited here. The first sample image and the second sample image may be color RGB images.

[0067] By partially occluding the human body pixel area in the first sample image, or cropping the human body pixel area in the first sample image, the key points in the human body pixel area in the first sample image do not completely exist in the human body pixel area in the second sample image, that is, the key points included in the human body pixel area in the second sample image are only part of the key points in the human body pixel area in the first sample image.

[0068] In some embodiments, the first sample image can be randomly cropped or partially occluded, so that multiple second sample images can be obtained based on the same first sample image, wherein the partially occluded area or the cropped area for the same first sample image can be set according to actual needs.

[0069] In some embodiments, a shadow can be added to a partial area (referred to as a first sub-area for ease of description) in the human body pixel area of ​​the first sample image, that is, the second sample image is obtained by covering the first sub-area in the first sample image with the shadow, wherein the pixel value of the added shadow is different from the pixel value of the first sub-area in the first sample image.

[0070] For example, if the first sample image includes a pixel area of ​​the upper body of a human body, if the first sub-area to be blocked is the area where the wrist is located, the area where the wrist is located is covered by a shadow, thereby obtaining a second sample image. Of course, in the partial blocking process of the human body pixel area in the first sample image, the area to be blocked can be set according to actual needs and is not specifically limited here.

[0071] In some embodiments, part of the pixel area in the human body pixel area in the first sample image may be cropped to obtain a second sample image. For example, if the first sample image includes a human body upper half pixel area, if the second sub-area to be cropped is the pixel area where the human body's elbow is located, the pixel area where the human body's elbow is located is cropped, and the other areas in the first sample image except the pixel area where the human body's elbow is located are retained, and the second sample image is obtained. Similarly, the second sub-area to be cropped can be set according to actual needs.

[0072] Step 220: construct a three-dimensional human body according to the first sample image to obtain a three-dimensional human body model.

[0073] In some embodiments, a three-dimensional human body can be constructed by a three-dimensional construction model. The three-dimensional construction model can be constructed by a convolutional neural network or a fully connected neural network, which is not specifically limited here.

[0074] In some embodiments, the three-dimensional human body model can be a multi-person linear skin SMPL model. As described above, the SMPL model includes 6890 vertices, and the three-dimensional human body model established by the SMPL model according to the first sample image includes 6890 vertices, and the three-dimensional coordinates of the 6890 vertices in the three-dimensional human body model are correspondingly determined.

[0075] Step 230: sampling key points in the three-dimensional human body model to obtain three-dimensional coordinates of the key points.

[0076] In some embodiments, a key point set can be pre-set, and for ease of distinction, the key point set is referred to as a first key point set. The first key point set includes key points for locating human body parts, such as joint points, bone points, etc., or other points for assisting in locating human body parts. Then, in step 230, key point sampling is performed in the three-dimensional human body model according to the first key point set, that is, the vertices corresponding to each key point in the first key point set in the constructed three-dimensional human body model are determined to locate the key points in the three-dimensional human body model, and the three-dimensional coordinates of each key point in the first key point set on the three-dimensional human body model are correspondingly determined. The set first key point set includes multiple key points, and correspondingly, in step 230, the sampled and determined three-dimensional coordinates of multiple key points in the three-dimensional human body model are also.

[0077] Figure 3 1 is a schematic diagram of the position of each key point in the first key point set in an SMPL model according to an embodiment of the present application. Figure 3 There are 24 key points shown in Figure 3 There are 24 key points numbered 0-23.

[0078] In different application scenarios, the key points in the first key point set may be different. For example, if the key point prediction is to identify the posture of the whole body of a human body, the first key point set may include Figure 3 24 key points numbered 0-23 are shown. Of course, other key points for reflecting the whole body posture may also be used.

[0079] For another example, if the predicted key points need to further identify the posture of the human face in addition to the posture of the whole body, then the key points in the first key point set include Figure 3In addition to the 24 key points shown, key points in the face area (such as nose key points, lip key points, left eye key points, right eye key points, left eyebrow key points, right eyebrow key points, etc.) can also be included.

[0080] For another example, if the key points are predicted for the purpose of identifying the posture of the upper body of a human body, the key points in the first key point set may only include the key points in the upper body of the human body.

[0081] For another example, if the prediction of key points is to recognize the hand posture, the first key point set may only include key points in the hand area (such as key points at the base of fingers (thumb, index finger, middle finger, little finger, etc.), key points at the ends of fingers, key points at the palm of the hand, key points at the wrist, etc.). The first key point set is set according to the specific application scenario, so that the training of the key point prediction model is more targeted, can better meet the actual needs, and improve the training efficiency of the key point prediction model.

[0082] In some implementations, the position of the key point in the three-dimensional human body model can be determined by determining the heat map of each key point in the first key point set. The heat map of the key point refers to the key point in the grayscale image of the original image size, represented by a circular Gaussian at the same position, that is, the probability of the pixel in the input image (second sample image) being a key point. For example, if a heat map represents the probability of a pixel being a left elbow key point, that is, the possibility of the left elbow key point appearing at the pixel point is reflected in the form of probability expression, the closer the pixel point is to the left elbow key point, the higher the probability, and the farther the pixel point is from the left elbow key point, the lower the probability, that is, the probability of the pixel point being the left elbow key point and the relative position relationship of the pixel point from the center point follow a Gaussian distribution.

[0083] Step 240, the key point prediction model predicts the key points according to the second sample image to obtain the predicted three-dimensional coordinates of the key points; the key point prediction model is used to predict the three-dimensional coordinates of the key points in the input image that have a gesture recognition function according to the input image.

[0084] The key point prediction model is used to predict the three-dimensional coordinates of the key points indicating the posture presented by the human body according to the input second sample image. The key point prediction model can be a model constructed based on a convolutional neural network, a fully connected neural network, etc. In some embodiments, the backbone network structure of the key point prediction model is a residual neural network (Residual neural network, ResNet) neural network, for example, a ResNet50 network.

[0085] Since the posture presented by the pixel region of the human body is generally described by multiple key points, the predicted three-dimensional coordinates of the multiple key points are also predicted in step 240 .

[0086] In some embodiments, key point prediction can be performed according to a set first key point set. For example, if the first key point set includes: left palm key point, left wrist key point, left elbow key point and left shoulder key point, then in step 240, the predicted three-dimensional coordinates of the left palm key point, left wrist key point, left elbow key point and left shoulder key point are predicted by the key point prediction model.

[0087] Figure 4 FIG. 1 is a schematic diagram showing a second sample image and a three-dimensional human body model constructed based on the first sample image according to an embodiment of the present application. Figure 4 As shown, the three-dimensional human body model M1 is constructed by the SMPL model based on the first sample image T1, and the second sample image S1 is obtained by partially clipping the human body pixel area in the first sample image T1. The three-dimensional human body model M2 is constructed by the SMPL model based on the first sample image T2, and the second sample image S2 is obtained by partially clipping the human body pixel area in the first sample image T2. The three-dimensional human body model M3 is constructed by the SMPL model based on the first sample image T3, and the second sample image S3 is obtained by partially clipping the human body pixel area in the first sample image T3.

[0088] like Figure 4 As shown, the human body pixel areas (i.e., the human body pixel areas of the upper body) in the second sample images S1, S2, and S3 are cropped to different degrees, so that the key points in the human body pixel areas in the second sample images S1-S3 are incomplete. However, the key points of the upper body corresponding to the three-dimensional human body model M1 (or M2, M3) constructed based on the first sample image T1 (or T2, T3) associated with the second sample image S1 (or S2, S3) are complete.

[0089] Step 250, calculating a first prediction loss according to the three-dimensional coordinates of the key point and the predicted three-dimensional coordinates of the key point.

[0090] The first prediction loss refers to the function value of the loss function of the key point prediction model calculated based on the three-dimensional coordinates of the key point (equivalent to the actual three-dimensional coordinates of the key point) and the predicted three-dimensional coordinates of the key point. The loss function of the key point prediction model can be an absolute value loss function, a square loss function, an exponential loss function, etc., which is not specifically limited here.

[0091] Step 260: reversely train the key point prediction model according to the first prediction loss.

[0092] In some embodiments, the model parameters of the key point prediction model can be reversely adjusted using a gradient descent algorithm according to the first prediction loss, and reverse training is stopped when the gradient is less than a threshold. The process of reversely adjusting the parameters of the key point prediction model according to the first prediction loss is the reverse training process of the key point prediction model.

[0093] In some embodiments, after adjusting the model parameters of the key point prediction model once, the key point prediction model is enabled to re-predict the key points of the second sample image to obtain the predicted three-dimensional coordinates of the new output of the key points, and the first prediction loss is recalculated. If the first prediction loss can make the loss function of the key point prediction model converge, the reverse training of the key point prediction model is stopped.

[0094] In this solution, the human body pixel area in the first sample image is occluded or cropped to obtain a second sample image, so that compared with the key points included in the portrait pixel area of ​​the first sample image, the key points included in the human body pixel area of ​​the second sample image are not complete; then the second sample image and the three-dimensional coordinates of the key points sampled in the three-dimensional human body model constructed by the first sample image are used to train the key point prediction model, so that the key point prediction model can learn the ability to automatically complete the occluded key points in the image, so that even if the image input to the key point prediction model is an image in which multiple key points expressing posture are partially occluded, the key point prediction model can accurately output the three-dimensional coordinates of the occluded key points, thereby ensuring the accuracy of the posture identified based on the output three-dimensional coordinates.

[0095] In actual scenarios, since the human body may be obscured by other objects or part of the body is not in the image, some key points of the human body used to characterize the posture are not in the input image. The key point prediction model provided by this application can be used to predict the three-dimensional coordinates of the key points in such scenarios.

[0096] In some embodiments, if a three-dimensional human body model is constructed according to the first sample image through the SMPL model, and the three-dimensional coordinates of key points are determined by sampling from the three-dimensional human body model, it can be understood that the sampled key points are derived from the vertices of the constructed three-dimensional human body model, and the vertices in the three-dimensional human body model are determined according to the protocol uniformly specified by the SMPL model. The relationship between the vertices in the three-dimensional human body model is stable, thereby ensuring the stability of the relationship between multiple key points sampled from the three-dimensional human body model, and there is no need to add additional constraints in the training process to ensure the stability between the key points, which facilitates training and reduces the difficulty of training.

[0097] Furthermore, if a three-dimensional human body model is constructed based on the SMPL model, the constructed three-dimensional human body model includes 6890 vertices, which provides enough vertices for key point sampling, and can avoid the problem of poor model training effect caused by incomplete number of vertices provided by the three-dimensional human body model. Moreover, this solution can be applied to scenarios with multiple numbers and positions of key points, and has a wide range of applications.

[0098] Figure 5 A shows a schematic diagram of key points output for an input target image according to a key point prediction model in the related art. Figure 5 As shown in A, in the input target image P1, the elbow key point of the human body is blocked, and the key point prediction model in the related art outputs the key points for the target image P1, but does not include the blocked elbow key point.

[0099] Figure 5 B shows a schematic diagram of the key points output by the key point prediction model after training according to this scheme for the input target image. Figure 5 As shown in B, the input image is also the target image P1. The three-dimensional key points output by the key point prediction model trained according to this scheme include not only the key points presented in the target image P1, but also the occluded elbow key points.

[0100] It can be seen that the key point prediction model trained based on this scheme can automatically complete the key points of images where the key points are partially occluded. Thus, it can be ensured that the three-dimensional coordinates of the key points output by the trained key point prediction model can accurately reflect the presented posture and characterize the accuracy of subsequent posture recognition.

[0101] In some embodiments of the present application, Figure 2 In step 220, a three-dimensional human body is constructed using the SMPL model. In this embodiment, Figure 6 As shown, in Figure 2 Before step 220, the method further includes:

[0102] Step 610: Obtain a third sample image and annotation information of the third sample image.

[0103] The third sample image refers to a sample image used to train the SMPL model, and the third sample image includes a human body pixel area.

[0104] In some embodiments, the annotation information of the third sample image includes annotated two-dimensional coordinates of annotated key points and / or annotated three-dimensional coordinates of annotated key points. For the sake of distinction, the key points annotated in the third sample image are referred to as annotated key points, the two-dimensional coordinates annotated for the annotated key points in the third sample image are referred to as annotated two-dimensional coordinates, and the three-dimensional coordinates annotated for the annotated key points in the third sample image are referred to as annotated three-dimensional coordinates.

[0105] In some embodiments of the present application, the annotation information includes the annotated two-dimensional coordinates of the annotated key points and the annotated three-dimensional coordinates of the annotated key points; in this embodiment, the annotated three-dimensional coordinates of the annotated key points in the third sample image can be Figure 7 The process shown yields:

[0106] Step 710: Obtain a depth image corresponding to the third sample image.

[0107] In some embodiments, the depth image corresponding to the third sample image may be acquired through a somatosensory device, such as Kinect. Of course, in other embodiments, the depth image may also be acquired through other somatosensory devices.

[0108] The depth image corresponding to the third sample image refers to a depth image acquired at the same position and the same viewing angle as the third sample image, so that the depth image corresponding to the third sample image can reflect the depth information of each pixel in the human body pixel area in the third sample image. The third sample image and the depth image corresponding to the third sample image are both acquired by the target image acquisition device.

[0109] Step 720: Obtain depth values ​​of the annotated key points according to the depth image.

[0110] In the depth image, the pixel value of each pixel is the depth value of the pixel. Therefore, based on the pixel where the marked key point is located in the depth image, the pixel value of the pixel where the marked key point is located in the depth image is used as the depth value of the marked key point.

[0111] Step 730, determining the annotated three-dimensional coordinates of the annotated key points according to the depth values ​​of the annotated key points, the internal parameter matrix of the target image acquisition device and the two-dimensional coordinates of the annotated key points; the target image acquisition device refers to the image acquisition device that acquires the third sample image.

[0112] In some embodiments of the present application, step 730 includes: determining the target transformation matrix corresponding to the labeled key point based on the intrinsic parameter matrix of the target image acquisition device and the depth value of the labeled key point; according to the target transformation matrix, transforming the two-dimensional coordinates of the labeled key point into three-dimensional space to obtain the labeled three-dimensional coordinates of the labeled key point.

[0113] The intrinsic matrix of the target image acquisition device reflects the projection relationship from the three-dimensional space to the two-dimensional image. The two-dimensional coordinates of the key points are the coordinates of the key points on the two-dimensional plane. Therefore, in order to determine the three-dimensional coordinates of the key points in the three-dimensional coordinate system, it is necessary to transform them with the help of the intrinsic matrix of the image acquisition device.

[0114] Assume that the internal parameter matrix of the target image acquisition device is assumed to be K, where the internal parameter matrix K is:

[0115]

[0116] Where f is the focal length, x 0 ,y 0 is the coordinate of the focus (optical center) of the target image acquisition device (such as a camera) relative to the imaging plane.

[0117] Assuming that the two-dimensional coordinates of the labeled key point L1 are (x, y), and the depth value of the labeled key point L1 obtained from the depth image is depth, the target transformation matrix K1 corresponding to the labeled key point is:

[0118] K1=defth*K -1 ; (Formula 2)

[0119] In order to transform the two-dimensional coordinates of the key point to the three-dimensional space, it is necessary to first combine the two-dimensional coordinates of the key point with 1 on the basis of the two-dimensional coordinates of the key point, and obtain the initial three-dimensional coordinates of the key point L1 as (x, y, 1). Then, in the process of passing through the target transformation matrix, the coordinates of the key point on the third coordinate axis are determined based on the parameters in the target transformation matrix. Specifically, the three-dimensional coordinates of the key point L1 in the three-dimensional space can be determined according to the following formula 3:

[0120] Pt3d = (x1, y1, z1) = K1* (x, y, 1); (Formula 3)

[0121] Please continue reading Figure 6 , step 620, the SMPL model constructs a three-dimensional human body model according to the third sample image to obtain a predicted three-dimensional human body model.

[0122] In this application, for the sake of distinction, the three-dimensional human body model established by the SMPL model according to the third sample image is referred to as a predicted three-dimensional human body model. As described above, the SMPL model includes 6890 vertices, and the predicted three-dimensional human body model established by the SMPL model according to the third sample image includes 6890 vertices, and the three-dimensional coordinates of the 6890 vertices in the predicted three-dimensional human body model are correspondingly determined.

[0123] Step 630: Calculate a second prediction loss based on the predicted three-dimensional human body model and the annotation information of the third sample image.

[0124] Step 640: reverse train the SMPL model according to the second prediction loss.

[0125] The second prediction loss refers to the function value of the loss function of the SML model calculated based on the annotation information of the predicted three-dimensional human body model and the third sample image. The loss function of the SMPL model can be a square loss function, an exponential loss function, etc., which is not specifically limited here.

[0126] In some embodiments, the model parameters of the SMPL model can be reversely adjusted using a gradient descent algorithm according to the second prediction loss, and reverse training is stopped when the gradient is less than a threshold. The process of reversely adjusting the model parameters of the SMPL model according to the second prediction loss is the process of reverse training the SML model.

[0127] Through the above training process, the SMPL model can establish a three-dimensional human body model of the human body pixel area in the image according to the input image.

[0128] In some embodiments, the annotation information includes the annotated two-dimensional coordinates of the annotated key points and the annotated three-dimensional coordinates of the annotated key points; in this embodiment, Figure 8 As shown, Figure 6 Step 630 in includes:

[0129] Step 810: sampling the annotated key points in the predicted three-dimensional human body model to obtain predicted three-dimensional coordinates corresponding to the annotated key points.

[0130] Sampling the labeled key points in the predicted three-dimensional human body model refers to locating the labeled key points in the predicted three-dimensional human body model, thereby determining the three-dimensional coordinates of the labeled key points located in the predicted three-dimensional human body model as the predicted three-dimensional coordinates of the labeled key points.

[0131] In some embodiments, a second key point set may be set, the second key point set including a plurality of annotated key points, so that the annotated key points are sampled according to the second key point set in step 810, that is, each annotated key point in the second key point set is located in the predicted three-dimensional human body model, and the predicted three-dimensional coordinates of each annotated key point in the predicted three-dimensional human body model are obtained. The second key point set and the first key point set mentioned above may be the same key point set or different key point sets, which is not specifically limited here.

[0132] Step 820, based on the internal parameter matrix of the target image acquisition device, project the predicted three-dimensional coordinates corresponding to the labeled key points onto the image plane where the third sample image is located to obtain the predicted two-dimensional coordinates of the labeled key points on the image plane; the target image acquisition device refers to the image acquisition device that acquires the third sample image.

[0133] Step 820 is equivalent to simulating the coordinate transformation process involved in the target image acquisition device acquiring the third sample image, and transforming the labeled key points from the three-dimensional space to the two-dimensional plane according to the internal parameter matrix of the target image acquisition device. The specific process of determining the predicted two-dimensional coordinates of the labeled key points can refer to the reverse process of formulas (1)-(3), which will not be repeated here.

[0134] Step 830, calculating the three-dimensional prediction loss according to the annotated three-dimensional coordinates of the annotated key points and the predicted three-dimensional coordinates of the annotated key points.

[0135] The three-dimensional prediction loss refers to the function value of the first loss function of the SMPL model calculated based on the three-dimensional coordinates of the reference key points and the three-dimensional coordinates of the annotated key points. The first loss function of the SMPL model can be a square loss function of the three-dimensional coordinates, an exponential loss function, etc., which is not specifically limited here.

[0136] In some embodiments, the first loss function of the SMPL model can be a weighted function of the first Euclidean distance corresponding to all the labeled key points, where the first Euclidean distance corresponding to the labeled key points refers to the Euclidean distance between the labeled three-dimensional coordinates corresponding to the labeled key points and the corresponding predicted three-dimensional coordinates.

[0137] Assume that the i-th labeled key point P i The three-dimensional coordinates of the annotation are P i (3D), the i-th labeled key point P i The predicted three-dimensional coordinates of i (Pred3D). In this case, the first loss function Loss1 of the SMPL model is:

[0138]

[0139] Among them, J is the total number of annotated key points, ||P i (3D)-P i (Pred3D)|| represents the i-th labeled key point P i The annotated three-dimensional coordinates and the corresponding predicted three-dimensional coordinates are P i (Pred3D) Euclidean distance between them; λ i is the i-th labeled key point P i The corresponding first weight coefficient.

[0140] Among them, the weight coefficient corresponding to each annotated key point can be the same or different. In some embodiments, since the key points on certain body parts contribute more to the gesture recognition in gesture recognition, the key points on these body parts can be set with a larger first weight coefficient. In some embodiments, the first weight coefficient corresponding to each key point can be set to (1 / n), so that the above formula 4 is equivalent to averaging the Euclidean distance between the three-dimensional coordinates of all the annotated key points and the annotated three-dimensional coordinates.

[0141] Step 840, calculating the two-dimensional prediction loss according to the predicted two-dimensional coordinates of the annotated key points and the annotated two-dimensional coordinates of the annotated key points.

[0142] The two-dimensional prediction loss refers to the function value of the second loss function of the SMPL model calculated based on the annotated two-dimensional coordinates of the annotated key points and the predicted two-dimensional coordinates corresponding to the annotated key points. The second loss function of the SMPL model can be a square loss function, an exponential loss function, etc. based on two-dimensional coordinates, which is not specifically limited here.

[0143] In some embodiments, the second loss function of the SMPL model can be a weighted function of the second Euclidean distance corresponding to all the annotated key points, where the second Euclidean distance corresponding to the annotated key point refers to the Euclidean distance between the annotated two-dimensional coordinate corresponding to the annotated key point and the corresponding predicted two-dimensional coordinate. i The two-dimensional coordinates of the annotation are P i (2D), the predicted two-dimensional coordinates corresponding to the i-th annotated key point are P i (Pred2D). In this case, the second loss function Loss2 of the SMPL model is:

[0144]

[0145] Among them, ||P i (2D)-P i (Pred2D)|| represents the i-th labeled key point P i The labeled two-dimensional coordinates P i (2D) and the corresponding predicted two-dimensional coordinate is P i (Pred2D) Euclidean distance between them; μ i is the i-th labeled key point P i The corresponding second weight coefficient. The second weight coefficient corresponding to the same annotation key point may be the same as or different from the corresponding first weight coefficient, which is not specifically limited here.

[0146] The weight coefficient corresponding to each marked key point may be the same as the first weight coefficient or may be different. In some embodiments, since the key points on certain parts of the body contribute more to gesture recognition in gesture recognition, a larger second weight coefficient may be set for the key points on these parts of the body. In some embodiments, the second weight coefficient corresponding to each key point may be set to (1 / n), so that the above formula 5 is equivalent to averaging the Euclidean distances between the two-dimensional coordinates of all marked key points and the marked two-dimensional coordinates.

[0147] Step 850, calculating a second prediction loss according to the three-dimensional prediction loss and the two-dimensional prediction loss.

[0148] In some embodiments, the three-dimensional prediction loss and the two-dimensional prediction loss may be weightedly summed, and the weighted sum result may be used as the second prediction loss.

[0149] In some embodiments, the larger value of the three-dimensional prediction loss and the two-dimensional prediction loss may be determined as the second prediction loss.

[0150] Fig. 9 It is a schematic diagram of training the SMPL model according to an embodiment of the present application.

[0151] like Fig. 9 As shown, after the third sample image is input into the SMPL model, the following steps are specifically included:

[0152] Step 910, constructing a three-dimensional human body model; constructing the three-dimensional human body model through the current model parameters of the SMPL model to obtain a predicted three-dimensional human body model, and correspondingly, the three-dimensional coordinates of each vertex in the predicted three-dimensional human body model are correspondingly determined.

[0153] Step 920: Sampling of labeled key points: Detecting and locating the labeled key points in the predicted three-dimensional human body model to determine the predicted three-dimensional coordinates of the labeled key points.

[0154] Step 930, three-dimensional coordinate projection; that is, according to the intrinsic parameter matrix of the image acquisition device from which the third sample image comes, the predicted three-dimensional coordinates of the marked key points are projected onto a plane perpendicular to the optical axis to obtain the predicted two-dimensional coordinates corresponding to the marked key points.

[0155] Step 940, calculating the 3D prediction loss. Specifically, the 3D prediction loss is calculated according to the annotated 3D coordinates and the predicted 3D coordinates of the annotated key points. The calculation process is described above and will not be repeated here.

[0156] Step 950, calculate the two-dimensional prediction loss. That is, calculate the two-dimensional prediction loss based on the annotated two-dimensional coordinates and the predicted two-dimensional coordinates of the annotated key points. The specific calculation process is described above and will not be repeated here.

[0157] After the three-dimensional prediction loss and the two-dimensional prediction loss are calculated, the model parameters of the SMPL model are reversely adjusted according to the three-dimensional prediction loss and the two-dimensional prediction loss.

[0158] In this solution, the second prediction loss is determined based on the three-dimensional prediction loss and the two-dimensional prediction loss, so that the Euclidean distance between the annotated three-dimensional coordinates and the predicted three-dimensional coordinates of the annotated key point is close to zero, and at the same time, the Euclidean distance between the annotated two-dimensional coordinates and the predicted two-dimensional coordinates of the annotated key point is close to zero. Since the second prediction loss is calculated by combining the two-dimensional coordinates of the annotated key point on the two-dimensional plane (annotated two-dimensional coordinates, predicted two-dimensional coordinates) and the three-dimensional coordinates in the three-dimensional space (annotated three-dimensional coordinates, predicted three-dimensional coordinates), and the SMPL model is reversely trained based on the second prediction loss, the accuracy of the depth value of the key point reflected by the SMPL model for the reconstruction of the three-dimensional human body model can be effectively improved, and the problem of low accuracy of the depth value of the key point in the SMPL model caused by training the SMPL model only based on the sample image and the two-dimensional coordinates of the key point in the related art can be avoided.

[0159] In some embodiments of the present application, the key point prediction model includes a feature extraction network and three branch networks; wherein one branch network is used to predict the coordinates of the key point on one coordinate axis; the predicted three-dimensional coordinates include the predicted coordinates of the key point on three coordinate axes; in this embodiment, Fig.10 As shown, Figure 2 Step 240 in the embodiment comprises:

[0160] Step 1010: extract features from the second sample image through a feature extraction network to obtain a feature map of the second sample image.

[0161] In some embodiments, the feature extraction network may be constructed by a convolutional neural network. In a specific embodiment, the feature extraction network may be a ResNet50 network. The branch network may also be constructed based on a convolutional neural network, which is not specifically limited here.

[0162] Step 1020, input the feature maps into the three branch networks respectively.

[0163] In step 1030, each branch network performs coordinate prediction based on the feature map to obtain the coordinates of the predicted key points on the coordinate axis corresponding to each branch network.

[0164] Since each branch network is used to output coordinates on a coordinate axis, the predicted three-dimensional coordinates of each predicted key point can be obtained by combining the coordinates on the corresponding coordinate axis output by each branch network.

[0165] Fig.11FIG. 1 is a schematic diagram of a key point prediction model predicting the three-dimensional coordinates of a key point according to an embodiment of the present application. Fig.11 As shown in FIG, the key point prediction model includes a ResNet50 network and three branch networks (i.e., an x-direction convolution branch network, a y-direction convolution branch network, and a z-direction convolution branch network).

[0166] like Fig.11 As shown in the figure, after the second sample image is input into the ResNet50 network, the ResNet50 network extracts features from the second sample image and outputs a feature map of the second sample image. Then, the x-direction convolution branch network performs convolution processing on the feature map of the second sample image and outputs the coordinates of the labeled key points in the x direction; the y-direction convolution branch network performs convolution processing on the feature map of the second sample image and outputs the coordinates of the labeled key points in the y direction; the z-direction convolution branch network performs convolution processing on the feature map of the second sample image and outputs the coordinates of the labeled key points in the z direction.

[0167] In some embodiments of the present application, the branch network includes a mean processing layer, an attention-based residual network, and a fully connected layer; in this embodiment, Fig.12 As shown, Fig.10 Step 1030 in the embodiment includes:

[0168] In step 1210, the mean processing layer takes the mean of the feature map along the direction of the coordinate axis corresponding to the branch network to obtain a first intermediate feature map.

[0169] Step 1220: Convolution is performed on the first intermediate feature map by an attention-based residual network to obtain a second intermediate feature map.

[0170] In some embodiments, the mean processing layer can be constructed based on an average pooling neural network. In some embodiments, the attention-based residual network can be a residual channel attention network (Deep Residual Channel Attention Networks, RCAN).

[0171] Step 1230: The fully connected layer outputs the coordinates on the coordinate axis corresponding to the branch network according to the second intermediate feature map.

[0172] For example, if the second sample image is input, the number of labeled key points is set to J. The feature map F1 of the second sample image can be obtained through the Renet50 network, with a scale of W*H*C.

[0173] For the X direction, the feature map F1 is averaged along the X direction, that is: W*H*C->W*1, and the first intermediate feature map F1_x in the X direction is obtained, and its scale is W*1.

[0174] Similarly, the first intermediate feature map F1_x in the Y direction can be obtained, with a scale of H*1; and the first intermediate feature map F1_z in the Z direction can be obtained, with a scale of C*1.

[0175] After that, the first intermediate feature map F1_x in the X direction is convolved through the attention-based residual network, and finally a vector of dimension J*1 is output through the fully connected layer. The vector of dimension J*1 is used to indicate the coordinate values ​​of all the labeled key points in the X direction. Similarly, the coordinate values ​​of all the labeled key points in the Y and Z directions can be output. Finally, the vectors output by the fully connected network corresponding to the X, Y, and Z directions are combined to obtain a matrix of dimension J*29, which is used to indicate the three-dimensional coordinates corresponding to the 29 labeled key points.

[0176] Fig.13 FIG. 1 is a schematic diagram showing the branch network prediction coordinates in the x direction according to an embodiment of the present application. Fig.13 As shown, the feature map of the third sample image is input into the mean processing layer to obtain the first intermediate feature map in the x direction; then the first intermediate feature map in the x direction is input into the attention-based residual network, and the attention-based residual network outputs the second intermediate feature map; then, the second intermediate feature map is fully connected by the fully connected layer to output the coordinates of the labeled key points in the x direction. The output dimension of the fully connected layer is J*1, where J is the total number of labeled key points.

[0177] Fig.14 is a flowchart of a three-dimensional key point prediction method according to an embodiment of the present application. The method can be executed by an electronic device with processing capabilities, and the electronic device can be Figure 1 The method may be executed by the server 120 or the terminal 110, or by the server 120 and the terminal 110, which is not specifically limited here. Fig.14 As shown, the method includes:

[0178] Step 1410, acquiring a target image, where the target image includes a human body pixel area.

[0179] Step 1420, a key point prediction model performs three-dimensional key point prediction based on the target image to obtain target three-dimensional key point information. The key point prediction model is trained according to the training method of the key point prediction model in any of the above embodiments.

[0180] The target image refers to an image to be predicted for three-dimensional key points, and the target image includes a human pixel region. The key points representing the posture presented by the human pixel region are predicted by inputting the target image into the human pixel region.

[0181] In some embodiments, the key point prediction model can be based on Fig.11 The process shown predicts the three-dimensional coordinates of each key point in the human body pixel area in the target image based on the target image.

[0182] In this solution, based on the trained key point prediction model, a target image can be input into the key point prediction model, and the key point prediction model outputs the three-dimensional coordinates of the key points in the human pixel area in the target image according to the target image, so as to facilitate the subsequent recognition of the posture presented by the human pixel area in the target image according to the three-dimensional coordinates of the output key points. Moreover, since the key point prediction model is trained by partially occluding or cropping the human pixel area in the first sample image during the training process to obtain the second sample image, the key point prediction model learns the ability to automatically complete the occluded key points in the image, so that even if there are occluded key points in the human pixel area of ​​the target image, the key point prediction model can also output the three-dimensional coordinates of the occluded key points.

[0183] Fig.15 is a flowchart of performing three-dimensional key point prediction according to a specific implementation. The method can be executed by an electronic device with processing capabilities, which can be Figure 1 The method may be executed by the server 120 or the terminal 110, or by the server 120 and the terminal 110, which is not specifically limited here. Fig.15 As shown, including:

[0184] Step 1510, obtaining training data of the SMPL model and training the SMPL model.

[0185] Specifically, the data of the SMPL model includes the third sample image and the annotation information of the third sample image, and the annotation information includes the annotated two-dimensional coordinates and the annotated three-dimensional coordinates of the annotated key points in the human body pixel area in the third sample image. Among them, the annotated three-dimensional coordinates of the annotated key points are determined by the depth image corresponding to the third sample image and the annotated two-dimensional coordinates of the annotated key points in the third sample image. The specific process of determining the annotated three-dimensional coordinates is described above and will not be repeated here. The depth image corresponding to the third sample image can be acquired by a mobile Kinect. The training process of the SMPL model is described above and will not be repeated here.

[0186] Step 1520: construct a three-dimensional human body model according to the first sample image using the SMPL model.

[0187] Step 1530: Sample key points in the three-dimensional human body model to obtain three-dimensional key point information.

[0188] Step 1540: Partially block and / or crop the human body pixel area in the first sample image to obtain a second sample image.

[0189] Step 1550: training a key point prediction model according to the three-dimensional key point information corresponding to the second sample image and the first sample image.

[0190] Step 1560: perform key point prediction based on the input target image using the trained key point prediction model, and output the three-dimensional coordinates of the key points.

[0191] The training method of the key point prediction model and the three-dimensional key point prediction method provided in this application belong to an important research direction in the field of computational computer vision understanding. By predicting the three-dimensional coordinates of the key points in the pixel area of ​​the human body in the image from the input image, the posture of each part of the human body shown in the image can be inferred and understood based on the predicted three-dimensional coordinates; and then the human body's behavior and actions can be understood according to the determined posture, so as to achieve functions such as intelligent detection, patient monitoring and human-computer interaction.

[0192] In some embodiments of the present application, Fig.14 After step 1420 in the method, the method further includes: determining the posture presented by the human body pixel area in the target image according to the target three-dimensional key point information.

[0193] In some embodiments, after obtaining the target 3D key point information, the key points indicated by the target 3D key point information can be connected in sequence to obtain a posture tree, and the posture presented by the human body pixel area in the target image can be identified based on the posture tree.

[0194] The three-dimensional key point prediction method provided in this solution can be applied to fields such as intelligent video detection, patient monitoring systems, human-computer interaction, virtual reality, human animation, smart homes, smart security, and athlete auxiliary training. The posture presented by the human pixel area in the input target image is determined based on the three-dimensional key point prediction method.

[0195] For example, if smart home control can be performed based on posture (such as hand gestures) in the smart home system, after determining the posture presented by the human body pixel area in the target image according to the method of the present application, the control instruction corresponding to the posture is determined, and then the control instruction corresponding to the identified posture is sent to the corresponding smart home device.

[0196] In some embodiments, the three-dimensional key point prediction method provided by the present application can be applied to human-computer interaction scenarios. In human-computer interaction scenarios, the posture of the real human body is recognized by the electronic device, and then the virtual object (such as a virtual human model, a virtual person, etc.) is controlled according to the recognized posture to imitate the posture of the real human body, so as to realize the somatosensory interaction of the user in the virtual scene (such as a game).

[0197] In this embodiment, Fig.14 After step 1420 in step 1420, the method further includes: controlling the virtual object to simulate the posture presented by the human body pixel area in the target image according to the multiple target three-dimensional key point information. By driving the virtual object to simulate the user's posture, the user can be prevented from being directly exposed in the picture, while allowing others to see the posture expressed by the user.

[0198] Fig.16 A-16C is a schematic diagram showing control of a virtual object based on a real user's gesture according to an embodiment of the present application. Fig.16 A), the target image is input into the key point prediction model, and the key point prediction model performs three-dimensional key point prediction based on the target image and outputs the target three-dimensional key point information. Fig.16 B shows a schematic diagram of displaying the key points indicated by the target three-dimensional key point information in the three-dimensional coordinate system. It can be seen that Fig.16 B shows the posture tree formed by connecting the key points in sequence. Fig.16 The posture presented by the human pixel area in the image shown in A is the same. Afterwards, the virtual object can be controlled according to the target three-dimensional key point information to simulate the posture presented by the human pixel area in the target image. Fig.16 C shows a schematic diagram of the simulated posture of the virtual object based on the target three-dimensional key point information. It can be seen that the simulated posture of the virtual object is the same as the posture presented by the human body pixel area in the target image.

[0199] Furthermore, Fig.16 D shows the SMPL model based on Fig.16 A is a schematic diagram of a three-dimensional human body model constructed from a target image, wherein: Fig.16 D shows the schematic diagram of the constructed three-dimensional human body model in two perspectives. It can be seen that Fig.16 The posture presented by the 3D human body model in D is also the same as Fig.16 The poses presented in the target images of A are the same.

[0200] In some embodiments, the scheme of the present application can also be applied to game scenarios, for example, displaying the posture of a game character in the user interface of the device so that the user can imitate the posture of the game character displayed in the user interface, collecting user images through an image acquisition device, identifying three-dimensional key point information in the user image based on a key point prediction model, and determining the similarity between the posture presented by the user in the user image and the posture presented by the game character in the user interface based on the three-dimensional key point information, and then scoring the posture according to the similarity.

[0201] In some embodiments, the present solution can also be applied to action interaction with an intelligent robot. An image acquisition device and a processor are provided in the intelligent robot. The image acquisition device can acquire images of the area in front of the intelligent robot. The processor can perform three-dimensional key point prediction based on the acquired images through a key point prediction model to obtain target three-dimensional key point information, and identify the user's posture in the image according to the coordinates of the key points indicated by the target three-dimensional key point information; if the user's posture is identified as a preset posture, the intelligent robot is controlled to make a preset response, for example, when the user's posture is identified as a waving gesture, the intelligent robot is controlled to make a welcoming action.

[0202] In some embodiments, the solution of the present application can also be applied to unmanned driving scenarios. In this scenario, an image acquisition device can be deployed on the vehicle terminal to capture images in front of the unmanned vehicle during driving. The processor in the vehicle terminal then uses a key point prediction model to perform three-dimensional key point prediction on the captured image to obtain target three-dimensional key point information, and then determines whether there is a person in front of the vehicle based on the target three-dimensional key point information. Subsequently, the actions of the unmanned vehicle, such as parking, are controlled based on the recognition results.

[0203] The following describes an apparatus embodiment of the present application, which can be used to execute the method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned method embodiment of the present application.

[0204] Fig.17 is a block diagram of a training device for a key point prediction model according to an embodiment of the present application, such as Fig.17As shown, the training device of the key point prediction model includes: a sample image pair acquisition module 1710, which is used to acquire a sample image pair, the sample image pair includes a first sample image and a second sample image, the second sample image is obtained after a partial occlusion process or a cropping process is performed on a human body pixel area in the first sample image; a first three-dimensional human body construction module 1720, which is used to construct a three-dimensional human body according to the first sample image to obtain a three-dimensional human body model; a first sampling module 1730, which is used to sample key points in the three-dimensional human body model to obtain the three-dimensional coordinates of the key points, and the three-dimensional coordinates of the key points are used to determine the posture presented by the human body pixel area; a first prediction module 1740, which is used to predict the key points according to the second sample image by the key point prediction model to obtain the predicted three-dimensional coordinates of the key points; the key point prediction model is used to predict the three-dimensional coordinates of the key points in the input image that have a posture recognition function according to the input image; a first prediction loss calculation module 1750, which is used to calculate the first prediction loss according to the three-dimensional coordinates of the key points and the predicted three-dimensional coordinates of the key points; a first reverse training module 1760, which is used to reversely train the key point prediction model according to the first prediction loss.

[0205] In some embodiments of the present application, a three-dimensional human body model is constructed by a multi-person linear skinning SMPL model; in this embodiment, the training device of the key point prediction model also includes: a second acquisition module, used to acquire a third sample image and annotation information of the third sample image; a model construction module, used to construct a three-dimensional human body model according to the third sample image by the SMPL model to obtain a predicted three-dimensional human body model; a second prediction loss calculation module, used to calculate the second prediction loss according to the predicted three-dimensional human body model and the annotation information of the third sample image; a second reverse training module, used to reverse train the SMPL model according to the second prediction loss.

[0206] In some embodiments of the present application, the annotation information includes the annotated two-dimensional coordinates of the annotated key points and the annotated three-dimensional coordinates of the annotated key points; the second prediction loss calculation module includes: a sampling unit, which is used to sample the annotated key points in the predicted three-dimensional human body model to obtain the predicted three-dimensional coordinates corresponding to the annotated key points; a predicted two-dimensional coordinate determination unit, which is used to project the predicted three-dimensional coordinates corresponding to the annotated key points to the image plane where the third sample image is located according to the internal parameter matrix of the target image acquisition device, to obtain the predicted two-dimensional coordinates of the annotated key points on the image plane; the target image acquisition device refers to an image acquisition device that acquires the third sample image; a three-dimensional prediction loss calculation unit, which is used to calculate the three-dimensional prediction loss according to the annotated three-dimensional coordinates of the annotated key points and the predicted three-dimensional coordinates of the annotated key points; a two-dimensional prediction loss calculation unit, which is used to calculate the two-dimensional prediction loss according to the predicted two-dimensional coordinates of the annotated key points and the annotated two-dimensional coordinates of the annotated key points; a second prediction loss calculation unit, which is used to calculate the second prediction loss according to the three-dimensional prediction loss and the two-dimensional prediction loss.

[0207] In some embodiments of the present application, the annotation information includes the annotated two-dimensional coordinates of the annotated key points and the annotated three-dimensional coordinates of the annotated key points; the training device of the key point prediction model also includes: a depth image acquisition module, used to acquire the depth image corresponding to the third sample image; a depth value acquisition module, used to acquire the depth value of the annotated key points according to the depth image; a three-dimensional coordinate determination module, used to determine the annotated three-dimensional coordinates of the annotated key points according to the depth value of the annotated key points, the internal parameter matrix of the target image acquisition device and the two-dimensional coordinates of the annotated key points; the target image acquisition device refers to an image acquisition device that acquires the third sample image.

[0208] In some embodiments of the present application, a three-dimensional coordinate determination module includes: a target transformation matrix determination unit, which is used to determine the target transformation matrix corresponding to the labeled key point based on the internal parameter matrix of the target image acquisition device and the depth value of the labeled key point; a labeled three-dimensional coordinate determination unit, which is used to transform the two-dimensional coordinates of the labeled key point into three-dimensional space according to the target transformation matrix to obtain the labeled three-dimensional coordinates of the labeled key point.

[0209] In some embodiments of the present application, the key point prediction model includes a feature extraction network and three branch networks; wherein one branch network is used to predict the coordinates of the key point on one coordinate axis; the predicted three-dimensional coordinates include the predicted coordinates of the key point on three coordinate axes; in this embodiment, Fig.17 The first prediction module 1740 includes: a feature extraction unit, which is used to extract features from the second sample image through a feature extraction network to obtain a feature map of the second sample image; an input unit, which is used to input the feature map into three branch networks respectively; and a coordinate output unit, which is used to predict the coordinates of each branch network according to the feature map to obtain the coordinates of the predicted key points on the coordinate axis corresponding to each branch network.

[0210] In some embodiments of the present application, the branch network includes a mean processing layer, an attention-based residual network and a fully connected layer; in this embodiment, the coordinate output unit includes: a mean processing unit, which is used to take the average of the feature map along the direction of the coordinate axis corresponding to the branch network by the mean processing layer to obtain a first intermediate feature map; a convolution processing unit, which is used to perform convolution processing on the first intermediate feature map by the attention-based residual network to obtain a second intermediate feature map; a fully connected output unit, which is used to output the coordinates on the coordinate axis corresponding to the branch network by the fully connected layer according to the second intermediate feature map.

[0211] Fig.18 is a block diagram of a three-dimensional key point prediction device according to an embodiment of the present application, such as Fig.18As shown, the three-dimensional key point prediction device includes: a target image acquisition module 1810, used to acquire a target image, the target image includes a human body pixel area; a second prediction module 1820, used to perform three-dimensional key point prediction based on the target image by a key point prediction model to obtain target three-dimensional key point information, and the key point prediction model is trained according to the training method of the key point prediction model in any of the above embodiments.

[0212] In some embodiments of the present application, the three-dimensional key point prediction device further includes: a posture determination module, which is used to determine the posture presented by the human body pixel area in the target image according to the target three-dimensional key point information.

[0213] In some embodiments of the present application, the three-dimensional key point prediction device also includes: a virtual object control module, which is used to control the virtual object to simulate the posture presented by the human body pixel area in the target image according to the target three-dimensional key point information.

[0214] Fig.19 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Fig.19 The computer system 1900 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0215] like Fig.19 As shown, the computer system 1900 includes a processor, which may be a central processing unit (CPU) 1901, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1902 or the program loaded from the storage part 1908 to the random access memory (RAM) 1903, such as executing the method in the above embodiment. In RAM 1903, various programs and data required for system operation are also stored. CPU 1901, ROM 1902 and RAM 1903 are connected to each other through bus 1904. Input / output (I / O) interface 1905 is also connected to bus 1904.

[0216] In some embodiments, the following components may be connected to the I / O interface 1905: an input section 1906 including a keyboard, a mouse, etc.; an output section 1907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 1908 including a hard disk, etc.; and a communication section 1909 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1909 performs communication processing via a network such as the Internet. A drive 1910 is also connected to the I / O interface 1905 as needed. A removable medium 1911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1910 as needed, so that a computer program read therefrom is installed into the storage section 1908 as needed.

[0217] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1909, and / or installed from a removable medium 1911. When the computer program is executed by a central processing unit (CPU) 1901, various functions defined in the system of the present application are executed.

[0218] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, a device or a device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0219] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0220] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0221] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries computer-readable instructions, and when the computer-readable storage instructions are executed by a processor, the method in any of the above embodiments is implemented.

[0222] According to one aspect of the present application, an electronic device is also provided, which includes: a processor; a memory, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the method in any of the above embodiments is implemented.

[0223] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the method in any of the above embodiments.

[0224] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0225] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0226] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0227] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A three-dimensional key point prediction method, characterized in that: include: Acquire a sample image pair, the sample image pair comprising a first sample image and a second sample image, the second sample image being obtained by partially blocking or cropping a human body pixel region in the first sample image; Constructing a three-dimensional human body according to the first sample image to obtain a three-dimensional human body model; Sampling key points in the three-dimensional human body model to obtain three-dimensional coordinates of the key points, wherein the three-dimensional coordinates of the key points are used to determine the posture presented by the human body pixel area; The key point prediction model predicts the key points according to the second sample image to obtain predicted three-dimensional coordinates of the key points; the key point prediction model is used to predict the three-dimensional coordinates of the key points in the input image with gesture recognition function according to the input image; the key point prediction model includes a feature extraction network and three branch networks; one branch network is used to predict the coordinates of the key points on one coordinate axis; the predicted three-dimensional coordinates include the predicted coordinates of the key points on three coordinate axes; Calculating a first prediction loss according to the three-dimensional coordinates of the key point and the predicted three-dimensional coordinates of the key point; Reverse training the key point prediction model according to the first prediction loss; The step of performing key point prediction according to the second sample image by the key point prediction model to obtain predicted three-dimensional coordinates of the key points includes: Extracting features from the second sample image using the feature extraction network to obtain a feature map of the second sample image; Inputting the feature maps into the three branch networks respectively; the branch networks include a mean processing layer, an attention-based residual network and a fully connected layer; Each of the branch networks performs coordinate prediction according to the feature map to obtain the coordinates of the key point on the coordinate axis corresponding to each branch network, including: taking the average of the feature map along the direction of the coordinate axis corresponding to the branch network by the mean processing layer to obtain a first intermediate feature map; The first intermediate feature map is convolved by the attention-based residual network to obtain a second intermediate feature map; The fully connected layer outputs the coordinates on the coordinate axis corresponding to the branch network according to the second intermediate feature map.

2. The method according to claim 1, characterized in that The three-dimensional human body model is constructed by a multi-person linear skin SMPL model; Before constructing a three-dimensional human body according to the first sample image to obtain a three-dimensional human body model, the method further includes: Acquire a third sample image and annotation information of the third sample image; The SMPL model constructs a three-dimensional human body model according to the third sample image to obtain a predicted three-dimensional human body model; Calculating a second prediction loss according to the predicted three-dimensional human body model and the annotation information of the third sample image; The SMPL model is reversely trained according to the second prediction loss.

3. The method according to claim 2, characterized in that The annotation information includes the two-dimensional coordinates of the annotation key points and the three-dimensional coordinates of the annotation key points; The calculating the second prediction loss according to the predicted three-dimensional human body model and the annotation information of the third sample image includes: Sampling the annotated key points in the predicted three-dimensional human body model to obtain predicted three-dimensional coordinates corresponding to the annotated key points; According to the internal parameter matrix of the target image acquisition device, the predicted three-dimensional coordinates corresponding to the annotated key points are projected onto the image plane where the third sample image is located, so as to obtain the predicted two-dimensional coordinates of the annotated key points on the image plane; the target image acquisition device refers to an image acquisition device that acquires the third sample image; Calculating a three-dimensional prediction loss according to the annotated three-dimensional coordinates of the annotated key points and the predicted three-dimensional coordinates of the annotated key points; Calculating a two-dimensional prediction loss based on the predicted two-dimensional coordinates of the annotated key points and the annotated two-dimensional coordinates of the annotated key points; The second prediction loss is calculated based on the three-dimensional prediction loss and the two-dimensional prediction loss.

4. The method according to claim 2, characterized in that: The annotation information includes the two-dimensional coordinates of the annotation key points and the three-dimensional coordinates of the annotation key points; Before acquiring the third sample image and the annotation information of the third sample image, the method further includes: Acquire a depth image corresponding to the third sample image; Acquire the depth value of the marked key point according to the depth image; The annotated three-dimensional coordinates of the annotated key points are determined according to the depth values ​​of the annotated key points, the internal parameter matrix of the target image acquisition device and the two-dimensional coordinates of the annotated key points. The target image acquisition device refers to an image acquisition device that acquires the third sample image.

5. The method according to claim 4, characterized in that Determining the annotated three-dimensional coordinates of the annotated key points according to the depth values ​​of the annotated key points, the intrinsic parameter matrix of the target image acquisition device and the two-dimensional coordinates of the annotated key points includes: Determine a target transformation matrix corresponding to the marked key point according to the intrinsic parameter matrix of the target image acquisition device and the depth value of the marked key point; According to the target transformation matrix, the two-dimensional coordinates of the annotated key points are transformed into three-dimensional space to obtain the annotated three-dimensional coordinates of the annotated key points.

6. The method according to any one of claims 1 to 5, characterized in that After reverse training the key point prediction model according to the first prediction loss, the method further includes: Acquire a target image, wherein the target image includes a human body pixel area; The key point prediction model performs three-dimensional key point prediction based on the target image to obtain target three-dimensional key point information.

7. The method according to claim 6, characterized in that After the key point prediction model performs three-dimensional key point prediction according to the target image to obtain the target three-dimensional key point information, the method further includes: The posture presented by the human body pixel area in the target image is determined according to the target three-dimensional key point information.

8. The method according to claim 6, characterized in that After the key point prediction model performs three-dimensional key point prediction according to the target image to obtain the target three-dimensional key point information, the method further includes: According to the target three-dimensional key point information, the virtual object is controlled to simulate the posture presented by the human body pixel area in the target image.

9. A three-dimensional key point prediction device, characterized in that: include: A sample image pair acquisition module, used to acquire a sample image pair, wherein the sample image pair includes a first sample image and a second sample image, wherein the second sample image is obtained by partially blocking or cropping a human body pixel region in the first sample image; A first three-dimensional human body construction module, used to construct a three-dimensional human body according to the first sample image to obtain a three-dimensional human body model; A first sampling module, used for sampling key points in the three-dimensional human body model to obtain three-dimensional coordinates of the key points, wherein the three-dimensional coordinates of the key points are used to determine the posture presented by the human body pixel area; A first prediction module is configured to predict key points according to the second sample image by a key point prediction model to obtain predicted three-dimensional coordinates of the key points; the key point prediction model is configured to predict the three-dimensional coordinates of key points in the input image having a gesture recognition function according to the input image; the key point prediction model includes a feature extraction network and three branch networks; one branch network is configured to predict the coordinates of the key points on one coordinate axis; the predicted three-dimensional coordinates include the predicted coordinates of the key points on three coordinate axes; A first prediction loss calculation module, used for calculating a first prediction loss according to the three-dimensional coordinates of the key point and the predicted three-dimensional coordinates of the key point; A first reverse training module, used for reverse training the key point prediction model according to the first prediction loss; Wherein, the first sampling module includes: a feature extraction unit, configured to extract features from the second sample image through the feature extraction network to obtain a feature map of the second sample image; An input unit, used to input the feature map into the three branch networks respectively; the branch networks include a mean processing layer, an attention-based residual network and a fully connected layer; A coordinate output unit, configured to perform coordinate prediction by each branch network according to the feature map to obtain the coordinates of the key point on the coordinate axis corresponding to each branch network; The coordinate output unit includes: a mean processing unit, which is used to take the mean of the feature map along the direction of the coordinate axis corresponding to the branch network by the mean processing layer to obtain a first intermediate feature map; a convolution processing unit, which is used to perform convolution processing on the first intermediate feature map by the attention-based residual network to obtain a second intermediate feature map; and a fully connected output unit, which is used to output the coordinates on the coordinate axis corresponding to the branch network by the fully connected layer according to the second intermediate feature map.

10. The device according to claim 9, characterized in that The three-dimensional human body model is constructed by a multi-person linear skin SMPL model; the three-dimensional key point prediction device also includes: A second acquisition module, used to acquire a third sample image and annotation information of the third sample image; A model construction module, configured to construct a three-dimensional human body model according to the third sample image using the SMPL model to obtain a predicted three-dimensional human body model; A second prediction loss calculation module, used to calculate a second prediction loss according to the predicted three-dimensional human body model and the annotation information of the third sample image; A second reverse training module is used to reversely train the SMPL model according to the second prediction loss.

11. The device according to claim 10, characterized in that The annotation information includes the annotated two-dimensional coordinates of the annotated key points and the annotated three-dimensional coordinates of the annotated key points; the second prediction loss calculation module includes: A sampling unit, used for sampling the annotated key points in the predicted three-dimensional human body model to obtain predicted three-dimensional coordinates corresponding to the annotated key points; a predicted two-dimensional coordinate determination unit, configured to project the predicted three-dimensional coordinates corresponding to the annotated key points onto the image plane where the third sample image is located according to the internal parameter matrix of the target image acquisition device, so as to obtain the predicted two-dimensional coordinates of the annotated key points on the image plane; the target image acquisition device refers to an image acquisition device that acquires the third sample image; A three-dimensional prediction loss calculation unit, used to calculate the three-dimensional prediction loss according to the annotated three-dimensional coordinates of the annotated key point and the predicted three-dimensional coordinates of the annotated key point; A two-dimensional prediction loss calculation unit, used to calculate the two-dimensional prediction loss according to the predicted two-dimensional coordinates of the annotated key point and the annotated two-dimensional coordinates of the annotated key point; The second prediction loss calculation unit is used to calculate the second prediction loss according to the three-dimensional prediction loss and the two-dimensional prediction loss.

12. The device according to claim 10, characterized in that The annotation information includes the annotated two-dimensional coordinates of the annotated key points and the annotated three-dimensional coordinates of the annotated key points; the three-dimensional key point prediction device also includes: A depth image acquisition module, used to acquire a depth image corresponding to the third sample image; A depth value acquisition module, used to acquire the depth value of the marked key point according to the depth image; A three-dimensional coordinate determination module is used to determine the annotated three-dimensional coordinates of the annotated key point according to the depth value of the annotated key point, the internal parameter matrix of the target image acquisition device and the two-dimensional coordinates of the annotated key point; the target image acquisition device refers to an image acquisition device that acquires the third sample image.

13. The device according to claim 12, characterized in that The three-dimensional coordinate determination module comprises: A target transformation matrix determination unit, used to determine the target transformation matrix corresponding to the marked key point according to the intrinsic parameter matrix of the target image acquisition device and the depth value of the marked key point; The annotated three-dimensional coordinate determining unit is used to transform the two-dimensional coordinates of the annotated key points into three-dimensional space according to the target transformation matrix to obtain the annotated three-dimensional coordinates of the annotated key points.

14. The device according to any one of claims 9 to 13, characterized in that The three-dimensional key point prediction device also includes: A target image acquisition module, used to acquire a target image, wherein the target image includes a human body pixel area; The second prediction module is used to perform three-dimensional key point prediction based on the target image by the key point prediction model to obtain target three-dimensional key point information.

15. The device according to claim 14, characterized in that The three-dimensional key point prediction device also includes: The posture determination module is used to determine the posture presented by the human body pixel area in the target image according to the target three-dimensional key point information.

16. The device according to claim 14, characterized in that The three-dimensional key point prediction device also includes: The virtual object control module is used to control the virtual object to simulate the posture presented by the human body pixel area in the target image according to the target three-dimensional key point information.

17. An electronic device, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 8 is implemented.

18. A computer-readable storage medium having computer-readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

19. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images