Posture recognition, training method and device of posture recognition model, equipment and medium

Abstract

The application discloses a posture recognition method and device, a posture recognition model training method and device, equipment and a medium. The embodiments of the application can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving. The method comprises the following steps: acquiring a first image of a target object; acquiring image position information of a first key point based on the first image, the image position information being used for indicating a position of the first key point in the first image, the first key point being a key point used for describing a two-dimensional posture of the target object; performing feature extraction on the image position information to obtain posture representation features of the first image; and identifying a three-dimensional posture of the target object based on the posture representation features of the first image to obtain target three-dimensional posture parameters. In this way, the three-dimensional posture of the target object is identified based on the posture representation features, the process is relatively simple, the calculation amount is saved, and the efficiency of posture recognition is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a posture recognition, posture recognition model training, method, apparatus, device and medium. Background Technology

[0002] With the development of computer technology, more and more application scenarios require the recognition of the three-dimensional pose of target objects in order to obtain three-dimensional pose parameters that indicate the three-dimensional pose of the target objects.

[0003] The pose recognition method of the related technology includes: determining the position of the first key point in the image of the target object, predicting the depth of the first key point, determining the position of the first key point in three-dimensional space based on the position and depth of the first key point in the image, recognizing the three-dimensional pose of the target object based on the position of the first key point in three-dimensional space, and obtaining three-dimensional pose parameters.

[0004] The above-mentioned pose recognition method has a relatively complex process and requires a large amount of computation, resulting in low efficiency of pose recognition. Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for posture recognition and training a posture recognition model, which can be used to improve the efficiency of posture recognition. The technical solution is as follows:

[0006] On one hand, embodiments of this application provide a pose recognition method, the method comprising:

[0007] Get the first image of the target object;

[0008] Based on the first image, image position information of the first key point is obtained. The image position information is used to indicate the position of the first key point in the first image. The first key point is a key point used to describe the two-dimensional pose of the target object.

[0009] Feature extraction is performed on the image position information to obtain the pose representation features of the first image;

[0010] Based on the pose representation features of the first image, the three-dimensional pose of the target object is identified, and the target three-dimensional pose parameters are obtained.

[0011] A method for training a pose recognition model is also provided, the method comprising:

[0012] Obtain the sample image of the sample object and the standard three-dimensional pose parameters corresponding to the sample image;

[0013] Based on the sample image, the sample image location information of the first sample key point is obtained. The sample image location information is used to indicate the position of the first sample key point in the sample image. The first sample key point is a key point used to describe the two-dimensional pose of the sample object.

[0014] The initial pose recognition model is invoked to extract features from the position information of the sample image to obtain sample pose representation features. Based on the sample pose representation features, the three-dimensional pose of the sample object is recognized to obtain the sample three-dimensional pose parameters.

[0015] Based on the standard three-dimensional pose parameters and the sample three-dimensional pose parameters, the target loss is obtained;

[0016] The target loss is used to update the model parameters of the initial pose recognition model to obtain the target pose recognition model.

[0017] On the other hand, a posture recognition device is provided, the device comprising:

[0018] The first acquisition unit is used to acquire a first image of the target object; and to acquire image position information of a first key point based on the first image, wherein the image position information is used to indicate the position of the first key point in the first image, and the first key point is a key point used to describe the two-dimensional pose of the target object.

[0019] The first feature extraction unit is used to extract features from the image position information to obtain the pose representation features of the first image;

[0020] The first recognition unit is used to recognize the three-dimensional pose of the target object based on the pose representation features of the first image, and obtain the target three-dimensional pose parameters.

[0021] In one possible implementation, the first image of the target object is an image located in the frame following the second image of the target object in the video of the target object; the first feature extraction unit is used to encode the image position information to obtain a first encoded feature of the first image; fuse the first encoded feature of the first image with the fusion feature of the second image to obtain a fusion feature of the first image; encode the fusion feature of the first image to obtain a second encoded feature of the first image; and fuse the second encoded feature of the first image with the pose representation feature of the second image to obtain a pose representation feature of the first image.

[0022] In one possible implementation, the device further includes:

[0023] The prediction unit is used to predict the target projection parameters based on the pose representation features of the first image;

[0024] The determining unit is used to determine the position of the second key point in three-dimensional space based on the target three-dimensional pose parameters, wherein the second key point is a key point used to describe the three-dimensional pose of the target object; and to project the position of the second key point in three-dimensional space onto the first image based on the target projection parameters to obtain the target projection position information of the second key point.

[0025] The first acquisition unit is further configured to acquire a special effects image corresponding to the first image based on the target projection position information.

[0026] In one possible implementation, the first acquisition unit is configured to extract the projection position information of a third key point from the target projection position information, extract the image position information of the third key point from the image position information, determine a first correction loss based on the projection position information and the image position information of the third key point, update the target projection parameters using the first correction loss to obtain first projection parameters, correct the target three-dimensional pose parameters based on the first projection parameters to obtain corrected three-dimensional pose parameters, correct the first projection parameters based on the corrected three-dimensional pose parameters to obtain corrected projection parameters, and acquire a special effects image corresponding to the first image based on the corrected three-dimensional pose parameters and the corrected projection parameters.

[0027] In one possible implementation, the first acquisition unit is configured to project the position of the second key point in the three-dimensional space onto the first image based on the first projection parameters to obtain the first projection position information of the second key point; extract the projection position information of the fourth key point from the first projection position information; extract the image position information of the fourth key point from the image position information; determine the second correction loss based on the projection position information and the image position information of the fourth key point; and update the target three-dimensional pose parameters using the second correction loss to obtain the corrected three-dimensional pose parameters.

[0028] In one possible implementation, the first acquisition unit is configured to: determine the corrected position of the second key point in the three-dimensional space based on the corrected three-dimensional pose information; project the corrected position of the second key point in the three-dimensional space onto the first image based on the first projection parameters to obtain second projection position information of the second key point; extract the projection position information of the fifth key point from the second projection position information; extract the image position information of the fifth key point from the image position information; determine a third correction loss based on the projection position information and the image position information of the fifth key point; and update the first projection parameters using the third correction loss to obtain the corrected projection parameters.

[0029] In one possible implementation, the first acquisition unit is configured to determine a target 3D effect model matching the target object based on the corrected 3D pose parameters; and project the target 3D effect model onto the first image based on the corrected projection parameters to obtain an effect image corresponding to the first image.

[0030] In one possible implementation, the first acquisition unit is configured to crop the image located within the object detection box in the first image, and acquire a target image based on the cropped image; perform key point detection on the target image to obtain reference position information of the first key point, the reference position information being used to indicate the position of the first key point in the target image; and convert the reference position information into image position information based on the position transformation relationship between the target image and the first image.

[0031] In one possible implementation, the determining unit is further configured to, in response to the first image satisfying the object detection condition, detect the region in the first image where the target object is located, and determine the object detection box based on the region; and in response to the first image not satisfying the object detection condition, determine the object detection box based on a reference detection box, wherein the reference detection box is determined based on an image that satisfies the object detection condition and is associated with the first image.

[0032] In one possible implementation, the first feature extraction unit is used to call the target pose recognition model to extract features from the image position information to obtain the pose representation features of the first image.

[0033] The first recognition unit is used to call the target pose recognition model to recognize the three-dimensional pose of the target object based on the pose representation features of the first image, and obtain the target three-dimensional pose parameters.

[0034] In one possible implementation, the prediction unit is used to call a target parameter prediction model to predict target projection parameters based on the pose representation features of the first image.

[0035] A training device for a pose recognition model is also provided, the device comprising:

[0036] The second acquisition unit is used to acquire a sample image of a sample object and standard three-dimensional pose parameters corresponding to the sample image; and to acquire sample image position information of a first sample key point based on the sample image, wherein the sample image position information is used to indicate the position of the first sample key point in the sample image, and the first sample key point is a key point used to describe the two-dimensional pose of the sample object.

[0037] The second feature extraction unit is used to call the initial pose recognition model to extract features from the position information of the sample image to obtain sample pose representation features.

[0038] The second identification unit is used to identify the three-dimensional pose of the sample object based on the sample pose characterization features, and obtain the three-dimensional pose parameters of the sample.

[0039] The second acquisition unit is further configured to acquire the target loss based on the standard three-dimensional pose parameters and the sample three-dimensional pose parameters;

[0040] The update unit is used to update the model parameters of the initial pose recognition model using the target loss, so as to obtain the target pose recognition model.

[0041] In one possible implementation, the second acquisition unit is further configured to call the initial parameter prediction model to predict the sample projection parameters based on the sample pose representation features; and to acquire the target loss based on the standard three-dimensional pose parameters, the sample three-dimensional pose parameters, and the sample projection parameters.

[0042] In one possible implementation, the second acquisition unit is further configured to: determine a first sub-loss based on the standard 3D pose parameters and the sample 3D pose parameters; determine the position of a second sample keypoint in 3D space based on the sample 3D pose parameters, wherein the second sample keypoint is a keypoint used to describe the 3D pose of the sample object; project the position of the second sample keypoint in 3D space onto the sample image based on the sample projection parameters to obtain sample projection position information of the second sample keypoint; extract sample projection position information of a third sample keypoint from the sample projection position information, extract sample image position information of the third sample keypoint from the sample image position information, determine a second sub-loss based on the sample projection position information and the sample image position information of the third sample keypoint; and acquire the target loss based on the first sub-loss and the second sub-loss.

[0043] In one possible implementation, the updating unit is further configured to update the model parameters of the initial parameter prediction model using the target loss, thereby obtaining the target parameter prediction model.

[0044] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to enable the computer device to implement any of the above-described pose recognition methods or pose recognition model training methods.

[0045] On the other hand, a computer-readable storage medium is also provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to enable a computer to implement any of the above-described pose recognition methods or pose recognition model training methods.

[0046] On the other hand, a computer program product is also provided, which includes a computer program or computer instructions, the computer program or computer instructions being loaded and executed by a processor to enable a computer to implement any of the above-described posture recognition methods or posture recognition model training methods.

[0047] The technical solution provided in this application has at least the following beneficial effects:

[0048] The technical solution provided in this application identifies the three-dimensional pose of the target object based on pose representation features. The pose representation features are obtained by extracting features from the image position information of the first key point. In this pose recognition process, it is not necessary to determine the depth of the first key point or the position of the first key point in three-dimensional space. The process is relatively simple, which helps to save computation and thus improve the efficiency of pose recognition. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0051] Figure 2 This is a flowchart of a posture recognition method provided in an embodiment of this application;

[0052] Figure 3 This is a schematic diagram of a first key point provided in an embodiment of this application;

[0053] Figure 4 This is a schematic diagram illustrating a key point detection process provided in an embodiment of this application;

[0054] Figure 5 This is a schematic diagram of a second key point provided in an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of a posture recognition process provided in an embodiment of this application;

[0056] Figure 7 This is a schematic diagram illustrating the process of alternately correcting the target projection parameters and the target three-dimensional attitude parameters according to an embodiment of this application;

[0057] Figure 8 This is a schematic diagram illustrating a process for establishing a connection between a second key point and a key point of a candidate special effects model, as provided in an embodiment of this application.

[0058] Figure 9 This is a schematic diagram illustrating a process for obtaining a special effects image of a first image according to an embodiment of this application;

[0059] Figure 10 This is a schematic diagram of a test result provided in an embodiment of this application;

[0060] Figure 11This is a schematic diagram of a comparison result provided in an embodiment of this application;

[0061] Figure 12 This is a schematic diagram of a comparison result provided in an embodiment of this application;

[0062] Figure 13 This is a flowchart illustrating a training method for a pose recognition model provided in an embodiment of this application;

[0063] Figure 14 This is a schematic diagram of a posture recognition device provided in an embodiment of this application;

[0064] Figure 15 This is a schematic diagram of a training device for a pose recognition model provided in an embodiment of this application;

[0065] Figure 16 This is a schematic diagram of the structure of a server provided in an embodiment of this application;

[0066] Figure 17 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0068] In exemplary embodiments, the posture recognition method and posture recognition model training method provided in this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0069] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science. AI attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0070] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, machine learning / deep learning, autonomous driving, and intelligent transportation. The solutions provided in this application's embodiments relate to computer vision and machine learning technologies within artificial intelligence.

[0071] Computer vision (CV) is the science of enabling machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing, detecting, and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, intelligent transportation, and posture recognition.

[0072] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0073] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, smart customer service, vehicle networking, and intelligent transportation. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0074] Figure 1A schematic diagram of an implementation environment provided in an embodiment of this application is shown. This implementation environment may include: terminal 11 and server 12.

[0075] The posture recognition method provided in this application embodiment can be executed by terminal 11, server 12, or jointly by terminal 11 and server 12; this application embodiment does not limit this. In the case where the posture recognition method provided in this application embodiment is jointly executed by terminal 11 and server 12, server 12 undertakes the main computational work, and terminal 11 undertakes the secondary computational work; or, server 12 undertakes the secondary computational work, and terminal 11 undertakes the main computational work; or, server 12 and terminal 11 use a distributed computing architecture for collaborative computation.

[0076] The training method for the pose recognition model provided in this application embodiment can be executed by terminal 11, server 12, or jointly by terminal 11 and server 12; this application embodiment does not limit this. In the case where the training method for the pose recognition model provided in this application embodiment is jointly executed by terminal 11 and server 12, server 12 undertakes the main computational work, and terminal 11 undertakes the secondary computational work; or, server 12 undertakes the secondary computational work, and terminal 11 undertakes the main computational work; or, server 12 and terminal 11 use a distributed computing architecture for collaborative computation.

[0077] It should be noted that the execution device of the pose recognition method and the execution device of the pose recognition model training method can be the same or different, and the embodiments of this application do not limit this.

[0078] In one possible implementation, terminal 11 can be any electronic product capable of human-computer interaction with a user through one or more methods such as a keyboard, touchpad, touchscreen, remote control, voice interaction, or handwriting device. Examples include PCs (Personal Computers), mobile phones, smartphones, PDAs (Personal Digital Assistants), wearable devices, PPCs (Pocket PCs), tablets, smart car systems, smart TVs, smart speakers, smart voice interaction devices, smart home appliances, in-vehicle terminals, and aircraft. Server 12 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. Terminal 11 and server 12 establish a communication connection via wired or wireless network.

[0079] Those skilled in the art should understand that the above-described terminal 11 and server 12 are merely examples. Other existing or future terminals or servers that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0080] Based on the above Figure 1 The implementation environment shown in this application embodiment provides a posture recognition method, which is executed by a computer device. This computer device can be a terminal 11 or a server 12; this application embodiment does not limit this. Figure 2 As shown, the posture recognition method provided in this application embodiment may include the following steps 201 to 204.

[0081] In step 201, the first image of the target object is obtained.

[0082] The target object refers to any object whose three-dimensional pose is to be identified. In this application embodiment, the type of the target object is not limited, as long as the target object has a certain three-dimensional pose. For example, the type of the target object is a person, or the type of the target object is an animal, etc.

[0083] The first image of the target object refers to any image obtained by image acquisition of the target object. This application embodiment identifies the three-dimensional pose of the target object based on the first image of the target object. This application embodiment does not limit the method of acquiring the first image of the target object.

[0084] For example, the methods of obtaining the first image of the target object include, but are not limited to, receiving the first image of the target object sent by an image acquisition device, receiving the first image of the target object uploaded by an interactive object, extracting the first image of the target object from an image library, or taking a screenshot of the first image of the target object from the network.

[0085] For example, acquiring the first image of the target object may further include: acquiring a video of the target object, and using any frame from the video of the target object as the first image of the target object. The video of the target object is a video obtained by capturing video of the target object.

[0086] In step 202, image position information of the first key point is obtained based on the first image. The image position information is used to indicate the position of the first key point in the first image. The first key point is a key point used to describe the two-dimensional pose of the target object.

[0087] After acquiring a first image of the target object, image position information of a first keypoint is obtained based on the first image. This image position information indicates the location of the first keypoint within the first image. The first keypoint is a keypoint used to describe the two-dimensional pose of the target object.

[0088] For example, the first key points are related to each other. Based on the position of the first key points in the first image and the relationship between the first key points, the two-dimensional pose of the target object can be located. The number of first key points and the relationship between the first key points are set based on experience or flexibly adjusted according to the application scenario. This application embodiment does not limit this.

[0089] For example, taking the type of the target object as a person, such as Figure 3 As shown, there are 19 first key points, namely: nose key point 0, left eye key point 1, right eye key point 2, left ear key point 3, right ear key point 4, left shoulder key point 5, right shoulder key point 6, left elbow key point 7, right elbow key point 8, left wrist key point 9, right wrist key point 10, left hip key point 11, right hip key point 12, left knee key point 13, right knee key point 14, left ankle key point 15, right ankle key point 16, left palm key point 17, and right palm key point 18. The relationships between these 19 first key points are as follows: Figure 3 The connection of each of the first key points is shown in the diagram. A first key point can also be called a first joint.

[0090] This application does not limit the form of the image location information, as long as it can indicate the position of the first key point in the first image. For example, the image location information can be in the form of two-dimensional coordinates, vectors, or matrices. It should be noted that the number of first key points may be one or more, and the image location information of the first key point refers to the image location information of each first key point.

[0091] In one possible implementation, the process of obtaining image location information of a first key point based on a first image includes steps 2021 to 2023.

[0092] Step 2021: Crop the image located within the object detection box in the first image, and obtain the image to be detected based on the cropped image.

[0093] The object detection bounding box is used to constrain the image to be cropped in the first image. For example, the image to be cropped constrained by the object detection bounding box is the image corresponding to the predicted or accurate target object region. After cropping the image within the object detection bounding box in the first image, the image to be detected is obtained based on the cropped image. This image to be detected has a relatively large proportion of the target object, which is beneficial to improving the accuracy of key point detection.

[0094] In an exemplary embodiment, the method for obtaining the image to be detected based on the cropped image can be: using the cropped image as the image to be detected. In another exemplary embodiment, the method for obtaining the image to be detected based on the cropped image can be: scaling the cropped image to a target size and using that target-size image as the image to be detected. The target size is set empirically or can be flexibly adjusted according to the application scenario; this embodiment does not limit this.

[0095] Before cropping the image within the object detection box in the first image, the object detection box needs to be determined. In one possible implementation, the process of determining the object detection box includes: in response to the first image satisfying the object detection conditions, detecting the region where the target object is located in the first image, and determining the object detection box based on the region; in response to the first image not satisfying the object detection conditions, determining the object detection box based on a reference detection box. The reference detection box is determined based on an image associated with the first image that satisfies the object detection conditions.

[0096] Object detection conditions are used to limit the area of ​​the image in which the object to be detected is located. The object detection conditions are set based on experience or can be flexibly adjusted according to the application scenario. This application embodiment does not impose any limitations on this.

[0097] For example, the first image is a frame from the video of the target object. An image that satisfies the object detection condition can refer to an image located in a reference number of frames within the video of the target object. The reference number is set empirically or adjusted flexibly according to the actual application scenario. There can be one or more reference numbers, and this embodiment does not limit this. For example, the reference number can be (1+n*k), where k is used to limit the number of frames between two adjacent frames that satisfy the object detection condition, and k is an integer. The value of k can be set empirically, for example, k can be 15, or k can be 10, etc. n is any integer that makes (1+n*k)≤M, and M is the total number of frames in the video of the target object. Of course, the reference number can also be other cases, which will not be elaborated here.

[0098] For example, the first image is a frame from the video of the target object, and the image that satisfies the object detection condition may also refer to a keyframe from the video of the target object.

[0099] When the first image meets the object detection criteria, the region where the target object is located in the first image is detected. For example, detecting the region where the target object is located in the first image includes calling an object region detection model to detect the region where the target object is located in the first image. The object region detection model is a model with the function of detecting the region where an object is located in an image, and the object region detection model can be trained through supervised training.

[0100] After detecting the region where the target object is located in the first image, an object detection box is determined based on the detected region. The image to be cropped, constrained by the object detection box determined based on the detected region, can be regarded as the image corresponding to the exact region where the target object is located. In an exemplary embodiment, the object detection box determined based on the detected region can be implemented by using the box corresponding to the region as the object detection box. For example, the box corresponding to the region can refer to the smallest box that includes the region.

[0101] For example, determining the object detection box based on the detected region can also be achieved by adjusting the boxes corresponding to the region to obtain the object detection box. The object detection box obtained by adjusting the boxes corresponding to the region has higher standardization and is more suitable for cropping the image to facilitate subsequent key point detection.

[0102] The method of adjusting the bounding box corresponding to the region to obtain the object detection box can be set based on experience or flexibly adjusted according to the application scenario, and this application embodiment does not limit it in this way. For example, the method of adjusting the bounding box corresponding to the region to obtain the object detection box may include: fixing the width of the bounding box corresponding to the region, adjusting the height of the bounding box corresponding to the region so that the ratio of the width to the adjusted height is the target ratio; keeping the center position of the currently obtained box unchanged, and enlarging the currently obtained box by the target number of times to obtain the object detection box.

[0103] The target ratio and target quantity multiple are set based on experience or can be flexibly adjusted according to the application scenario; this application embodiment does not limit them in this regard. For example, the target ratio can be 0.6:1 or 0.5:1, etc. The target quantity multiple can be 1.2 times or 1.5 times, etc.

[0104] Of course, in some embodiments, adjusting the bounding box corresponding to the region to obtain the object detection box can also be implemented in other ways. For example, adjusting the bounding box corresponding to the region to obtain the object detection box can also be implemented by: keeping the center position of the bounding box corresponding to the region unchanged, and enlarging the bounding box corresponding to the region by a target number of times to obtain the object detection box. For example, adjusting the bounding box corresponding to the region to obtain the object detection box can also be implemented by: keeping the center position of the bounding box corresponding to the region unchanged, and enlarging the bounding box corresponding to the region by a target number of times to obtain the object detection box.

[0105] In an exemplary embodiment, after detecting the region where the target object is located in the first image, the bounding box corresponding to the region can be stored in the detection bounding box buffer.

[0106] The first image may not meet the object detection criteria. In this case, the object detection box is determined based on a reference detection box, which is determined based on an image that meets the object detection criteria and is associated with the first image. That is, the reference detection box is a pre-determined box. If the first image does not meet the object detection criteria, the object detection box is directly determined based on the pre-determined reference detection box, which helps improve the efficiency of object detection box determination. For example, the reference detection box can be stored in a detection box buffer, allowing it to be retrieved from the buffer.

[0107] Since the reference detection box is determined based on an image that meets the object detection conditions and is associated with the first image, the image that needs to be cropped based on the object detection box determined by the reference detection box can be regarded as the image corresponding to the region where the predicted target object is located.

[0108] If the region containing the target object in an image that satisfies the object detection criteria and is associated with the first image is at the same location or has a small difference from the region containing the target object in the first image, then the first image can be cropped using a reference detection box determined based on the image that satisfies the object detection criteria and is associated with the first image. For example, taking a frame from a video where the first image is the target object, the image that satisfies the object detection criteria and is associated with the first image refers to the image in the video containing the target object that satisfies the object detection criteria and is located before and closest to the first image.

[0109] For example, the implementation of determining the object detection box based on the reference detection box is the same as the principle of determining the object detection box based on the detected region when the first image meets the object detection conditions, and will not be repeated here.

[0110] Before determining the object detection box based on the reference detection box, it is necessary to first determine the reference detection box based on an image that meets the object detection conditions and is associated with the first image. The image that meets the object detection conditions and is associated with the first image is called the reference image. For example, the process of determining the reference detection box based on the reference image may include: detecting the region where the target object is located in the reference image, and using the box corresponding to that region as the reference detection box. For example, the process of determining the reference detection box based on the reference image may also be: determining the reference detection box based on the position of the first keypoint in the reference image; the reference detection box determined in this way is a more accurate detection box. For example, determining the reference detection box based on the position of the first keypoint in the reference image can be done by using the smallest box including the position of the first keypoint in the reference image as the reference detection box.

[0111] In the exemplary embodiments, the above-described implementation of determining the object detection box is merely an example, and the embodiments of this application are not limited thereto. In some embodiments, the implementation of determining the object detection box can also be: detecting the region where the target object is located in the first image, and determining the object detection box based on the region. That is, without judging whether the first image meets the object detection conditions, the region where the target object is located in the first image is directly detected and the object detection box is determined based on the region. The object detection box determined in this way has a high degree of matching with the first image.

[0112] Step 2022: Perform key point detection on the image to be detected to obtain the reference position information of the first key point. The reference position information is used to indicate the position of the first key point in the image to be detected.

[0113] After acquiring the image to be detected, keypoint detection is performed on the image to obtain the reference position information of the first keypoint. This reference position information indicates the location of the first keypoint within the image. For example, keypoint detection can be implemented by calling a keypoint detection model. The keypoint detection model is a model capable of detecting keypoints in an image, and it can be obtained through supervised training. The structure of the keypoint detection model can be set empirically or flexibly adjusted according to the application scenario; for example, it can be a neural network model. After calling the keypoint detection model to perform keypoint detection on the image, the model outputs a detection result, which includes the reference position information of the first keypoint, allowing the extraction of this information from the detection result.

[0114] For example, the detection results output by the keypoint detection model may include not only the reference position information of the first keypoint, but also the confidence level of the first keypoint. The confidence level of the first keypoint is used to indicate the degree of confidence of the reference position information of the first keypoint.

[0115] Step 2023: Based on the position transformation relationship between the image to be detected and the first image, convert the reference position information into image position information.

[0116] The image to be detected is acquired based on the first image. The positional transformation relationship between the image to be detected and the first image can be analyzed based on the acquisition process of the image to be detected from the first image. This positional transformation relationship is used to indicate how to convert the positions in the image to be detected into the positions in the first image.

[0117] Since the reference position information is used to indicate the location of the first key point in the image to be detected, and the image position information is used to indicate the location of the first key point in the first image, after determining the position transformation relationship between the image to be detected and the first image, the reference position information can be converted into image position information based on the position transformation relationship.

[0118] For example, after obtaining the image location information of the first key point, a first detection box can be determined based on the position of the first key point in the first image, and the first detection box can be stored in a detection box buffer for subsequent retrieval and utilization. The first detection box refers to the detection box determined based on the first image. For example, the process of determining the first detection box based on the position of the first key point in the first image includes: using the smallest box including the position of the first key point in the first image as the first detection box.

[0119] For example, the keypoint detection process can be as follows: Figure 4 As shown. The process involves: acquiring a first image; determining whether the first image meets the object detection criteria; if the first image meets the object detection criteria, calling the object region detection model to detect the region where the target object is located in the first image, storing the corresponding bounding box in the detection box buffer, and adjusting the corresponding bounding box to obtain the object detection box; if the first image does not meet the object detection criteria, extracting a reference detection box from the detection box buffer, adjusting the reference detection box to obtain the object detection box; after obtaining the object detection box, cropping and scaling the first image based on the object detection box to obtain the image to be detected; calling the keypoint detection model to perform keypoint detection on the features to be detected, obtaining the reference position information and confidence level of the first keypoint, and outputting the detection result.

[0120] Furthermore, after obtaining the reference position information of the first keypoint, the reference position information can be converted into image position information based on the position transformation relationship between the image to be detected and the first image. Based on the position of the first keypoint in the first image indicated by the image position information, a first detection box is determined and stored in a detection box buffer for subsequent retrieval and utilization. In this approach, a more accurate detection box is calculated based on the image position information of the first keypoint detected in each frame, providing a reference for the keypoint detection process in the next frame, thereby obtaining more accurate and stable keypoint detection results.

[0121] It should be noted that the above-described implementation methods for obtaining the image location information of the first key point according to steps 2021 to 2023 are all implementation methods of one embodiment, and the embodiments of this application are not limited thereto. In one possible implementation method, the process of obtaining the image location information of the first key point based on the first image may further include: performing key point detection on the first image to obtain the image location information of the first key point.

[0122] In step 203, feature extraction is performed on the image position information to obtain the pose representation features of the first image.

[0123] After acquiring the image location information, feature extraction is performed on the image location information to obtain the pose representation features of the first image. It should be noted that, for cases where there are multiple first key points, feature extraction is performed on the image location information of all first key points in step 203. The pose representation features of the first image are features used to identify the three-dimensional pose of the target object. In this embodiment, the representation form of the pose representation features is not limited; it can be a vector, a matrix, etc.

[0124] In one possible implementation, the first image of the target object is the image located in the frame following the second image of the target object in the video of the target object. The process of extracting the positional features of the image to obtain the pose representation features of the first image includes the following steps 2031 to 2034.

[0125] Step 2031: Encode the image location information to obtain the first encoded feature of the first image.

[0126] The first encoded feature of the first image is a feature obtained by encoding the image location information. For example, the process of encoding the image location information to obtain the first encoded feature of the first image can be implemented by running a pre-written computer program or by calling a model; this application embodiment does not limit this.

[0127] Step 2032: Fuse the first coding feature of the first image with the fusion feature of the second image to obtain the fusion feature of the first image.

[0128] The second image is the image located in the frame preceding the first image in the video of the target object. That is, the fusion feature of the first image is obtained by fusing the first encoded feature of the first image with the fusion feature of the previous frame image. For example, the first encoded feature of the first image and the fusion feature of the second image have the same dimension to facilitate fusion.

[0129] This application does not limit the method of fusing two features. For example, if both features are represented by matrices, fusing the two features can mean adding the matrix elements at corresponding positions in the two matrices, or it can mean averaging the matrix elements at corresponding positions in the two matrices.

[0130] For example, the process of fusing the first encoded features of the first image with the fusion features of the second image to obtain the fusion features of the first image can be implemented by running a pre-written computer program or by calling a model. This application embodiment does not limit this.

[0131] Before proceeding to step 2032, it is necessary to obtain the fusion features of the second image. The second image may be the first frame of the target object's video, or it may not be the first frame of the target object's video. In different cases, the method for obtaining the fusion features of the second image will differ.

[0132] In an exemplary embodiment, if the second image is not the first frame image in the video of the target object, the principle for obtaining the fusion features of the second image is the same as the principle for obtaining the fusion features of the first image. That is, the first encoded features of the second image are first obtained, and then the first encoded features of the second image are fused with the fusion features of the previous frame image to obtain the fusion features of the second image. The principle for obtaining the first encoded features of the second image is the same as the principle for obtaining the first encoded features of the first image, and will not be repeated here.

[0133] In an exemplary embodiment, if the second image is the first frame image in the video of the target object, the method for obtaining the fusion features of the second image can be: obtaining the first encoding features of the second image and using the first encoding features of the second image as the fusion features of the second image.

[0134] For example, after obtaining the fusion features of the first image, the fusion features of the first image can be output so that they can be fused with the first coded features of the next frame image during the process of obtaining the pose representation features of the next frame image. The next frame image refers to the image in the video of the target object located in the frame following the first image.

[0135] Step 2033: Encode the fusion features of the first image to obtain the second encoded features of the first image.

[0136] The second encoded feature of the first image is a feature obtained by encoding the fusion features of the first image. For example, the process of encoding the fusion features of the first image to obtain the second encoded feature of the first image can be implemented by running a pre-written computer program or by calling a model; this application embodiment does not limit this.

[0137] Step 2034: Fuse the second coding features of the first image with the pose representation features of the second image to obtain the pose representation features of the first image.

[0138] The second image is the image in the video of the target object that is located in the frame preceding the first image. That is, the pose representation features of the first image are obtained by fusing the second encoded features of the first image with the pose representation features of the previous frame. For example, the second encoded features of the first image and the pose representation features of the second image have the same dimension to facilitate fusion.

[0139] For example, the process of fusing the second coded features of the first image with the pose representation features of the second image to obtain the pose representation features of the first image can be implemented by running a pre-written computer program or by calling a model. This application embodiment does not limit this.

[0140] Before proceeding to step 2034, it is necessary to obtain the pose representation features of the second image. The second image may be the first frame of the video of the target object, or it may not be the first frame of the video of the target object. In different cases, the method for obtaining the pose representation features of the second image will be different.

[0141] In an exemplary embodiment, if the second image is not the first frame image in the video of the target object, the principle for obtaining the pose representation features of the second image is the same as the principle for obtaining the pose representation features of the first image. That is, the second coded features of the second image are first obtained, and then the second coded features of the second image are fused with the pose representation features of the previous frame image to obtain the pose representation features of the second image. The principle for obtaining the second coded features of the second image is the same as the principle for obtaining the second coded features of the first image, and will not be repeated here.

[0142] In an exemplary embodiment, if the second image is the first frame image in the video of the target object, the way to obtain the pose representation features of the second image can be: obtaining the second encoding features of the second image and using the second encoding features of the second image as the pose representation features of the second image.

[0143] For example, after obtaining the pose representation features of the first image, the pose representation features of the first image can be output so that they can be fused with the second encoded features of the next frame image during the process of obtaining the pose representation features of the next frame image. The next frame image refers to the image in the video of the target object that is located after the first image.

[0144] It should be noted that the process of extracting features from the image position features according to the above steps 2031 to 2034 to obtain the pose representation features of the first image is only an exemplary description of the case where the first image is an image located in the frame after the second image of the target object in the video of the target object, and the embodiments of this application are not limited thereto.

[0145] In an exemplary embodiment, the first image may also be the first frame image in the video of the target object, i.e., there is no second image. In this case, the implementation of extracting features from the image position features to obtain the pose representation features of the first image may also be as follows: encoding the image position information to obtain the first encoded features of the first image; using the first encoded features of the first image as the fusion features of the first image; encoding the fusion features of the first image to obtain the second encoded features of the first image; and using the second encoded features of the first image as the pose representation features of the first image.

[0146] In an exemplary embodiment, the method of extracting features from image position features to obtain pose representation features of the first image can also be as follows: encoding the image position information and using the encoded features as pose representation features of the first image. In an exemplary embodiment, the method of extracting features from image position features to obtain pose representation features of the first image can also be as follows: encoding the image position features to obtain first encoded features of the first image; encoding the first encoded features of the first image, and using the encoded features as pose representation features of the first image.

[0147] In step 204, the three-dimensional pose of the target object is identified based on the pose representation features of the first image, and the three-dimensional pose parameters of the target are obtained.

[0148] After acquiring the pose representation features of the first image, the three-dimensional pose of the target object is identified based on the pose representation features of the first image to obtain target three-dimensional pose parameters used to indicate the three-dimensional pose of the target object. This application embodiment does not limit the content of the target three-dimensional pose parameters, as long as they can uniquely indicate a three-dimensional pose of the target object.

[0149] For example, the target's three-dimensional pose parameters include the target's three-dimensional position information of the second keypoints. The three-dimensional position information of the second keypoints is used to indicate the location of the second keypoints in three-dimensional space. The second keypoints are keypoints used to describe the three-dimensional pose of the target object. There are correlations between the second keypoints, and the three-dimensional pose of the target object can be located based on the location of the second keypoints in three-dimensional space and the correlations between the second keypoints.

[0150] The number of second key points and the relationships between them are set based on experience or flexibly adjusted according to the application scenario; this application embodiment does not limit this. For example, taking a human as the target object type, such as... Figure 5 As shown, there are 24 second key points. These 24 second key points are: the baseline key point (also called the root node) 0, left hip key point 1, right hip key point 2, first vertebra key point 3, left knee key point 4, right knee key point 5, second vertebra key point 6, left ankle key point 7, right ankle key point 8, third vertebra key point 9, left foot key point 10, right foot key point 11, neck key point 12, left neck key point 13, right neck key point 14, chin key point 15, left shoulder key point 16, right shoulder key point 17, left elbow key point 18, right elbow key point 19, left wrist key point 20, right wrist key point 21, left palm key point 22, and right palm key point 23. The relationships between these 24 second key points are as follows: Figure 5 As shown. For example, the second key point can also be referred to as the second joint point.

[0151] For example, Figure 5 The relationships between the various second key points can be viewed as relationships defined based on a 3D human body model. For example, the 3D human body model can refer to an SMPL (A Skinned Multi-Person Linear) model.

[0152] This application does not limit the form of the three-dimensional position information, as long as it can indicate the location of the second key point in three-dimensional space. For example, the form of the three-dimensional position information can be three-dimensional coordinates, vectors, or matrices. It should be noted that the number of second key points may be one or more, and the three-dimensional position information of the second key point refers to the three-dimensional position information of each individual second key point. For example, three-dimensional space refers to three-dimensional space in the world coordinate system; that is, the three-dimensional position information is information determined based on the world coordinate system.

[0153] It should be noted that the second key point may be exactly the same as the first key point, or it may be partially the same as the first key point. This application does not limit this.

[0154] For example, the target's three-dimensional pose parameters include target reference position information and target rotation parameters of the second keypoint. The target reference position information is used to indicate the target reference position, which is the position on which the position of the second keypoint in three-dimensional space is determined based on the rotation parameters of the second keypoint. For example, the target reference position is the position of the reference keypoint in three-dimensional space.

[0155] The target rotation parameter of a second keypoint refers to the target rotation parameter of each second keypoint. The target rotation parameter of any second keypoint indicates the rotation of that second keypoint relative to its reference state in the 3D pose of the target object. The reference state of any second keypoint refers to the state of that second keypoint relative to its associated keypoints in the reference 3D pose. The reference 3D pose is set empirically; for example, the reference 3D pose is... Figure 5 The posture shown.

[0156] For example, the movement of any associated keypoint of a second keypoint will cause the movement of that second keypoint itself. The associated keypoint of any second keypoint can also be called the parent keypoint of that second keypoint. For example, in... Figure 5 Among the various second key points shown, the key points associated with the left hip key point 1, the right hip key point 2, and the first vertebra key point 3 are all the reference key point 0, and the key point associated with the left knee key point 4 is the left hip key point 1.

[0157] For example, the target rotation parameters of any second key point can be represented in the form of Euler angles, rotation matrix, rotation vector, quaternion, etc., and the embodiments of this application do not limit this.

[0158] Based on the target reference position information and the target rotation parameters of the second keypoint, the position of the second keypoint in three-dimensional space can be calculated using forward kinematics. Based on the position of the second keypoint in three-dimensional space and the relationships between the second keypoints, the three-dimensional pose of the target object can be determined. Forward kinematics, a term in robotics, specifically refers to calculating the position and pose of the robot's end effector based on given joint variables (generally the relative motion of adjacent joints).

[0159] In an exemplary embodiment, steps 203 and 204 can be implemented by running a pre-edited computer program or by calling a target pose recognition model. This application embodiment does not limit this implementation; however, it uses the example of steps 203 and 204 being implemented by calling a target pose recognition model for illustration. That is, the target pose recognition model is called to extract features from the image position information to obtain the pose representation features of the first image; the target pose recognition model is then called to identify the three-dimensional pose of the target object based on the pose representation features of the first image, thus obtaining the target's three-dimensional pose parameters.

[0160] Image position information is input into the target pose recognition model. The model can extract features from the image position information and, based on the extracted pose representation features, identify the 3D pose of the target object, outputting the target's 3D pose parameters. This application does not limit the model structure of the target pose recognition model, as long as it has the function of extracting features from image position information and identifying the 3D pose of the object based on the pose representation parameters. The target pose recognition model is trained using supervised training; the training process is detailed in [link to documentation]. Figure 13 The embodiments shown are not described in detail here.

[0161] In one possible implementation, the target pose recognition model includes a target feature extraction model and a target recognition model. In this case, calling the target pose recognition model to extract features from the image position information to obtain the pose representation features of the first image means calling the target feature extraction model within the target pose recognition model to extract features from the image position information to obtain the pose representation features of the first image. Calling the target pose recognition model to recognize the 3D pose of the target object based on the pose representation features of the first image to obtain the target 3D pose parameters means calling the target recognition model within the target pose recognition model to recognize the 3D pose of the target object based on the pose representation features of the first image to obtain the target 3D pose parameters.

[0162] For example, the process of calling the target feature extraction model in the target pose recognition model to extract features from the image position information and obtain the pose representation features of the first image is an internal processing process of the target feature extraction model, which is related to the model structure of the target feature extraction model. This application embodiment does not limit this.

[0163] For example, the target feature extraction model includes a first target encoding sub-model, a first target fusion sub-model, a second target encoding sub-model, and a second target fusion sub-model. Steps 2031 to 2034 described above can be achieved by calling these four sub-models. In other words, the process of using the target feature extraction model to extract features from image position information and obtain the pose representation features of the first image includes: calling the first target encoding sub-model to encode the image position information to obtain the first encoded features of the first image; calling the first target fusion sub-model to fuse the first encoded features of the first image with the fusion features of the second image to obtain the fusion features of the first image; calling the second target encoding sub-model to encode the fusion features of the first image to obtain the second encoded features of the first image; and calling the second target fusion sub-model to fuse the second encoded features of the first image with the pose representation features of the second image to obtain the pose representation features of the first image.

[0164] For example, the first target encoding sub-model can also be called the target pre-processing network, the first target fusion sub-model can also be called the target pre-processing fusion network, the second target encoding sub-model can also be called the target post-processing network, and the second target fusion sub-model can also be called the target post-processing fusion network. For example, the fusion features can also be called pre-processing features, and the pose representation features can also be called post-processing features.

[0165] For example, the process of calling the target recognition model in the target pose recognition model to identify the three-dimensional pose of the target object based on the pose representation features of the first image and obtaining the target's three-dimensional pose parameters is an internal processing process of the target recognition model, which is related to the model structure of the target recognition model and the content included in the target's three-dimensional pose parameters. This application embodiment does not limit this.

[0166] For example, the target's three-dimensional pose parameters include target reference position information and rotation parameters of the second keypoint. The target recognition model includes a target reference position prediction sub-model and a target rotation prediction sub-model. The process of calling the target recognition model in the target pose recognition model to identify the three-dimensional pose of the target object based on the pose representation features of the first image and obtaining the target's three-dimensional pose parameters includes: calling the target reference position prediction sub-model to predict the target reference position information based on the pose representation features of the first image; and calling the target rotation prediction sub-model to predict the rotation parameters of the second keypoint based on the pose representation features of the first image.

[0167] In an exemplary embodiment, after acquiring the pose representation features of the first image, in addition to identifying the 3D pose of the target object based on the pose representation features of the first image, target projection parameters can also be predicted based on the pose representation features of the first image. Target projection parameters refer to the parameters required to project the object to be projected in 3D space onto the target image. The object to be projected in 3D space can refer to a position in 3D space, or it can refer to a special effects model in 3D space, etc., and this application embodiment does not limit this. This application embodiment does not limit the representation method of the target projection parameters; it can be a matrix, vector, etc.

[0168] For example, the projection process is implemented through a camera, and the projection parameters can also be called camera parameters. For example, camera parameters mainly refer to the extrinsic parameters of the camera. For example, camera parameters can include at least one of translation parameters and rotation parameters. Both translation and rotation parameters are parameters specific to the camera used for projection; that is, translation parameters indicate the translation of the camera in three-dimensional space, and rotation parameters indicate the rotation of the camera in three-dimensional space.

[0169] In an exemplary embodiment, the process of predicting target projection parameters based on pose representation features of the first image can be implemented by running a pre-written computer program or by calling a target parameter prediction model; this embodiment does not limit the implementation in this way. This embodiment illustrates the process of predicting target projection parameters based on pose representation features of the first image by calling a target parameter prediction model, that is, calling the target parameter prediction model to predict the target projection parameters based on the pose representation features of the first image. The target parameter prediction model is trained using supervised training; the process of training the target parameter prediction model is detailed in [link to relevant documentation]. Figure 13 The embodiments shown are not described in detail here.

[0170] For example, taking the t-th frame (t is an integer greater than 1 and not greater than the total number of frames in the video of the target object) in a video with the first image as the target object as an example, the pose recognition process can be as follows: Figure 6 As shown. The image location information of the first key point is input into the target first encoding sub-model. The target first encoding sub-model encodes the image location information to obtain the first encoded feature of the first image and sends the first encoded feature of the first image into the target first fusion sub-model. The target first fusion sub-model combines the first encoded feature of the first image with the fusion feature of the second image. The images are fused to obtain the fused features of the first image. The data is then fed into the target second encoding sub-model. Additionally, the target first fusion sub-model outputs the fusion features of the first image. So that it can be used in the process of pose recognition based on the next frame image.

[0171] The fusion features of the target second coding sub-model on the first image Encoding is performed to obtain the second encoded features of the first image, and these features are then fed into the target second fusion sub-model. The target second fusion sub-model combines the second encoded features of the first image with the pose representation features of the second image. The images are fused to obtain the pose representation features of the first image. The data are then fed into the target reference position prediction sub-model, the target rotation prediction sub-model, and the target parameter prediction model, respectively. In addition, the target second fusion sub-model also outputs the pose representation features of the first image. So that it can be used in the process of pose recognition based on the next frame image.

[0172] The target reference position prediction sub-model is based on the pose representation features of the first image. The target reference position information is predicted, and the target rotation prediction sub-model is based on the pose representation features of the first image. The target rotation parameters of the second keypoint are predicted, and the target parameter prediction model is based on the pose representation features of the first image. Predict the target projection parameters. For example, the target reference position information, the target rotation parameters of the second key point, and the target projection parameters can constitute the output result.

[0173] For example, the fusion features of the second image Pose representation features of the second image Features from the previous frame; fused features from the first image. pose representation features of the first image Features for the next frame of the image.

[0174] In an exemplary embodiment, after obtaining the target's three-dimensional pose parameters and target projection parameters, a special effects image corresponding to the first image can be obtained based on the target's three-dimensional pose parameters and target projection parameters. The special effects image corresponding to the first image is obtained by projecting a special effects model in three-dimensional space onto the first image. The pose of the special effects model in three-dimensional space matches the three-dimensional pose indicated by the three-dimensional pose parameters. The process of projecting the special effects model in three-dimensional space onto the first image is implemented based on the projection parameters.

[0175] In one possible implementation, the process of obtaining the special effects image corresponding to the first image based on the target's three-dimensional pose parameters and target projection parameters includes the following steps 1 to 3.

[0176] Step 1: Based on the target's 3D pose parameters, determine the location of the second key point in 3D space. The second key point is the key point used to describe the 3D pose of the target object.

[0177] The target's three-dimensional pose parameters are used to indicate the target object's three-dimensional pose. The target object's three-dimensional pose can be located using the position of the second keypoint in three-dimensional space and the relationship between the second keypoints. Based on the target's three-dimensional pose parameters, the position of the second keypoint in three-dimensional space can be determined.

[0178] Depending on the content included in the target's three-dimensional attitude parameters, the method for determining the position of the second keypoint in three-dimensional space based on these parameters varies. For example, if the target's three-dimensional attitude parameters include the target's three-dimensional position information of the second keypoint, the method for determining the position of the second keypoint in three-dimensional space based on these parameters includes using the position indicated by the target's three-dimensional position information as the position of the second keypoint in three-dimensional space.

[0179] For example, the target's three-dimensional pose parameters include target reference position information and target rotation parameters of the second keypoint. The method for determining the position of the second keypoint in three-dimensional space based on the target's three-dimensional pose parameters includes: calculating the position of the second keypoint in three-dimensional space based on forward kinematics, using the target reference position information and the target rotation parameters of the second keypoint. The target reference position information is used to indicate the position of the reference keypoint within the second keypoint in three-dimensional space.

[0180] In forward kinematics, given the target rotation parameters of the second keypoint and the position of the reference keypoint in three-dimensional space, the position of the second keypoint in three-dimensional space can be calculated based on the correlation between the second keypoints. For example, taking a second keypoint comprising keypoint A, keypoint B, and keypoint C, where keypoint A is the reference keypoint, keypoint B's associated keypoint is keypoint A, and keypoint C's associated keypoint is keypoint B, the process of calculating the position of the second keypoint in three-dimensional space based on the target reference position information and the target rotation parameters of the second keypoints, according to forward kinematics, includes: calculating the position of keypoint B in three-dimensional space based on the position of keypoint A and the target rotation parameters of keypoint B; and then calculating the position of keypoint C in three-dimensional space based on the position of keypoint B and the target rotation parameters of keypoint C.

[0181] Step 2: Based on the target projection parameters, project the position of the second key point in three-dimensional space onto the first image to obtain the target projection position information of the second key point.

[0182] Since the target projection parameters indicate the projection method for projecting the object to be projected in three-dimensional space onto the first image, after determining the position of the second keypoint in three-dimensional space, the position of the second keypoint in three-dimensional space can be projected onto the first image based on the projection method indicated by the target projection parameters. After projection, the projected position of the second keypoint in the first image can be obtained, and the information indicating the projected position of the second keypoint in the first image is used as the target projection position information of the second keypoint. In other words, the target projection position information of the second keypoint is used to indicate the projected position of the second keypoint in the first image.

[0183] For example, the target projection position information of the second key point and the image position information of the first key point are represented in the same way to facilitate comparison. For example, both are represented in the form of two-dimensional coordinates.

[0184] Step 3: Obtain the special effects image corresponding to the first image based on the target projection position information.

[0185] In one possible implementation, the process of obtaining the special effects image corresponding to the first image based on the target projection position information is as follows: extract the projection position information of the third key point from the target projection position information, extract the image position information of the third key point from the image position information, determine the first correction loss based on the projection position information and the image position information of the third key point, and obtain the special effects image corresponding to the first image based on the first correction loss.

[0186] The third key point is a key point used to calculate the first correction loss from the key points shared by the first and second key points. Which or more of the shared key points are used as the third key point is determined empirically, or flexibly adjusted according to the type of parameters used for correction with the first correction loss as needed; this application's embodiments limit this. For example, all shared key points are used as the third key point; for example, a first number of key points selected from the shared key points are used as the third key point.

[0187] For example, taking the first key point as Figure 3 The key point shown is the second key point. Figure 5Taking the shown key points as an example, the first and second key points together contain 14 key points. These 14 key points are: left shoulder key point, right shoulder key point, left elbow key point, right elbow key point, left wrist key point, right wrist key point, left knee key point, right knee key point, left ankle key point, right ankle key point, left hip key point, right hip key point, left palm key point, and right palm key point. All 14 key points can be used as the third key point, or one or more key points can be selected from these 14 key points.

[0188] Since the third key point is a key point present in both the first and second key points, its projection position information can be extracted from the target projection position information of the second key point, and its image position information can also be extracted from the image position information of the first key point. After extracting the projection and image position information of the third key point, a first correction loss is determined based on these two information.

[0189] The first correction loss is used to measure the difference between the projected position information and the image position information of the third key point. This application does not limit the method by which the first correction loss is determined based on the projected position information and the image position information of the third key point.

[0190] For example, the Euclidean distance between the position indicated by the projected position information of the third keypoint and the position indicated by the image position information of the third keypoint is used as the first correction loss. In this way, the first correction loss can be calculated based on Equation 1:

[0191]

[0192] Among them, L proj This represents the first correction loss; N (where N is an integer not less than 1) represents the number of third keypoints; This represents the image location information of the i-th (i is an integer not less than 1 and not greater than N) third keypoint. This represents the projection position information of the i-th third key point.

[0193] For example, the mean square error loss between the projected position information of the third key point and the image position information of the third key point can also be used as the first correction loss, etc.

[0194] In one possible implementation, the process of obtaining the special effects image corresponding to the first image based on the first correction loss may include the following steps 3-1 and 3-2.

[0195] Step 3-1: Based on the first correction loss, correct at least one of the target projection parameters and the target three-dimensional attitude parameters to obtain at least one of the corrected projection parameters and the corrected three-dimensional attitude parameters.

[0196] After obtaining the first correction loss, the target projection parameters are corrected separately based on the first correction loss to obtain the corrected projection parameters; or, the target three-dimensional attitude parameters are corrected separately based on the first correction loss to obtain the corrected three-dimensional attitude parameters; or, the target projection parameters and the target three-dimensional attitude parameters are corrected based on the first correction loss to obtain the corrected projection parameters and the corrected three-dimensional attitude parameters.

[0197] In one possible implementation, the process of correcting the target projection parameters individually based on the first correction loss to obtain the corrected projection parameters includes: updating the target projection parameters using the first correction loss to obtain the updated projection parameters; then obtaining a new first correction loss based on the updated projection parameters and the target's three-dimensional attitude parameters, updating the updated projection parameters using the new first correction loss to obtain the updated projection parameters, and so on, until the first update termination condition is met, and using the projection parameters obtained when the first update termination condition is met as the corrected projection parameters.

[0198] For example, satisfying the first update termination condition includes, but is not limited to: the number of times the projection parameters are updated reaches the first number threshold, the first correction loss converges, and the first correction loss is less than the first loss threshold.

[0199] In one possible implementation, the process of individually correcting the target's three-dimensional attitude parameters based on the first correction loss to obtain the corrected three-dimensional attitude parameters includes: updating the target's three-dimensional attitude parameters using the first correction loss to obtain the updated three-dimensional attitude parameters; then obtaining a new first correction loss based on the updated three-dimensional attitude parameters and the target projection parameters, updating the updated three-dimensional attitude parameters using the new first correction loss to obtain the updated three-dimensional attitude parameters, and so on, until the second update termination condition is met, and using the three-dimensional attitude parameters obtained when the second update termination condition is met as the corrected three-dimensional attitude parameters.

[0200] For example, satisfying the second update termination condition includes, but is not limited to: the number of times the three-dimensional pose parameters are updated reaches the second threshold, the first correction loss converges, and the first correction loss is less than the second loss threshold.

[0201] It should be noted that, for cases where the target's three-dimensional attitude parameters include the target's reference position information and the target's rotation parameters at the second key point, updating the target's three-dimensional attitude parameters may refer to updating the target's reference position information and the target's rotation parameters at the second key point, or it may refer to updating only the target's rotation parameters at the second key point or updating the target's reference position information. This application does not limit this aspect.

[0202] In one possible implementation, the target projection parameters and target three-dimensional attitude parameters are corrected based on the first correction loss in the following ways, including but not limited to two: simultaneously correcting the target projection parameters and target three-dimensional attitude parameters based on the first correction loss; and alternately correcting the target projection parameters and target three-dimensional attitude parameters based on the first correction loss.

[0203] For example, when simultaneously correcting the target projection parameters and the target 3D attitude parameters based on the first correction loss, the process of correcting the target projection parameters and the target 3D attitude parameters based on the first correction loss to obtain the corrected projection parameters and the corrected 3D attitude parameters includes: updating the target projection parameters and the target 3D attitude parameters using the first correction loss to obtain the updated projection parameters and the updated 3D attitude parameters; then obtaining a new first correction loss based on the updated projection parameters and the updated 3D attitude parameters, and updating the updated projection parameters and the updated 3D attitude parameters using the new first correction loss to obtain the updated projection parameters and the updated 3D attitude parameters; and so on, until the target update termination condition is met. The projection parameters obtained when the target update termination condition is met are taken as the corrected projection parameters, and the 3D attitude parameters obtained when the target update termination condition is met are taken as the corrected 3D attitude parameters.

[0204] For example, the target update termination conditions include, but are not limited to: the number of times the projection parameters and 3D pose parameters are updated reaches the target number threshold, the first correction loss converges, and the first correction loss is less than the loss threshold.

[0205] For example, when alternating corrections are performed on the target projection parameters and target 3D attitude parameters based on the first correction loss, the process of obtaining the corrected projection parameters and corrected 3D attitude parameters is related to the order and number of alternating corrections. The order and number of alternating corrections are set based on experience or flexibly adjusted according to the application scenario, and this application embodiment does not limit them. It should be noted that correcting the parameters (target projection parameters or target 3D attitude parameters) once means updating the parameters once or multiple times.

[0206] Taking the alternating correction order as first correcting the target projection parameters and then correcting the target three-dimensional attitude parameters, and the alternating correction number as two corrections of the target projection parameters and one correction of the target three-dimensional attitude parameters, the process of correcting the target projection parameters and the target three-dimensional attitude parameters based on the first correction loss includes the following steps 4-1 to 4-3.

[0207] Step 4-1: Update the target projection parameters using the first correction loss to obtain the first projection parameters.

[0208] The first projection parameter refers to the parameter obtained after correcting the target projection parameter once. The first correction loss is the loss used in the first correction of the target projection parameter. For example, in this case, all key points shared by the first key point and the second key point can be used as the third key point. All key points shared by the first key point and the second key point are the 14 key points of the whole body. That is to say, in the projection parameter correction stage, the correction loss can be calculated using the 14 key points of the whole body (left shoulder key point, right shoulder key point, left elbow key point, right elbow key point, left wrist key point, right wrist key point, left knee key point, right knee key point, left ankle key point, right ankle key point, left hip key point, right hip key point, left palm key point, and right palm key point).

[0209] For example, the implementation process of step 4-1 includes: updating the target projection parameters using the first correction loss to obtain the updated projection parameters; obtaining a new first correction loss based on the updated projection parameters and the target three-dimensional attitude parameters; updating the updated projection parameters using the new first correction loss to obtain the updated projection parameters; and so on, until the first correction termination condition is met, and using the projection parameters obtained when the first correction termination condition is met as the first projection parameters.

[0210] For example, the conditions for terminating the first correction include, but are not limited to: the number of times the projection parameters are updated reaches the third threshold, the first correction loss converges, and the first correction loss is less than the third loss threshold.

[0211] Step 4-2: Based on the first projection parameters, calibrate the target's three-dimensional attitude parameters to obtain the corrected three-dimensional attitude parameters.

[0212] In one possible implementation, step 4-2 includes: projecting the position of the second key point in three-dimensional space onto the first image based on the first projection parameters to obtain the first projection position information of the second key point; extracting the projection position information of the fourth key point from the first projection position information; extracting the image position information of the fourth key point from the image position information; determining the second correction loss based on the projection position information and the image position information of the fourth key point; updating the target three-dimensional pose parameters using the second correction loss to obtain the corrected three-dimensional pose parameters.

[0213] The method of projecting the position of the second key point in three-dimensional space onto the first image based on the first projection parameters to obtain the first projection position information of the second key point is described in step 2 above, and will not be repeated here.

[0214] The fourth keypoint is a keypoint shared by the first and second keypoints used to calculate the second correction loss. The second correction loss is the loss used to update the target's three-dimensional attitude parameters. For example, which keypoints(s) shared by the first and second keypoints are used as keypoints for calculating the second correction loss can be set empirically or flexibly adjusted according to the correction process of the target's three-dimensional attitude parameters.

[0215] For example, taking a human as the target object, the correction process for the target's 3D pose parameters might involve first performing preliminary correction based on key points of the legs, and then correcting the preliminary 3D pose parameters based on key points of the arms. In this case, the fourth key point could refer to the key points shared by the first and second key points that are related to the legs, such as the six key points: left knee key point, right knee key point, left ankle key point, right ankle key point, left hip key point, and right hip key point.

[0216] In this case, the process of updating the target's three-dimensional attitude parameters using the second correction loss to obtain the corrected three-dimensional attitude parameters includes: updating the target's three-dimensional attitude parameters using the second correction loss to obtain the initially corrected three-dimensional attitude parameters; determining the initially corrected position of the second keypoint in three-dimensional space based on the initially corrected three-dimensional attitude parameters; projecting the initially corrected position of the second keypoint in three-dimensional space onto the first image based on the first projection parameters to obtain the third projection position information of the second keypoint; extracting the projection position information of the sixth keypoint from the third projection position information, extracting the projection position information of the sixth keypoint from the image position information, determining the fourth correction loss based on the projection position information of the sixth keypoint, and updating the initially corrected three-dimensional attitude parameters using the fourth correction loss to obtain the corrected three-dimensional attitude parameters.

[0217] The sixth key point can refer to the key points related to the arm that are shared by the first and second key points, such as the left shoulder key point, right shoulder key point, left elbow key point, right elbow key point, and left wrist key point.

[0218] For example, updating the target's three-dimensional attitude parameters using the second correction loss to obtain the initially corrected three-dimensional attitude parameters and updating the initially corrected three-dimensional attitude parameters using the fourth correction loss are both iterative processes to obtain the corrected three-dimensional attitude parameters. The iterative principle is the same as that of updating the target's projection parameters using the first correction loss to obtain the first projection parameters, and will not be elaborated here.

[0219] Of course, in some embodiments, the correction process for the target's three-dimensional posture parameters may also involve first performing preliminary correction of the target's three-dimensional posture based on the key points of the arm, and then correcting the preliminary corrected three-dimensional posture parameters based on the key points of the leg; or, the correction process for the target's three-dimensional posture parameters may also involve directly correcting the target's three-dimensional posture based on the key points of the whole body, etc., which will not be elaborated in detail in the embodiments of this application.

[0220] Step 4-3: Correct the first projection parameters based on the corrected 3D pose information to obtain the corrected projection parameters.

[0221] In one possible implementation, step 4-3 includes: determining the corrected position of the second key point in three-dimensional space based on the corrected three-dimensional pose information; projecting the corrected position of the second key point in three-dimensional space onto the first image based on the first projection parameters to obtain the second projection position information of the second key point; extracting the projection position information of the fifth key point from the second projection position information, extracting the image position information of the fifth key point from the image position information, determining the third correction loss based on the projection position information and the image position information of the fifth key point, and updating the first projection parameters using the third correction loss to obtain the corrected projection parameters.

[0222] The second projection position information of the second key point is used to indicate the corrected position of the second key point in three-dimensional space.

[0223] The fifth key point is a key point shared by the first and second key points used to calculate the third correction loss. The third correction loss is the loss used for the first correction of the first projection parameter, i.e., the second correction of the target projection parameter. For example, the fifth key point can be the same as the third key point on which the first correction loss is based; for instance, both the fifth and third key points can be all key points shared by the first and second key points. Of course, the fifth key point can also be different from the third key point, and this embodiment does not limit this.

[0224] The process of updating the first projection parameters using the third correction loss to obtain the corrected projection parameters is described in step 4-1, and will not be repeated here.

[0225] In the process of alternatingly correcting the target projection parameters and the target 3D pose parameters according to steps 4-1 to 4-3, the target 3D pose parameters are first fixed, and the target projection parameters are corrected for the first time. Then, based on the initially corrected projection parameters obtained after the first correction, the target 3D pose parameters are corrected again. For example, the target 3D pose parameters are first initially corrected based on the key points of the legs to optimize the leg pose, and then the initially corrected 3D pose parameters are corrected based on the key points of the arms to optimize the arm pose. Finally, based on the corrected 3D pose parameters, the target projection parameters are corrected a second time to obtain the corrected projection parameters. This alternating correction method helps to improve the stability of the joint correction results.

[0226] For example, the process of alternately correcting the target projection parameters and the target three-dimensional attitude parameters is as follows: Figure 7 As shown. Based on the target's three-dimensional attitude parameters, the position of the second key point in three-dimensional space is determined according to forward kinematics; the position of the second key point in three-dimensional space is projected onto the first image based on the target projection parameters to obtain the target projection position information of the second key point; the first correction loss is calculated based on the target projection position information of the second key point and the image position information of the first key point; the target projection parameters and the target's three-dimensional attitude parameters are alternately corrected based on the first correction loss.

[0227] It should be noted that steps 4-1 to 4-3 described above only illustrate the process of correcting the target projection parameters and the target 3D attitude parameters based on the first correction loss, using the example of correcting the target projection parameters first and then the target 3D attitude parameters, and the number of corrections being twice for the target projection parameters and once for the target 3D attitude parameters. In some embodiments, the order and number of corrections may vary. In other cases, the process of correcting the target projection parameters and the target 3D attitude parameters based on the first correction loss to obtain the corrected projection parameters and the corrected 3D attitude parameters will be different and will not be elaborated here.

[0228] Step 3-2: In response to obtaining the corrected 3D pose parameters and the corrected projection parameters, obtain the special effects image corresponding to the first image based on the corrected 3D pose parameters and the corrected projection parameters; In response to obtaining the corrected projection parameters, obtain the special effects image corresponding to the first image based on the target 3D pose parameters and the corrected projection parameters; In response to obtaining the corrected 3D pose parameters, obtain the special effects image corresponding to the first image based on the corrected 3D pose parameters and the target projection parameters.

[0229] This application embodiment uses the obtained corrected 3D pose parameters and corrected projection parameters as an example for illustration. In one possible implementation, the process of obtaining the special effects image corresponding to the first image based on the corrected projection parameters and corrected 3D pose parameters includes steps A and B.

[0230] Step A: Based on the corrected 3D pose parameters, determine the target 3D special effects model that matches the target object.

[0231] A 3D special effects model is a model whose posture changes in real time as the 3D posture of an object changes. This application does not limit the type of 3D special effects model. For example, the type of 3D special effects model can be a 3D driving model, a 3D accessory model, etc. For example, if the target object is a human, then the 3D special effects model is a humanoid character model.

[0232] The corrected 3D pose parameters are used to indicate the corrected 3D pose of the target object. The corrected 3D pose of the target object is closer to the real 3D pose of the target object. The target 3D special effects model that matches the target object based on the corrected 3D pose parameters refers to the 3D special effects model whose pose matches the corrected 3D pose of the target object. The target 3D special effects model has a better effect.

[0233] In one possible implementation, the process of determining the target 3D special effects model that matches the target object based on the corrected 3D pose parameters includes: determining the pose parameters of the special effects model based on the corrected 3D pose parameters, wherein the pose parameters of the special effects model are used to indicate the 3D pose of the target 3D special effects model; and taking the 3D special effects model with the 3D pose indicated by the pose parameters of the special effects model as the target 3D special effects model.

[0234] In an exemplary embodiment, the method for determining the special effects model's posture parameters based on the corrected 3D posture parameters is related to the content of the corrected 3D posture parameters. In an exemplary embodiment, the corrected 3D posture parameters include the corrected 3D position information of the second key point. In this case, the method for determining the special effects model's posture parameters based on the corrected 3D posture parameters includes: extracting the corrected 3D position information of the target key point from the corrected 3D position information of the second key point; determining the 3D position information of the special effects model key point corresponding to any target key point based on the corrected 3D position information of any target key point; and using the parameters including the 3D position information of the special effects model key points corresponding to each target key point as the special effects model's posture parameters.

[0235] A target keypoint refers to a second keypoint that corresponds to any keypoint in the candidate effect model keypoints. Candidate effect model keypoints are predefined keypoints used on any 3D effect model. For example, a correspondence between any second keypoint and any effect model keypoint means that the second keypoint and that effect model keypoint represent the same location.

[0236] For example, before determining the target 3D special effects model, a connection can be established between second keypoints and candidate special effects model keypoints. This involves mapping second keypoints representing the same areas to candidate special effects model keypoints, thereby identifying which of the second keypoints(s) are the target keypoints. For example, establishing the connection between second keypoints and candidate special effects model keypoints can be achieved by parsing the relationships between second keypoints and between candidate special effects model keypoints. For example, the relationships between second keypoints can be constructed... Figure 8 The skeletal structure shown in (a) allows us to construct the relationships between key points in the candidate special effects model. Figure 8 The skeletal structure shown in (b) is illustrated. By analyzing the relationships, it is possible to correlate the second keypoints representing the same location with the keypoints of the candidate special effects model, such as... Figure 8 As shown, the second key point representing the left elbow is mapped to the candidate effect key point representing the left elbow.

[0237] The 3D position information of the special effects model key point corresponding to any target key point is used to indicate the position of the special effects model key point corresponding to any target key point in the special effects model space. The special effects model space is the space referenced for constructing the target 3D special effects model. This special effects model space may be a 3D space in the world coordinate system or a 3D space in the target coordinate system. This application does not limit this. The target coordinate system is a coordinate system different from the world coordinate system, such as a local x-axis coordinate system, a local y-axis coordinate system, etc.

[0238] The method for determining the 3D position information of the special effects model key point corresponding to any target key point based on the corrected 3D position information of any target key point is related to the type of special effects model space. For example, if the special effects model space is a 3D space in the world coordinate system, the method for determining the 3D position information of the special effects model key point corresponding to any target key point based on the corrected 3D position information of any target key point includes: using the corrected 3D position information of any target key point as the 3D position information of the special effects model key point corresponding to that target key point.

[0239] For example, if the special effects model space is a three-dimensional space under the target coordinate system, the method of determining the three-dimensional position information of the special effects model key point corresponding to any target key point based on the corrected three-dimensional position information of any target key point includes: converting the corrected three-dimensional position information of any target key point into the three-dimensional position information of the special effects model key point corresponding to any target key point according to the position transformation relationship between the target coordinate system and the world coordinate system.

[0240] By referring to the method of obtaining the three-dimensional position information of the special effects model key points corresponding to any target key point, it is possible to obtain the three-dimensional position information of the special effects model key points corresponding to each target key point, and then use the parameters including the three-dimensional position information of the special effects model key points corresponding to each target key point as the special effects model posture parameters.

[0241] In an exemplary embodiment, the corrected 3D posture parameters include corrected reference position information and corrected rotation parameters of the second key point. In this case, the method for determining the posture parameters of the special effects model based on the corrected 3D posture parameters includes: converting the corrected reference position information into reference position information of the special effects model according to the position transformation relationship between the target coordinate system and the world coordinate system; extracting the corrected rotation parameters of the target key point from the corrected rotation parameters of the second key point; determining the rotation parameters of the special effects model key point corresponding to any target key point based on the corrected rotation parameters of any target key point; and using the parameters including the reference position information of the special effects model and the rotation parameters of the special effects model key points corresponding to each target key point as the posture parameters of the special effects model.

[0242] For example, the rotation parameters are in the form of a matrix. The process of determining the rotation parameters of the special effects model keypoints corresponding to any target keypoint based on the corrected rotation parameters of any target keypoint is implemented based on Formula 2:

[0243]

[0244] Among them, M local The rotation matrix representing the key points of the special effects model corresponding to any target key point; M represents the inverse of the rotation matrix of the keypoints in the effect model corresponding to any target keypoint; SMPL Represents the corrected rotation matrix for any target keypoint; This represents the inverse of the offset matrix. Under the constraint of the association relationships between candidate effect model keypoints, this offset matrix indicates the offset between the effect model keypoint corresponding to any target keypoint and its associated effect model keypoints, representing the state of the effect model keypoint corresponding to any target keypoint relative to its associated effect model keypoints in the baseline effect model pose. The baseline effect model pose is preset, and the offset matrix is ​​generated based on this preset pose.

[0245] By referring to the method of obtaining the rotation parameters of the special effects model key points corresponding to any target key point, it is possible to obtain the rotation parameters of the special effects model key points corresponding to each target key point. Then, the parameters including the reference position information of the special effects model and the rotation parameters of the special effects model key points corresponding to each target key point are used as the posture parameters of the special effects model.

[0246] After obtaining the pose parameters of the special effects model, the 3D special effects model with the 3D pose indicated by the pose parameters is used as the target 3D special effects model. For example, the 3D special effects model with the 3D pose indicated by the pose parameters can be directly rendered and generated based on the pose parameters, or it can be obtained by driving a 3D special effects model with a reference pose based on the pose parameters. This application embodiment does not limit this. For example, driving a 3D special effects model with a reference pose can refer to adjusting the position of key points in the 3D special effects model with the reference pose, or rotating the key points of the special effects model according to the rotation method indicated by the rotation parameters, etc.

[0247] Step B: Based on the corrected projection parameters, project the target 3D special effects model onto the first image to obtain the special effects image corresponding to the first image.

[0248] After acquiring the 3D special effects model, it is projected onto the first image based on the corrected projection parameters. The resulting projected image is then used as the special effects image corresponding to the first image. The special effects image corresponding to the first image not only displays the target object but also the projected model of the 3D special effects model.

[0249] For example, after obtaining the special effects image corresponding to the first image, if the computer device is a server, the server can send the special effects image to the terminal of the interactive object so that the terminal of the interactive object can display the special effects image corresponding to the first image. If the computer device is the terminal of the interactive object, the terminal of the interactive object can directly display the special effects image corresponding to the first image.

[0250] It should be noted that steps A and B above only illustrate the process of obtaining the special effects image corresponding to the first image by taking the obtained corrected 3D pose parameters and corrected projection parameters as examples. The embodiments of this application are not limited to this.

[0251] In one possible implementation, when only the corrected 3D pose parameters are obtained, the special effects image corresponding to the first image is acquired based on the corrected 3D pose parameters and the target projection parameters. This process includes: determining a target 3D special effects model matching the target object based on the corrected 3D pose parameters; and projecting the target 3D special effects model onto the first image based on the target projection parameters to obtain the special effects image corresponding to the first image. The implementation of this process is described in steps A and B above and will not be repeated here.

[0252] In one possible implementation, for cases where only the corrected projection parameters are obtained, the special effects image corresponding to the first image is acquired based on the target's 3D pose parameters and the corrected projection parameters. This process includes: determining a target 3D special effects model matching the target object based on the target's 3D pose parameters; and projecting the target 3D special effects model onto the first image based on the corrected projection parameters to obtain the special effects image corresponding to the first image. The implementation of this process is described in steps A and B above and will not be repeated here.

[0253] It should be noted that steps 1 to 3 above are merely exemplary descriptions of obtaining the special effects image corresponding to the first image based on the target's 3D pose parameters and target projection parameters, and this application embodiment does not limit this. In one possible implementation, the process of obtaining the special effects image corresponding to the first image based on the target's 3D pose parameters and target projection parameters may include: determining a target 3D special effects model matching the target object based on the target's 3D pose parameters; and projecting the target 3D special effects model onto the first image based on the target projection parameters to obtain the special effects image corresponding to the first image. The implementation of this process is described in steps A and B above, and will not be repeated here.

[0254] This application provides a complete workflow for implementing a 3D special effects algorithm, applicable to the entertainment industry. The complete workflow mainly includes three modules: object detection and keypoint detection, pose recognition, and joint correction. The parameters output after joint correction can be used for 3D special effects such as drivers and add-ons. For example, the process of obtaining the special effects image of the first image is as follows: Figure 9 As shown, key point detection is performed on the first image to obtain the image position information of the first key point; pose recognition and parameter prediction are performed based on the image position information to obtain the target reference position information, the rotation parameters of the second key point, and the target projection parameters; the target reference position information, the rotation parameters of the second key point, and the target projection parameters are jointly corrected; based on the parameters obtained after joint correction, a 3D driving model and a 3D pendant model are obtained; the 3D driving model is projected onto the first image to obtain the driving effect image; the 3D pendant model is projected onto the first image to obtain the pendant effect image.

[0255] The technical solution provided in this application employs a lightweight network at each stage of the algorithm, enabling direct identification of 3D pose parameters based on the image position information of the first key point, thus achieving real-time performance. Furthermore, joint correction is proposed to improve the accuracy of the 3D pose parameters. For example, the technical solution provided in this application can be applied to mobile devices, such as video scenarios.

[0256] The technical solutions provided in the embodiments of this application have been quantitatively and qualitatively evaluated to demonstrate the effects of the technical solutions provided in the embodiments of this application compared with related technologies.

[0257] For example, the speed of acquiring special effects images was evaluated. The technical solution provided in this application embodiment takes 7.62ms to acquire a single frame of special effects image on the test terminal, while the related technology takes 11ms to acquire a single frame of special effects image on the test terminal. Compared with the related technology, the technical solution provided in this application embodiment improves the speed by 30%.

[0258] The time distribution for acquiring a single frame of special effects image on a test terminal using the technical solution provided in this application is shown in Table 1. The time consumption at each stage reaches the industry-leading level. It should be noted that the projection parameter smoothing and three-dimensional pose parameter smoothing in Table 1 are performed during the acquisition of the special effects image to reduce noise in the special effects image and improve its quality.

[0259] Table 1

[0260]

[0261] For example, the effect of the joint correction algorithm on special effects images was tested, and the test results are as follows: Figure 10 As shown. Figure 10 (a) and (e) in the figure are visualizations of the keypoint detection results; Figure 10 (b) and (f) in the image are special effects images obtained directly based on the target's 3D pose parameters and target projection parameters; Figure 10 (c) and (g) in the figure are special effects images obtained based on the corrected projection parameters and the target's three-dimensional pose parameters; Figure 10 Images (d) and (h) in China are special effects images obtained based on corrected projection parameters and corrected 3D pose parameters. According to... Figure 10 As can be seen, the target pose recognition model and the target parameter prediction model will output roughly accurate target 3D pose parameters and target projection parameters. Adding correction to the target projection parameters can improve the overall fit of the 3D special effects model, and adding correction to the target 3D pose parameters can improve the fit of limbs such as legs and arms, and the pose accuracy is further improved.

[0262] For example, a comparison was made between a special effects image obtained based on the technical solution provided in the embodiments of this application and a special effects image obtained based on two related technologies. The comparison result of the special effects image obtained based on the technical solution provided in the embodiments of this application and the special effects image obtained based on the first related technology is as follows: Figure 11 As shown, Figure 11 The left image of (1), (2), and (3) in the figure is a special effects image obtained based on the first related technology, and the right image is a special effects image obtained based on the technical solution provided in the embodiments of this application. The comparison results of the special effects image obtained based on the technical solution provided in the embodiments of this application and the special effects image obtained based on the second related technology are as follows: Figure 12 As shown, Figure 12 The left image of (1), (2), and (3) in the figure is a special effects image obtained based on the second related technology, and the right image is a special effects image obtained based on the technical solution provided in the embodiment of this application. Figure 11 and Figure 12It can be seen that the projection model of the three-dimensional special effects model in the special effects image obtained based on the technical solution provided in the embodiments of this application has significantly improved the pose accuracy and the degree of character fit.

[0263] The pose recognition method provided in this application identifies the three-dimensional pose of the target object based on pose representation features. The pose representation features are obtained by extracting features from the image position information of the first key point. In this pose recognition process, it is not necessary to determine the depth of the first key point or the position of the first key point in three-dimensional space. The process is relatively simple, which helps to save computation and thus improve the efficiency of pose recognition.

[0264] Based on the above Figure 1 The implementation environment shown in this application embodiment provides a training method for a pose recognition model. This method is executed by a computer device, which can be a terminal 11 or a server 12; this application embodiment does not limit the specific computer device used. Figure 13 As shown, the training method for the pose recognition model provided in this application embodiment may include the following steps 1301 to 1305.

[0265] In step 1301, the sample image of the sample object and the standard three-dimensional pose parameters corresponding to the sample image are obtained.

[0266] A sample image of a sample object refers to an image obtained by acquiring images of a sample object. For example, a sample object is... Figure 2 In the embodiments shown, the target objects are of the same type to ensure the training effect of the model. For example, the sample objects and the target objects are both human, or the sample objects and the target objects are both animal, etc.

[0267] The standard 3D pose parameters corresponding to the sample images are used to provide supervision information for the training process of the pose recognition model. The standard 3D pose corresponding to the sample images refers to the relatively accurate 3D pose parameters that can be obtained by pose recognition based on the sample images. For example, the standard 3D pose parameters corresponding to the sample images are obtained by technicians through analysis of the sample images.

[0268] For example, the sample image of the sample object and the standard 3D pose parameters corresponding to the sample image can be stored in a database, so that the sample image of the sample object and the standard 3D pose parameters corresponding to the sample image can be extracted from the database.

[0269] For example, a sample image is first acquired, and then sent to a technician. The technician then provides feedback on the standard 3D pose parameters corresponding to the sample image. For example, methods for acquiring the sample image include, but are not limited to: extracting the sample image from an image library; receiving a sample image acquired by an image acquisition device; or receiving a sample image uploaded by an interactive object.

[0270] It should be noted that the sample images obtained in step 1301 are the images used to obtain the target loss. The number of sample images may be one or more, and this embodiment does not limit this. Each sample image corresponds to standard three-dimensional pose parameters. A loss can be obtained based on each sample image and its corresponding standard three-dimensional pose parameters. When there are multiple sample images, the target loss is the sum or average of the losses obtained from each sample image. This embodiment uses one sample image as an example for explanation.

[0271] In step 1302, the sample image position information of the first sample key point is obtained based on the sample image. The sample image position information is used to indicate the position of the first sample key point in the sample image. The first sample key point is a key point used to describe the two-dimensional pose of the sample object.

[0272] The sample image location information of the first sample key point refers to the information used to indicate the position of the first sample key point in the sample image. In this application embodiment, the form of the sample image location information is not limited, as long as it can indicate the position of the first sample key point in the sample image. For example, the form of the sample image location information is two-dimensional coordinates, or a vector, or a matrix, etc.

[0273] It should be noted that the number of first sample keypoints may be one or multiple. If there are multiple first sample keypoints, it is necessary to obtain the sample image location information for each first sample keypoint. For example, the first sample keypoint is... Figure 2 In the illustrated embodiments, the first keypoints are keypoints with the same semantics to ensure the effectiveness of model training. For example, having the same semantics can refer to representing the same part.

[0274] The implementation process of obtaining the sample image location information of the first sample key points based on the sample image is described in [reference]. Figure 2 Step 202 in the illustrated embodiment will not be repeated here.

[0275] In step 1303, the initial pose recognition model is called to extract features from the position information of the sample image to obtain the sample pose representation features. Based on the sample pose representation features, the three-dimensional pose of the sample object is recognized to obtain the sample three-dimensional pose parameters.

[0276] The initial pose recognition model refers to the pose recognition model to be trained. The sample pose representation features refer to the features extracted from the position information of the sample image to represent the three-dimensional pose of the sample object. The sample three-dimensional pose parameters are the parameters used to indicate the three-dimensional pose of the sample object.

[0277] The initial pose recognition model is invoked to extract features from the positional information of the sample image, resulting in sample pose representation features. Based on these features, the 3D pose of the sample object is identified, and the implementation principle of obtaining the sample's 3D pose parameters is as follows: Figure 2 In the illustrated embodiment, the target pose recognition model is called to extract features from the image position information to obtain the pose representation features of the first image. The target recognition model is then called to identify the three-dimensional pose of the target object based on the pose representation features to obtain the three-dimensional pose parameters. The implementation principle is the same and will not be repeated here.

[0278] For example, the initial pose recognition model includes an initial feature extraction model and an initial recognition model. In this case, the initial pose recognition model is called to extract features from the position information of the sample image to obtain sample pose representation features. Based on the sample pose representation features, the three-dimensional pose of the sample object is recognized to obtain the sample three-dimensional pose parameters. The implementation process includes: calling the initial feature extraction model to extract features from the position information of the sample image to obtain sample pose representation features; calling the initial recognition model to recognize the three-dimensional pose of the sample object based on the sample pose representation features to obtain the sample three-dimensional pose parameters.

[0279] For example, the initial feature extraction model includes an initial first encoding sub-model, an initial first fusion sub-model, an initial second encoding sub-model, and an initial second fusion sub-model. In this case, the process of calling the initial feature extraction model to extract features from the position information of the sample image and obtain the sample pose representation features may include: calling the initial first encoding sub-model to encode the position information of the sample image to obtain the first encoded features of the sample image; calling the initial first fusion sub-model to fuse the first encoded features with the first reference features to obtain the fused features of the sample image; calling the initial second encoding sub-model to encode the fused features to obtain the second encoded features of the sample image; and calling the initial second fusion sub-model to fuse the second encoded features with the second reference features to obtain the sample pose representation features.

[0280] The first and second reference features are set based on experience or flexibly adjusted according to the application scenario, and this application embodiment does not limit this. For example, the dimension of the first reference feature is the same as the dimension of the first coded feature, and the dimension of the second reference feature is the same as the dimension of the second coded feature, so as to ensure that the fusion between features can be achieved.

[0281] For example, taking the sample's three-dimensional pose parameters as including the sample's reference position information and the rotation parameters of the second sample's key points as an example, the initial recognition model may also include an initial reference position prediction sub-model and an initial rotation prediction sub-model. In this case, the process of calling the initial recognition model to identify the three-dimensional pose of the sample object based on the sample pose characterization features and obtaining the sample's three-dimensional pose parameters includes: calling the initial reference position prediction sub-model to predict the sample's reference position information based on the sample pose characterization features; and calling the initial rotation prediction sub-model to predict the rotation parameters of the second sample's key points based on the sample pose characterization features.

[0282] In step 1304, the target loss is obtained based on the standard three-dimensional attitude parameters and the sample three-dimensional attitude parameters.

[0283] Standard 3D pose parameters refer to the actual 3D pose parameters of the sample object, which can provide supervision information for model training. After obtaining the sample 3D pose parameters, the target loss can be obtained based on the sample 3D pose parameters and the standard 3D pose parameters, so as to update the model parameters of the initial pose recognition model using the target loss.

[0284] In an exemplary embodiment, obtaining the target loss based on the sample 3D attitude parameters and the standard 3D attitude parameters can refer to directly obtaining the target loss based on the sample 3D attitude parameters and the standard 3D attitude parameters. In this case, the target loss is used to indicate the difference between the sample 3D attitude parameters and the standard 3D attitude parameters. This application embodiment does not limit the implementation method of directly obtaining the target loss based on the sample 3D attitude parameters and the standard 3D attitude parameters. For example, the cross-entropy loss between the sample 3D attitude parameters and the standard 3D attitude parameters can be used as the target loss; or, the mean squared error loss between the sample 3D attitude parameters and the standard 3D attitude parameters can be used as the target loss; or, the Euclidean distance between the sample 3D attitude parameters and the standard 3D attitude parameters can be used as the target loss, etc.

[0285] It should be noted that the types of content included in the sample 3D pose parameters are the same as those included in the standard 3D pose parameters. For example, the sample 3D pose parameters include the sample 3D position information of the second sample keypoints, and the standard 3D pose parameters include the standard 3D position information of the second sample keypoints. In this case, the loss between the sample 3D pose parameters and the standard 3D pose parameters is the sum of the losses between the sample 3D position information and the standard 3D position information for each second sample keypoint.

[0286] For example, the sample 3D pose parameters include sample reference position information and sample rotation parameters of the second sample keypoints, while the standard 3D pose parameters include standard reference position information and standard rotation parameters of the second sample keypoints. In this case, the loss between the sample 3D pose parameters and the standard 3D pose parameters is the sum of position loss and rotation loss, where position loss refers to the loss between the sample reference position information and the standard reference position information, and rotation loss refers to the sum of the loss between the sample rotation parameters and the standard rotation parameters of each second sample keypoint.

[0287] In one possible implementation, the process of obtaining the target loss based on standard 3D pose parameters and sample 3D pose parameters includes: calling the initial parameter prediction model to predict the sample projection parameters based on the sample pose representation features; and obtaining the target loss based on the standard 3D pose parameters, sample 3D pose parameters, and sample projection parameters.

[0288] The initial parameter prediction model is a model that predicts the projection parameters used for projection based on pose representation features. The initial parameter prediction model is input into the sample pose representation features, and it outputs the sample projection parameters predicted based on these features. Sample projection parameters refer to the parameters required to project the object to be projected in 3D space onto the sample image.

[0289] When the sample projection parameters are obtained, the target loss can be obtained by comprehensively considering the standard 3D pose parameters, the sample 3D pose parameters, and the sample projection parameters, so as to improve the training effect of the model.

[0290] In one possible implementation, the process of obtaining the target loss based on standard three-dimensional pose parameters, sample three-dimensional pose parameters, and sample projection parameters includes the following steps 13041 to 13045.

[0291] Step 13041: Determine the first sub-loss based on the standard 3D pose parameters and the sample 3D pose parameters.

[0292] The implementation method for determining the first sub-loss based on standard 3D pose parameters and sample 3D pose parameters is described in the section on determining the target loss based directly on standard 3D pose parameters and sample 3D pose parameters, and will not be repeated here.

[0293] Step 13042: Based on the sample's three-dimensional pose parameters, determine the location of the second sample key point in three-dimensional space. The second sample key point is the key point used to describe the three-dimensional pose of the sample object.

[0294] Step 13043: Based on the sample projection parameters, project the position of the second sample key point in three-dimensional space onto the sample image to obtain the sample projection position information of the second sample key point.

[0295] The sample projection position information of the second sample key point is used to indicate the projection position of the second sample key point in the sample image. For example, the sample projection position information of the second sample key point uses the same representation method as the sample image position information of the first sample key point to represent the position, so as to facilitate comparison. For example, both use two-dimensional coordinates to represent the position.

[0296] Step 13044: Extract the sample projection position information of the third sample key point from the sample projection position information, extract the sample image position information of the third sample key point from the sample image position information, and determine the second sub-loss based on the sample projection position information and the sample image position information of the third sample key point.

[0297] The third sample key point is the sample key point used to calculate the second sub-loss from the sample key points shared by the first and second sample key points. Which or more of the shared sample key points are used as the third sample key point is determined empirically or flexibly adjusted according to the application scenario; this application embodiment limits this selection.

[0298] The second sub-loss is used to measure the difference between the sample projection position information and the sample image position information of the third sample keypoint. This application embodiment does not limit the method of determining the second sub-loss based on the sample projection position information and the sample image position information of the third sample keypoint. For example, the Euclidean distance between the position indicated by the sample projection position information and the position indicated by the sample image position information of the third sample keypoint is used as the second sub-loss. For example, the mean square error loss between the sample projection position information and the sample image position information of the third sample keypoint is used as the second sub-loss, etc.

[0299] Step 13045: Obtain the target loss based on the first sub-loss and the second sub-loss.

[0300] After determining the first and second sub-losses, the target loss is obtained based on these sub-losses. This method of obtaining the target loss considers more information, which is beneficial for improving the training effect of the pose recognition model.

[0301] This application does not limit the implementation method of obtaining the target loss based on the first sub-loss and the second sub-loss. For example, obtaining the target loss based on the first sub-loss and the second sub-loss includes: using the sum of the first sub-loss and the second sub-loss as the target loss. For example, obtaining the target loss based on the first sub-loss and the second sub-loss includes: calculating the product of the first sub-loss and the first weight, and the product of the second sub-loss and the second weight, and using the sum of the two calculated products as the target loss. The first weight and the second weight are set based on experience or flexibly adjusted according to the application scenario; this application does not limit this.

[0302] In step 1305, the model parameters of the initial pose recognition model are updated using the target loss to obtain the target pose recognition model.

[0303] After obtaining the target loss, the model parameters of the initial pose recognition model are updated using the target loss to obtain a trained target pose recognition model. The trained target pose recognition model has the ability to predict 3D pose parameters relatively accurately based on image position information.

[0304] The embodiments of this application do not limit the implementation method of updating the model parameters of the initial pose recognition model using the target loss. For example, the model parameters of the initial pose recognition model are updated using the target loss based on the gradient descent method.

[0305] It should be noted that when the initial pose recognition model includes multiple models, updating the model parameters of the initial pose recognition model using the target loss means updating the model parameters of each model in the initial pose recognition model using the target loss.

[0306] In an exemplary embodiment, the process of updating the model parameters of the initial pose recognition model using the target loss to obtain the target pose recognition model is an iterative process. This iterative process includes: updating the model parameters of the initial pose recognition model using the target loss; after updating, determining whether the update process meets the training termination condition; if the update process meets the training termination condition, then the currently obtained pose recognition model is taken as the target pose recognition model; if the update process does not meet the training termination condition, then a new target loss is obtained, and the model parameters of the pose recognition model are updated using the new target loss, and so on, until the update process meets the training termination condition, and the pose recognition model obtained when the update process meets the training termination condition is taken as the target pose recognition model.

[0307] The principle for obtaining the new target loss is the same as that for obtaining the target loss according to steps 1301 to 1304, and will not be repeated in this embodiment. It should be noted that the sample image used in obtaining the new target loss can be the same as or different from the sample image used in obtaining the target loss according to steps 1301 to 1304, and this embodiment does not limit this.

[0308] The training termination conditions for the update process can be set based on experience or flexibly adjusted according to the application scenario, and this application embodiment does not limit this. For example, the first training termination condition for the update process includes, but is not limited to: the number of times the model parameters have been updated has reached a reference threshold; the target loss has converged; the target loss is not greater than a reference loss threshold, etc.

[0309] In the exemplary embodiment, the process of obtaining the target loss considers not only the sample 3D pose parameters and standard 3D pose parameters, but also the case where the sample projection parameters are predicted based on the sample pose representation features by calling the initial parameter prediction model. After obtaining the target loss, in addition to updating the model parameters of the initial pose recognition model, the model parameters of the initial parameter prediction model can also be updated using the target loss. By updating the model parameters of the initial parameter prediction model using the target loss, a well-trained target parameter prediction model can be obtained. The target parameter prediction model has the ability to predict the projection parameters used to achieve the projection relatively accurately based on the pose representation features.

[0310] In an exemplary embodiment, the process of updating the model parameters of the initial pose recognition model using the target loss to obtain the target pose recognition model and the process of updating the model parameters of the initial parameter prediction model using the target loss to obtain the target parameter prediction model are performed simultaneously. That is, the target pose recognition model and the target parameter prediction model are obtained by updating the model parameters of the initial pose recognition model and the initial parameter prediction model using the target loss.

[0311] The process of updating the model parameters of the initial pose recognition model and the initial parameter prediction model using the target loss to obtain the target pose recognition model and the target parameter prediction model is an iterative process. This iterative process includes: updating the model parameters of the initial pose recognition model and the initial parameter prediction model using the target loss; after updating, determining whether the update process meets the training termination condition; if the update process meets the training termination condition, then the currently obtained pose recognition model is taken as the target pose recognition model, and the currently obtained parameter prediction model is taken as the target parameter prediction model; if the update process does not meet the training termination condition, then a new target loss is obtained, and the model parameters of the pose recognition model and the parameter prediction model are updated using the new target loss, and so on, until the update process meets the training termination condition. The pose recognition model obtained when the update process meets the training termination condition is taken as the target pose recognition model, and the parameter prediction model obtained when the update process meets the training termination condition is taken as the target parameter prediction model.

[0312] The training method for the pose recognition model provided in this application uses sample image position information and standard three-dimensional pose parameters to train the target pose recognition model. This allows the target pose recognition model to identify the three-dimensional pose of the object based on the extracted pose representation features. The pose recognition process does not require obtaining the depth of key points or the position of key points in three-dimensional space. The process is relatively simple and the computation is small, which helps to improve the efficiency of pose recognition.

[0313] See Figure 14 This application provides a posture recognition device, which includes:

[0314] The first acquisition unit 1401 is used to acquire a first image of the target object; and to acquire image position information of a first key point based on the first image. The image position information is used to indicate the position of the first key point in the first image. The first key point is a key point used to describe the two-dimensional pose of the target object.

[0315] The first feature extraction unit 1402 is used to extract features from the image position information to obtain the pose representation features of the first image;

[0316] The first recognition unit 1403 is used to recognize the three-dimensional pose of the target object based on the pose representation features of the first image, and obtain the three-dimensional pose parameters of the target.

[0317] In one possible implementation, the first image of the target object is the image located in the frame following the second image of the target object in the video of the target object; the first feature extraction unit 1402 is used to encode the image position information to obtain the first encoded feature of the first image; the first encoded feature of the first image is fused with the fusion feature of the second image to obtain the fusion feature of the first image; the fusion feature of the first image is encoded to obtain the second encoded feature of the first image; the second encoded feature of the first image is fused with the pose representation feature of the second image to obtain the pose representation feature of the first image.

[0318] In one possible implementation, the device further includes:

[0319] The prediction unit is used to predict the target projection parameters based on the pose representation features of the first image.

[0320] The determining unit is used to determine the position of the second key point in three-dimensional space based on the target's three-dimensional pose parameters. The second key point is a key point used to describe the three-dimensional pose of the target object. Based on the target projection parameters, the position of the second key point in three-dimensional space is projected onto the first image to obtain the target projection position information of the second key point.

[0321] The first acquisition unit 1401 is also used to acquire the special effects image corresponding to the first image based on the target projection position information.

[0322] In one possible implementation, the first acquisition unit 1401 is configured to extract the projection position information of the third key point from the target projection position information, extract the image position information of the third key point from the image position information, determine a first correction loss based on the projection position information and the image position information of the third key point, update the target projection parameters using the first correction loss to obtain the first projection parameters, correct the target three-dimensional pose parameters based on the first projection parameters to obtain the corrected three-dimensional pose parameters, correct the first projection parameters based on the corrected three-dimensional pose parameters to obtain the corrected projection parameters, and acquire the special effects image corresponding to the first image based on the corrected three-dimensional pose parameters and the corrected projection parameters.

[0323] In one possible implementation, the first acquisition unit 1401 is used to project the position of the second key point in three-dimensional space onto the first image based on the first projection parameters to obtain the first projection position information of the second key point; extract the projection position information of the fourth key point from the first projection position information, extract the image position information of the fourth key point from the image position information, determine the second correction loss based on the projection position information and the image position information of the fourth key point, update the target three-dimensional pose parameters using the second correction loss, and obtain the corrected three-dimensional pose parameters.

[0324] In one possible implementation, the first acquisition unit 1401 is used to determine the corrected position of the second key point in three-dimensional space based on the corrected three-dimensional pose information; project the corrected position of the second key point in three-dimensional space onto the first image based on the first projection parameters to obtain the second projection position information of the second key point; extract the projection position information of the fifth key point from the second projection position information, extract the image position information of the fifth key point from the image position information, determine the third correction loss based on the projection position information and the image position information of the fifth key point, and update the first projection parameters using the third correction loss to obtain the corrected projection parameters.

[0325] In one possible implementation, the first acquisition unit 1401 is used to determine a target 3D effect model that matches the target object based on the corrected 3D posture parameters; and to project the target 3D effect model onto the first image based on the corrected projection parameters to obtain the effect image corresponding to the first image.

[0326] In one possible implementation, the first acquisition unit 1401 is used to crop the image located within the object detection box in the first image, and acquire the image to be detected based on the cropped image; perform key point detection on the image to be detected to obtain reference position information of the first key point, the reference position information being used to indicate the position of the first key point in the image to be detected; and convert the reference position information into image position information based on the position transformation relationship between the image to be detected and the first image.

[0327] In one possible implementation, the determining unit is further configured to, in response to the first image satisfying the object detection conditions, detect the region in the first image where the target object is located, and determine an object detection box based on the region; and in response to the first image not satisfying the object detection conditions, determine an object detection box based on a reference detection box, wherein the reference detection box is determined based on an image that satisfies the object detection conditions and is associated with the first image.

[0328] In one possible implementation, the first feature extraction unit 1402 is used to call the target pose recognition model to extract features from the image position information to obtain the pose representation features of the first image.

[0329] The first recognition unit 1403 is used to call the target pose recognition model to recognize the three-dimensional pose of the target object based on the pose representation features of the first image, and obtain the target three-dimensional pose parameters.

[0330] In one possible implementation, a prediction unit is used to call a target parameter prediction model to predict target projection parameters based on the pose representation features of the first image.

[0331] The posture recognition device provided in this application recognizes the three-dimensional posture of the target object based on posture representation features. The posture representation features are obtained by extracting features from the image position information of the first key point. In this posture recognition process, it is not necessary to determine the depth of the first key point or the position of the first key point in three-dimensional space. The process is relatively simple, which helps to save computation and thus improve the efficiency of posture recognition.

[0332] See Figure 15 This application provides a training apparatus for a pose recognition model, the apparatus comprising:

[0333] The second acquisition unit 1501 is used to acquire a sample image of a sample object and standard three-dimensional pose parameters corresponding to the sample image; and to acquire sample image position information of a first sample key point based on the sample image. The sample image position information is used to indicate the position of the first sample key point in the sample image. The first sample key point is a key point used to describe the two-dimensional pose of the sample object.

[0334] The second feature extraction unit 1502 is used to call the initial pose recognition model to extract features from the position information of the sample image and obtain the sample pose representation features.

[0335] The second identification unit 1503 is used to identify the three-dimensional pose of the sample object based on the sample pose characterization features, and obtain the three-dimensional pose parameters of the sample.

[0336] The second acquisition unit 1501 is also used to acquire the target loss based on the standard three-dimensional attitude parameters and the sample three-dimensional attitude parameters.

[0337] Update unit 1504 is used to update the model parameters of the initial pose recognition model using the target loss, so as to obtain the target pose recognition model.

[0338] In one possible implementation, the second acquisition unit 1501 is further configured to call the initial parameter prediction model to predict the sample projection parameters based on the sample pose representation features; and to acquire the target loss based on the standard three-dimensional pose parameters, the sample three-dimensional pose parameters, and the sample projection parameters.

[0339] In one possible implementation, the second acquisition unit 1501 is further configured to: determine a first sub-loss based on standard 3D pose parameters and sample 3D pose parameters; determine the position of a second sample keypoint in 3D space based on the sample 3D pose parameters, wherein the second sample keypoint is a keypoint used to describe the 3D pose of the sample object; project the position of the second sample keypoint in 3D space onto a sample image based on sample projection parameters to obtain sample projection position information of the second sample keypoint; extract sample projection position information of a third sample keypoint from the sample projection position information, extract sample image position information of the third sample keypoint from the sample image position information, determine a second sub-loss based on the sample projection position information and the sample image position information of the third sample keypoint; and obtain a target loss based on the first sub-loss and the second sub-loss.

[0340] In one possible implementation, the update unit 1504 is also used to update the model parameters of the initial parameter prediction model using the target loss, so as to obtain the target parameter prediction model.

[0341] The posture recognition model training device provided in this application uses sample image position information and standard three-dimensional posture parameters to train a target posture recognition model. This allows the target posture recognition model to identify the three-dimensional posture of the object based on the extracted posture representation features. The posture recognition process does not require obtaining the depth of key points or the position of key points in three-dimensional space. The process is relatively simple and the computation is small, which helps to improve the efficiency of posture recognition.

[0342] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional units. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the device can be divided into different functional units to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0343] In an exemplary embodiment, a computer device is also provided, comprising a processor and a memory, wherein at least one computer program is stored in the memory. The at least one computer program is loaded and executed by one or more processors to enable the computer device to implement any of the above-described pose recognition methods or pose recognition model training methods. The computer device can be a server or a terminal, and this embodiment does not limit this. The structures of the server and the terminal will be described separately below.

[0344] Figure 16This is a schematic diagram of a server structure provided in an embodiment of this application. The server can vary significantly due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 1601 and one or more memories 1602. The one or more memories 1602 store at least one computer program, which is loaded and executed by the one or more processors 1601 to enable the server to implement the posture recognition method or posture recognition model training method provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0345] Figure 17 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal can be: a PC, mobile phone, smartphone, PDA, wearable device, PPC, tablet computer, smart car system, smart TV, smart speaker, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc. The terminal may also be referred to as user equipment, user terminal, portable terminal, laptop terminal, desktop terminal, or other names.

[0346] Typically, a terminal includes a processor 1701 and a memory 1702.

[0347] Processor 1701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1701 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1701 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0348] The memory 1702 may include one or more computer-readable storage media, which may be non-transitory. The memory 1702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1702 is used to store at least one instruction, which is executed by the processor 1701 to enable the terminal to implement the attitude recognition method or attitude recognition model training method provided in the method embodiments of this application.

[0349] In some embodiments, the terminal may also optionally include: a peripheral device interface 1703 and at least one peripheral device. The processor 1701, memory 1702, and peripheral device interface 1703 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1703 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1704, a display screen 1705, a camera assembly 1706, an audio circuit 1707, and a power supply 1708.

[0350] Peripheral interface 1703 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1701 and memory 1702. In some embodiments, processor 1701, memory 1702 and peripheral interface 1703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1701, memory 1702 and peripheral interface 1703 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0351] The radio frequency (RF) circuit 1704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1704 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1704 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0352] Display screen 1705 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1705 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1701 for processing. In this case, display screen 1705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1705, located on the front panel of the terminal; in other embodiments, there may be at least two display screens, respectively located on different surfaces of the terminal or in a folded design; in other embodiments, display screen 1705 may be a flexible display screen, located on a curved or folded surface of the terminal. Furthermore, display screen 1705 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 1705 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0353] The camera assembly 1706 is used to acquire images or videos. Optionally, the camera assembly 1706 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1706 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0354] The audio circuit 1707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1701 for processing, or input to the radio frequency circuit 1704 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1701 or the radio frequency circuit 1704 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1707 may also include a headphone jack.

[0355] Power supply 1708 is used to power the various components in the terminal. Power supply 1708 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1708 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0356] In some embodiments, the terminal further includes one or more sensors 1709. The one or more sensors 1709 include, but are not limited to: an accelerometer 1710, a gyroscope 1711, a pressure sensor 1712, an optical sensor 1713, and a proximity sensor 1714.

[0357] Accelerometer 1710 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by the terminal. For example, accelerometer 1710 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1701 can control display screen 1705 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1710. Accelerometer 1710 can also be used for games or for acquiring user motion data.

[0358] The gyroscope sensor 1711 can detect the terminal's orientation and rotation angle. The gyroscope sensor 1711 can work in conjunction with the accelerometer sensor 1710 to collect the user's 3D movements on the terminal. Based on the data collected by the gyroscope sensor 1711, the processor 1701 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0359] The pressure sensor 1712 can be disposed on the side bezel of the terminal and / or the lower layer of the display screen 1705. When the pressure sensor 1712 is disposed on the side bezel of the terminal, it can detect the user's grip signal on the terminal, and the processor 1701 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1712. When the pressure sensor 1712 is disposed on the lower layer of the display screen 1705, the processor 1701 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1705. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0360] Optical sensor 1713 is used to collect ambient light intensity. In one embodiment, processor 1701 can control the display brightness of display screen 1705 based on the ambient light intensity collected by optical sensor 1713. Specifically, when the ambient light intensity is high, the display brightness of display screen 1705 is increased; when the ambient light intensity is low, the display brightness of display screen 1705 is decreased. In another embodiment, processor 1701 can also dynamically adjust the shooting parameters of camera assembly 1706 based on the ambient light intensity collected by optical sensor 1713.

[0361] The proximity sensor 1714, also known as a distance sensor, is typically installed on the front panel of the terminal. The proximity sensor 1714 is used to detect the distance between the user and the front of the terminal. In one embodiment, when the proximity sensor 1714 detects that the distance between the user and the front of the terminal is gradually decreasing, the processor 1701 controls the display screen 1705 to switch from a screen-on state to a screen-off state; when the proximity sensor 1714 detects that the distance between the user and the front of the terminal is gradually increasing, the processor 1701 controls the display screen 1705 to switch from a screen-off state to a screen-on state.

[0362] Those skilled in the art will understand that Figure 17 The structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0363] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program, which is loaded and executed by a processor of a computer device to enable the computer to implement any of the above-described pose recognition methods or pose recognition model training methods.

[0364] In one possible implementation, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0365] In an exemplary embodiment, a computer program product is also provided, which includes a computer program or computer instructions that are loaded and executed by a processor to enable a computer to implement any of the above-described pose recognition methods or pose recognition model training methods.

[0366] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the first image of the target object involved in this application was obtained with full authorization.

[0367] It should also be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the above exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0368] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0369] The above are merely exemplary embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A gesture recognition method, characterized by, The method comprises: obtaining a first image of a target object; obtaining image position information of a first key point based on the first image, the image position information being used to indicate a position of the first key point in the first image, the first key point being a key point used to describe a two-dimensional pose of the target object; performing feature extraction on the image position information to obtain a pose representation feature of the first image; identifying a three-dimensional pose of the target object based on the pose representation feature of the first image to obtain a target three-dimensional pose parameter; predicting a target projection parameter based on the pose representation feature of the first image; determining a position of a second key point in a three-dimensional space based on the target three-dimensional pose parameter, the second key point being a key point used to describe a three-dimensional pose of the target object; projecting the position of the second key point in the three-dimensional space into the first image based on the target projection parameter to obtain target projection position information of the second key point; extracting projection position information of a third key point from the target projection position information and extracting image position information of the third key point from the image position information, and determining a first correction loss based on the projection position information of the third key point and the image position information of the third key point; updating the target projection parameter using the first correction loss to obtain a first projection parameter; correcting the target three-dimensional pose parameter based on the first projection parameter to obtain a corrected three-dimensional pose parameter, and correcting the first projection parameter based on the corrected three-dimensional pose parameter to obtain a corrected projection parameter; obtaining a special effect image corresponding to the first image based on the corrected three-dimensional pose parameter and the corrected projection parameter.

2. The method of claim 1, wherein, The first image of the target object is an image located after a second image of the target object in a video of the target object; The feature extraction on the image position information to obtain the pose representation feature of the first image comprises: encoding the image position information to obtain a first encoded feature of the first image; fusing the first encoded feature of the first image with a fusion feature of the second image to obtain a fusion feature of the first image; encoding the fusion feature of the first image to obtain a second encoded feature of the first image; fusing the second encoded feature of the first image with a pose representation feature of the second image to obtain the pose representation feature of the first image.

3. The method of claim 1, wherein, The correction of the target three-dimensional pose parameter based on the first projection parameter to obtain the corrected three-dimensional pose parameter comprises: projecting the position of the second key point in the three-dimensional space into the first image based on the first projection parameter to obtain first projection position information of the second key point; extract fourth key point projection position information from the first projection position information, extract fourth key point image position information from the image position information, determine a second correction loss based on the fourth key point projection position information and the fourth key point image position information, update the target three-dimensional pose parameter by using the second correction loss to obtain the corrected three-dimensional pose parameter.

4. The method of claim 1, wherein, The correcting the first projection parameter based on the corrected three-dimensional pose parameter to obtain a corrected projection parameter comprises: determining a corrected position of the second key point in the three-dimensional space based on the corrected three-dimensional pose information; projecting the corrected position of the second key point in the three-dimensional space into the first image based on the first projection parameter to obtain second projection position information of the second key point; extract fifth key point projection position information from the second projection position information, extract fifth key point image position information from the image position information, determine a third correction loss based on the fifth key point projection position information and the fifth key point image position information, and update the first projection parameter by using the third correction loss to obtain the corrected projection parameter.

5. The method of claim 1, wherein, The obtaining the special effect image corresponding to the first image based on the corrected three-dimensional pose parameter and the corrected projection parameter comprises: determining a target three-dimensional special effect model matched with the target object based on the corrected three-dimensional pose parameter; projecting the target three-dimensional special effect model into the first image based on the corrected projection parameter to obtain the special effect image corresponding to the first image.

6. The method according to any one of claims 1 to 5, characterized in that, The obtaining the image position information of the first key point based on the first image comprises: cropping an image located in an object detection frame in the first image, and obtaining a to-be-detected image based on the cropped image; performing key point detection on the to-be-detected image to obtain reference position information of the first key point, the reference position information being used to indicate a position of the first key point in the to-be-detected image; converting the reference position information into the image position information based on a position conversion relationship between the to-be-detected image and the first image.

7. The method of claim 6, wherein, Before the cropping the image located in the object detection frame in the first image, the method further comprises: in response to the first image satisfying an object detection condition, detecting a region where the target object is located in the first image, and determining the object detection frame based on the region; in response to the first image not satisfying the object detection condition, determining the object detection frame based on a reference detection frame, the reference detection frame being determined based on an image satisfying the object detection condition and associated with the first image.

8. The method according to any of claims 1-5, 7, characterized by, The performing feature extraction on the image position information to obtain the pose representation feature of the first image comprises: calling a target pose recognition model to perform feature extraction on the image position information to obtain the pose representation feature of the first image; The pose representation feature based on the first image identifies a three-dimensional pose of the target object, and target three-dimensional pose parameters are obtained. The pose representation feature based on the first image is called to identify a three-dimensional pose of the target object based on the target pose recognition model, and target three-dimensional pose parameters are obtained.

9. The method of claim 1, wherein, The pose representation feature based on the first image is called to identify a three-dimensional pose of the target object based on the target pose recognition model, and target three-dimensional pose parameters are obtained. The pose representation feature based on the first image is called to identify a three-dimensional pose of the target object based on the target pose recognition model, and target three-dimensional pose parameters are obtained. 10.A method for training a pose recognition model, the method comprising: The method comprises: Obtaining a sample image of a sample object and standard three-dimensional pose parameters corresponding to the sample image; Based on the sample image, sample image position information of a first sample key point is obtained, the sample image position information being used to indicate a position of the first sample key point in the sample image, the first sample key point being a key point used to describe a two-dimensional pose of the sample object; An initial pose recognition model is called to extract features from the sample image position information, and sample pose representation features are obtained, based on which a three-dimensional pose of the sample object is identified, and sample three-dimensional pose parameters are obtained; Based on the standard three-dimensional pose parameters and the sample three-dimensional pose parameters, a target loss is obtained; The model parameters of the initial pose recognition model are updated using the target loss, and a target pose recognition model is obtained, which is used to obtain target three-dimensional pose parameters according to any one of claims 1 to 9, and the target three-dimensional pose parameters are used to obtain a special effect image corresponding to a first object according to any one of claims 1 to 9.

11. The method of claim 10, wherein, Based on the standard three-dimensional pose parameters and the sample three-dimensional pose parameters, a target loss is obtained, comprising: An initial parameter prediction model is called to predict sample projection parameters based on the sample pose representation features; Based on the standard three-dimensional pose parameters, the sample three-dimensional pose parameters and the sample projection parameters, the target loss is obtained.

12. The method of claim 11, wherein, Based on the standard three-dimensional pose parameters and the sample three-dimensional pose parameters, a target loss is obtained, comprising: Based on the standard three-dimensional pose parameters and the sample three-dimensional pose parameters, a first sub-loss is determined; Based on the sample three-dimensional pose parameters, a position of a second sample key point in a three-dimensional space is determined, the second sample key point being a key point used to describe a three-dimensional pose of the sample object; Based on the sample projection parameters, the position of the second sample key point in the three-dimensional space is projected into the sample image, and sample projection position information of the second sample key point is obtained; From the sample projection position information, sample projection position information of a third sample key point is extracted, and from the sample image position information, sample image position information of the third sample key point is extracted, based on which a second sub-loss is determined. Based on the first sub-loss and the second sub-loss, the target loss is obtained.

13. The method according to claim 11 or 12, characterized in that, The method further comprises: Update a model parameter of the initial parameter prediction model by using the target loss to obtain a target parameter prediction model.

14. A gesture recognition apparatus, characterized by The device comprises: The first acquisition unit is configured to acquire a first image of a target object, acquire image position information of a first key point based on the first image, and use the image position information to indicate a position of the first key point in the first image, wherein the first key point is used to describe a two-dimensional pose of the target object. The first feature extraction unit is configured to perform feature extraction on the image position information to obtain pose representation features of the first image. The first identification unit is configured to identify a three-dimensional pose of the target object based on the pose representation features of the first image to obtain target three-dimensional pose parameters. The prediction unit is configured to predict target projection parameters based on the pose representation features of the first image. The determination unit is configured to determine a position of a second key point in a three-dimensional space based on the target three-dimensional pose parameters, wherein the second key point is used to describe a three-dimensional pose of the target object, project the position of the second key point in the three-dimensional space into the first image based on the target projection parameters to obtain target projection position information of the second key point. The first acquisition unit is further configured to extract projection position information of a third key point from the target projection position information, extract image position information of the third key point from the image position information, determine a first correction loss based on the projection position information of the third key point and the image position information of the third key point, update the target projection parameters by using the first correction loss to obtain first projection parameters, correct the target three-dimensional pose parameters based on the first projection parameters to obtain corrected three-dimensional pose parameters, correct the first projection parameters based on the corrected three-dimensional pose parameters to obtain corrected projection parameters, and obtain a special effect image corresponding to the first image based on the corrected three-dimensional pose parameters and the corrected projection parameters.

15. The apparatus of claim 14, wherein, The first image of the target object is an image located at a frame after a second image of the target object in a video of the target object. The first feature extraction unit is configured to encode the image position information to obtain first encoded features of the first image, fuse the first encoded features of the first image with fusion features of the second image to obtain fusion features of the first image, encode the fusion features of the first image to obtain second encoded features of the first image, and fuse the second encoded features of the first image with the pose representation features of the second image to obtain the pose representation features of the first image. The first acquisition unit is configured to project the position of the second key point in the three-dimensional space into the first image based on the first projection parameters to obtain first projection position information of the second key point.

16. The apparatus of claim 14, wherein, ​ extract fourth key point projection position information from the first projection position information, extract fourth key point image position information from the image position information, determine a second correction loss based on the fourth key point projection position information and the fourth key point image position information, update the target three-dimensional pose parameter by using the second correction loss to obtain the corrected three-dimensional pose parameter.

17. The apparatus of claim 14, wherein, The first obtaining unit is configured to determine a corrected position of the second key point in the three-dimensional space based on the corrected three-dimensional pose information, and project the corrected position of the second key point in the three-dimensional space into the first image based on the first projection parameter to obtain second projection position information of the second key point. extract fifth key point projection position information from the second projection position information, extract fifth key point image position information from the image position information, determine a third correction loss based on the fifth key point projection position information and the fifth key point image position information, and update the first projection parameter by using the third correction loss to obtain the corrected projection parameter.

18. The apparatus of claim 14, wherein, The first obtaining unit is configured to determine a target three-dimensional special effect model matched with the target object based on the corrected three-dimensional pose parameter, and project the target three-dimensional special effect model into the first image based on the corrected projection parameter to obtain a special effect image corresponding to the first image.

19. The apparatus of any of claims 14-18, wherein The first obtaining unit is configured to crop an image located in the object detection frame in the first image, and obtain a to-be-detected image based on the cropped image. perform key point detection on the to-be-detected image to obtain reference position information of the first key point, the reference position information being used to indicate a position of the first key point in the to-be-detected image. convert the reference position information into the image position information based on a position conversion relationship between the to-be-detected image and the first image.

20. The apparatus of claim 19, wherein, The determining unit is further configured to, in response to the first image satisfying an object detection condition, detect a region where the target object is located in the first image, and determine the object detection frame based on the region, and in response to the first image not satisfying the object detection condition, determine the object detection frame based on a reference detection frame, the reference detection frame being determined based on an image that satisfies the object detection condition and is associated with the first image.

21. The apparatus of any of claims 14-18, 20, wherein, The first feature extraction unit is configured to call a target pose recognition model to perform feature extraction on the image position information to obtain pose representation features of the first image. The first recognition unit is configured to call the target pose recognition model to recognize a three-dimensional pose of the target object based on the pose representation features of the first image to obtain a target three-dimensional pose parameter.

22. The apparatus of claim 14, wherein, The prediction unit is configured to call a target parameter prediction model to predict a target projection parameter based on the pose representation features of the first image.

23. A training device for a pose recognition model, characterized in that, The apparatus comprises: The second acquisition unit is configured to acquire a sample image of a sample object and a standard three-dimensional pose parameter corresponding to the sample image; acquire sample image position information of a first sample key point based on the sample image, the sample image position information being used to indicate a position of the first sample key point in the sample image, the first sample key point being a key point used to describe a two-dimensional pose of the sample object; The second feature extraction unit is configured to call an initial pose recognition model to perform feature extraction on the sample image position information to obtain sample pose representation features. The second recognition unit is configured to recognize a three-dimensional pose of the sample object based on the sample pose representation features to obtain a sample three-dimensional pose parameter. The second acquisition unit is further configured to acquire a target loss based on the standard three-dimensional pose parameter and the sample three-dimensional pose parameter. The update unit is configured to update model parameters of the initial pose recognition model by using the target loss to obtain a target pose recognition model, the target pose recognition model being used to acquire a target three-dimensional pose parameter according to the pose recognition method in any one of claims 1 to 9, and the target three-dimensional pose parameter being used to acquire a special effect image corresponding to a first object according to the pose recognition method in any one of claims 1 to 9.

24. The apparatus of claim 23, wherein, The second acquisition unit is further configured to call an initial parameter prediction model to predict sample projection parameters based on the sample pose representation features; and acquire the target loss based on the standard three-dimensional pose parameter, the sample three-dimensional pose parameter, and the sample projection parameters.

25. The apparatus of claim 24, wherein, The second acquisition unit is further configured to determine a first sub-loss based on the standard three-dimensional pose parameter and the sample three-dimensional pose parameter; and determine a position of a second sample key point in a three-dimensional space based on the sample three-dimensional pose parameter, the second sample key point being a key point used to describe a three-dimensional pose of the sample object. The second acquisition unit is further configured to project the position of the second sample key point in the three-dimensional space into the sample image based on the sample projection parameters to obtain sample projection position information of the second sample key point. The second acquisition unit is further configured to extract sample projection position information of a third sample key point from the sample projection position information, extract sample image position information of the third sample key point from the sample image position information, determine a second sub-loss based on the sample projection position information of the third sample key point and the sample image position information of the third sample key point, and acquire the target loss based on the first sub-loss and the second sub-loss.

26. The apparatus of claim 24 or 25, wherein, The update unit is further configured to update model parameters of the initial parameter prediction model by using the target loss to obtain a target parameter prediction model.

27. A computer device, comprising: The computer device includes a processor and a memory, and the memory stores at least one computer program, which is loaded and executed by the processor, so that the computer device implements the pose recognition method in any one of claims 1 to 9 or the training method of the pose recognition model in any one of claims 10 to 13.

28. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to enable the computer to implement the posture recognition method according to any one of claims 1 to 9, or the training method of the posture recognition model according to any one of claims 10 to 13.

29. A computer program product, characterised in that, The computer program product includes a computer program or computer instructions, and the computer program or the computer instructions are loaded and executed by the processor to enable the computer to implement the posture recognition method according to any one of claims 1 to 9, or the training method of the posture recognition model according to any one of claims 10 to 13.

Images