Method, apparatus and device for detecting a shape parameter of a target object in an image

Abstract

Embodiments of the present specification provide a method, device and equipment for detecting a shape parameter of a target object in an image. By performing feature extraction on a to-be-detected image containing a target object, a first feature can be obtained, and a second feature can be obtained based on position information of key points of the target object in the to-be-detected image. The second feature is fused into the first feature, and a first shape parameter is determined based on the updated first feature. Meanwhile, the first feature is also fused into the second feature, and a second shape parameter is determined based on the updated second feature. Then, the first shape parameter and the second shape parameter can be integrated to obtain a final target shape parameter. Embodiments of the present application consider the correlation and interaction between the features obtained in different ways by interacting the features obtained in different ways, so as to improve the accuracy of the finally determined shape parameter of the target object in the image.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of artificial intelligence, and in particular, to a method and device for detecting a morphological parameter of a target object in an image. BACKGROUND

[0002] In many scenarios, it is necessary to detect a morphological parameter of a target object in an image, such as detecting a pose parameter (e.g., a rotation angle of each joint) of a target object in an image or detecting a shape parameter (e.g., height, weight, etc.) of a target object in an image, and then the detected morphological parameter can be used for subsequent applications. For example, the detected morphological parameter can be used to drive a digital person to present a consistent morphology with the target object in the image, or an intelligent terminal can be controlled based on the detected pose parameter to execute an instruction matching the pose. Currently, related technologies for detecting a morphological parameter of a target object in an image either have a large amount of calculation and cannot be applied to a terminal with low performance, or the detection result has low accuracy and the obtained morphological parameter is not accurate enough to affect subsequent use. Therefore, it is necessary to provide a scheme for determining a morphological parameter of a target object in an image with less calculation and high accuracy. SUMMARY

[0003] Based on this, the present specification provides a method, device and equipment for detecting a morphological parameter of a target object in an image.

[0004] According to a first aspect of an embodiment of the present specification, a method for detecting a morphological parameter of a target object in an image is provided, the morphological parameter including a shape parameter and / or a pose parameter of the target object, and the method includes:

[0005] obtaining a to-be-detected image including a target object;

[0006] performing feature extraction on the to-be-detected image to obtain a first feature;

[0007] determining key point map data of the target object based on the to-be-detected image, and obtaining a second feature based on the key point map data, wherein the key point map data includes position information of the key points and a connection relationship between the key points;

[0008] fusing the second feature into the first feature to obtain an updated first feature, and determining a first morphological parameter of the target object based on the updated first feature;

[0009] fusing the first feature into the second feature to obtain an updated second feature, and determining a second morphological parameter of the target object based on the updated second feature, so as to obtain a target morphological parameter according to the first morphological parameter and the second morphological parameter.

[0010] According to a second aspect of the embodiments of the present specification, an apparatus for detecting a morphological parameter of a target object in an image is provided, the morphological parameter comprising a shape parameter and / or a pose parameter of the target object, the apparatus comprising:

[0011] an acquisition module configured to acquire a to-be-detected image comprising a target object;

[0012] a first feature extraction module configured to perform feature extraction on the to-be-detected image to obtain a first feature;

[0013] a second feature extraction module configured to determine key point map data of the target object based on the to-be-detected image, and obtain a second feature based on the key point map data, wherein the key point map data comprises position information of the key points and connection relationships between the key points;

[0014] a first prediction module configured to fuse the second feature into the first feature to obtain an updated first feature, and determine a first morphological parameter of the target object based on the updated first feature;

[0015] a second prediction module configured to fuse the first feature into the second feature to obtain an updated second feature, and determine a second morphological parameter of the target object based on the updated second feature, so as to obtain a target morphological parameter according to the first morphological parameter and the second morphological parameter.

[0016] According to a third aspect of the embodiments of the present specification, an electronic device is provided, the electronic device comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, the processor implementing the method mentioned in the first aspect when executing the computer program.

[0017] By applying the scheme of the embodiments of this specification, feature extraction can be performed on the image to be detected containing the target object to obtain a first feature. The location information of key points of the target object and the connection relationships between key points can be determined from the image to be detected. Based on the location information and connection relationships of the key points, a second feature can be obtained. Then, the first and second features obtained using the above two methods are interacted. For example, the second feature can be fused into the first feature to update the first feature, and a first morphological parameter can be determined based on the updated first feature. Simultaneously, the first feature can also be fused into the second feature to update the second feature, and a second morphological parameter can be determined based on the updated second feature. Finally, the first and second morphological parameters obtained by the above two methods can be combined to obtain the final target morphological parameter. The embodiments of this application, by interacting with features obtained through different methods, consider the correlation and interaction between features obtained through different methods, thereby improving the accuracy of the morphological parameters of the target object in the final determined image.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0020] Figure 1 This is a flowchart of a method for detecting the morphological parameters of a target object in an image, according to one embodiment of this specification.

[0021] Figure 2 This is a schematic diagram of a key point connection diagram of one embodiment of this specification.

[0022] Figure 3 This is a schematic diagram of the structure of a neural network according to one embodiment of this specification.

[0023] Figure 4 This is a schematic diagram of the structure of a neural network according to one embodiment of this specification.

[0024] Figure 5 This is a logical structure block diagram of an apparatus for detecting the morphological parameters of a target object in an image, according to one embodiment of this specification.

[0025] Figure 6 This is a logical structure block diagram of an electronic device according to one embodiment of this specification. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0027] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0028] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0029] In many scenarios, it is necessary to detect the morphological parameters of target objects in images. For example, detecting the pose parameters (such as the rotation angles of the joints) or the shape parameters (such as height, weight, etc.) of target objects in images. These detected morphological parameters can then be used for subsequent applications. Currently, some techniques for detecting the morphological parameters of target objects in images extract image features using models and predict the morphological parameters based on these features. To obtain accurate feature extraction, the models used are usually quite complex, computationally intensive, and demanding on terminal performance, while also exhibiting long latency, making them unsuitable for applications with high real-time requirements.

[0030] Some techniques determine the location of keypoints in an image, then use these keypoints and their connections to create a keypoint connectivity map. Based on this map, the morphological parameters of the target object are predicted. This method is computationally less expensive, but the prediction accuracy is low due to the loss of depth information, especially when keypoint locations are predicted incorrectly, which severely impacts the final result's accuracy. Therefore, it is necessary to provide a computationally less expensive yet highly accurate method for determining the morphological parameters of target objects in an image.

[0031] Based on this, embodiments of this disclosure provide a method for detecting the morphological parameters of a target object in an image. By performing feature extraction on the image to be detected containing the target object, a first feature can be obtained. The location information of key points of the target object and the connection relationship between key points can be determined through the image to be detected. A second feature can be obtained based on the location information and connection relationship of key points. Then, the first feature and the second feature obtained by the above two methods are interacted. For example, the second feature can be fused into the first feature to update the first feature, and a first morphological parameter can be determined based on the updated first feature. At the same time, the first feature can also be fused into the second feature to update the second feature, and a second morphological parameter can be determined based on the updated second feature. Then, the first morphological parameter and the second morphological parameter obtained by the above two methods can be combined to obtain the final target morphological parameter.

[0032] This application's embodiments improve the accuracy of the morphological parameters of the target object in the final determined image by interacting features obtained through different methods (e.g., first features extracted from the image and second features obtained based on key points of the target object). Furthermore, because this application interacts features obtained through different methods, the interacted features are more accurate. Therefore, when extracting features from the image, a simpler model can be used to obtain accurate prediction results, meaning that high accuracy can be achieved even in scenarios with relatively low computational requirements.

[0033] The target object in this application embodiment can be any object that requires morphological parameters to be determined, such as the human body, a part of the human body, an animal, a cartoon character, or a virtual character. This disclosure embodiment does not impose any restrictions.

[0034] The morphological parameters in this application embodiment include at least one or more of the shape parameters and posture parameters of the target object. The shape parameters can be parameters characterizing the size of various parts of the target object, such as parameters characterizing the height or weight of the target object. The posture parameters can be parameters characterizing the rotation angles of various joints of the target object, such as the rotation angles of joints like the head and wrists of the target object.

[0035] Specifically, such as Figure 1 As shown, the method for detecting the morphological parameters of a target object in an image in this embodiment of the present disclosure may include the following steps:

[0036] S102. Obtain the image to be detected, which includes the target object;

[0037] S104. Extract features from the image to be detected to obtain the first feature;

[0038] S106. Determine the key point map data of the target object based on the image to be detected, and obtain the second feature based on the key point map data, wherein the key point map data includes the location information of the key points and the connection relationship between the key points;

[0039] S108. The second feature is fused into the first feature to obtain the updated first feature, and the first morphological parameter of the target object is determined based on the updated first feature.

[0040] S1010. The first feature is fused into the second feature to obtain an updated second feature, and the second morphological parameter of the target object is determined based on the updated second feature, so as to obtain the target morphological parameter according to the first morphological parameter and the second morphological parameter.

[0041] In step S102, the image to be detected can be obtained first. The image to be detected includes the target object. The image to be detected can be any image in which the morphological parameters of the target object in the image need to be detected.

[0042] In step S104, after acquiring the image to be detected, feature extraction can be performed on the image to obtain a first feature. The first feature can be features of various dimensions such as the image's contour and color obtained through feature extraction. Feature extraction can be performed using a pre-trained feature extraction network, or other methods can be employed; this application embodiment does not impose any limitations.

[0043] In step S106, after acquiring the image to be detected, keypoint map data of the target object can be determined based on the image. This keypoint map data includes the location information of keypoints and the connection relationships between them. A keypoint connection map can be obtained from the keypoint map data, and then a second feature can be determined based on the keypoint connection map. The second feature represents the association relationship between keypoints. The location information of the keypoints of the target object in the image to be detected can be determined by a pre-trained model, and the connection relationships between keypoints can be pre-set. Based on the location information of the keypoints and the connection relationships between them, the keypoint connection map can be obtained. When the target object is a human body, its keypoint connection map is as follows: Figure 2 As shown, feature extraction can be performed on the keypoint connection graph to obtain a second feature representing the relationship between keypoints. This second feature can be extracted using a pre-trained neural network, or other methods can be employed; this embodiment of the application does not impose limitations.

[0044] Of course, since the features obtained through image feature extraction and the features obtained based on key points can be regarded as features of two different modalities, and the features of the two different modalities are related to a certain extent, the features obtained from only one modality may not be accurate enough. Therefore, it is possible to consider interacting the features of the two different modalities. For any feature of a modality, features of other modalities can be used to update it. Therefore, in step S108, the second feature can be fused into the first feature to update the first feature and obtain the updated first feature. Then, the first morphological parameters of the target object can be determined based on the updated first feature.

[0045] Similarly, in step S1010, the first feature can be fused into the second feature, the second feature can be updated to obtain the updated second feature, and then the second morphological parameters of the target object can be determined based on the updated second feature.

[0046] There are many ways to fuse the first feature into the second feature or vice versa. For example, to fuse the first feature into the second feature, one can directly concatenate the first feature into the second feature, sum the first and second features, or calculate the first and second features according to a preset formula to obtain the second feature influenced by the first feature. Of course, the specific fusion method can be set according to actual needs. The main idea is to use the first feature to influence the second feature, or use the second feature to influence the first feature, to obtain the feature influenced by the other, and then use the influenced feature to predict morphological parameters.

[0047] After obtaining the first and second morphological parameters determined using different methods, the target morphological parameters can be obtained by combining the first and second morphological parameters. For example, the two parameters can be averaged, weighted averaged, or calculated according to a preset method to obtain the final target morphological parameters.

[0048] In some embodiments, when fusing the second feature into the first feature to obtain the updated first feature, a first weight coefficient set can be determined based on the first and second features. This first weight coefficient set is used to characterize the degree of influence of the second feature on the first feature. Then, the second feature can be weighted using the first weight coefficient set, and the weighted second feature can be summed with the first feature to obtain the updated first feature.

[0049] In some embodiments, when fusing the first feature into the second feature to obtain the updated second feature, a second weight coefficient group can be determined based on the first feature and the second feature. The second weight coefficient group is used to characterize the degree of influence of the first feature on the second feature. Then, the first feature can be weighted using the second weight coefficient group, and the weighted first feature and the second feature can be summed to obtain the updated second feature.

[0050] In some embodiments, the first feature and the second feature can be represented by vectors, where the values ​​of different dimensions of the vector represent the features of the image to be detected in different dimensions. The first weight coefficient set can be determined based on the product of the vector representing the first feature and the transpose of the vector representing the second feature. The first weight coefficient set can be a matrix, where each element represents the magnitude of the influence of a certain dimension of the second feature on the same dimension of the first feature. Similarly, the second weight coefficients can be determined based on the product of the vector representing the second feature and the transpose of the vector representing the first feature. The second weight coefficient set can also be a matrix, where each element represents the magnitude of the influence of a certain dimension of the first feature on the same dimension of the second feature. The obtained weight coefficient set can then be used to weight either the first or second feature, and the weighted feature can be summed with the other feature to obtain the feature affected by the first feature.

[0051] In some embodiments, when determining keypoint map data of a target object based on an image to be detected, features can first be extracted from the image to be detected to obtain a feature map. Then, the location information of keypoints in the image to be detected can be determined based on this feature map. Since the connection relationships between keypoints are pre-set, after determining the location information of the keypoints, the keypoint map data can be obtained by combining the pre-set keypoint connection relationships. Feature extraction of the image to be detected can be implemented using a pre-trained feature extraction network, and the determination of keypoint location information using the feature map can also be achieved using a pre-trained model.

[0052] In some embodiments, after obtaining the updated first feature and the updated second feature, the updated first feature can be directly used to determine the first morphological parameter, and the updated second feature can be used to determine the second morphological parameter. In some embodiments, to make the determined first morphological parameter more accurate, after obtaining the updated first feature, feature extraction can be further performed on the updated first feature, and the first morphological parameter of the target object can be determined based on the extracted features. By continuing to extract features from the first feature after it has been influenced by the second feature, the final extracted features can be made more accurate, thereby making the determined first morphological parameter more accurate.

[0053] Similarly, in some embodiments, to ensure the accuracy of the determined second morphological parameters, after obtaining the updated second feature, further feature extraction can be performed on the updated second feature, and the second morphological parameters of the target object can be determined based on the extracted features. By continuing feature extraction on the second feature influenced by the first feature, the final extracted features can be made more accurate, thereby making the determined second morphological parameters more accurate.

[0054] In some embodiments, both the first feature and the second feature can be represented by vectors, matrices, or other representational information. Since the two features are extracted using different methods, their representational information may have different dimensions. To facilitate the fusion of the two features, the dimensions of their representational information can be unified first. For example, the representational information of the first feature and / or the second feature can be deformed to make the dimensions of the first feature's representational information consistent with those of the second feature's representational information. For instance, since the first feature is obtained directly from feature extraction of the image to be detected, it can typically be represented by a two-dimensional matrix. The second feature, on the other hand, is a feature extracted based on key points of the target object and is typically represented by a one-dimensional vector. To unify the representational information of the two features into a single dimension, the two-dimensional matrix representing the first feature can be deformed into a one-dimensional vector, thus facilitating subsequent fusion processing.

[0055] In some embodiments, when obtaining the target morphological parameter based on the first morphological parameter and the second morphological parameter, the first weight corresponding to the first morphological parameter and the second weight corresponding to the second morphological parameter can be determined respectively. Then, the first morphological parameter and the second morphological parameter are weighted and summed based on the first weight and the second weight to obtain the target morphological parameter.

[0056] In some embodiments, extensive experiments have shown that as the first weight increases from 0 to 1, the accuracy of the obtained target shape parameters exhibits a trend of first increasing and then decreasing. Furthermore, the accuracy of the obtained target shape parameters is highest when the first weight is 0.8. Therefore, the first weight can be set to 0.8, and the second weight can be set to 0.2.

[0057] In some embodiments, after obtaining the target morphological parameters based on the first morphological parameters and the second morphological parameters, a driving signal can be generated based on the target morphological parameters. This driving signal is then used to drive a preset 3D model to present a shape that matches the target object in the image to be detected. For example, in some scenarios, it is necessary to drive a preset 3D model to perform the same action based on the action of a human body in the image to be detected. In order to make the action and shape of the 3D model consistent with the target object in the image, the action parameters and shape parameters of the target object in the image can be determined first, and a driving signal can be generated based on these parameters to drive the 3D model to present the same shape.

[0058] In some embodiments, the method for determining the morphological parameters of a target object from an image to be detected can be implemented using a pre-trained neural network, wherein the network structure of the neural network is as follows: Figure 3 As shown, the network can include a feature extraction sub-network, a first network branch, and a second network branch. The feature extraction sub-network extracts features from the image to be detected, obtaining a feature map. For example, the feature extraction sub-network can use the relatively lightweight MobileNet V3 network, making it usable even on low-performance mobile devices. The first network branch obtains a first feature based on the feature map, fuses a second feature into the first feature to obtain an updated first feature, and determines the first morphological parameters of the target object based on the updated first feature. The first network branch can use a common network for image feature extraction, such as a convolutional neural network (CNN).

[0059] The second network branch is used to determine the keypoint map data of the target object based on the feature map, and to obtain the second feature based on the keypoint map data. The first feature is then fused into the second feature to obtain the updated second feature, and the second morphological parameters of the target object are determined based on the updated second feature. The second network branch can consist of two sub-networks: one for predicting the location information of the keypoints of the target object based on the feature map, and the other for extracting features from the keypoint connection map obtained from the keypoint map data to obtain the second feature. For the sub-network used to predict the location information of the keypoints of the target object, pre-training can be performed before training the neural network to make the determined keypoint location information more accurate, thereby accelerating the training speed during the neural network training process. For the sub-network used to extract features from the keypoint connection map, since it extracts features from graph data, a graph neural network (GCN) can be selected.

[0060] In some embodiments, the method for determining the morphological parameters of a target object from an image to be detected can be implemented by a pre-trained neural network. In order to improve the prediction accuracy of the trained neural network, when training the neural network, the target loss constructed can not only consider the deviation between the prediction results and the actual results of the two network branches, but also further use the deviation of the information on the key point positions of the target object obtained through the two network branches as a constraint, so that the two network branches can be associated, making the trained neural network more accurate.

[0061] For example, a neural network can be trained in the following way: First, a sample image carrying labeled information, including the target object, can be acquired. The labeled information is used to indicate the true value of the morphological parameters of the target object. Then, the sample image can be input into a preset initial neural network. The initial neural network determines the first position information of the key points of the target object, the predicted value of the morphological parameters of the target object, and the parameters of the image acquisition device that acquired the sample image. Then, the three-dimensional model corresponding to the target object can be determined based on the predicted value. The three-dimensional model is then projected based on the parameters of the image acquisition device to obtain the second position information of the key points after projection. The target loss is constructed based on the deviation between the first position information and the second position information. The initial neural network is then trained using the target loss to obtain the trained neural network.

[0062] The neural network extracts image features through two branches and then predicts morphological parameters based on these features. The second branch, in predicting morphological parameters, needs to determine the location information of 2D keypoints of the target object. The first branch predicts the morphological parameters of the target object in the image, as well as the parameters of the image acquisition device (e.g., intrinsic and extrinsic parameters). Based on the predicted morphological parameters, it determines the corresponding 3D model of the target object and projects the 3D model onto the image based on the intrinsic and extrinsic parameters of the image acquisition device, obtaining the location information of the projected 2D keypoints. Since both branches can obtain the location information of the target object's 2D keypoints, the deviation between the keypoint location information obtained by the two branches can be used to constrain the target loss. By correlating the outputs of the two branches, the accuracy of the trained neural network can be improved.

[0063] To further explain the method for detecting the morphological parameters of a target object in an image provided in this disclosure, the following explanation is based on a specific embodiment.

[0064] In live streaming and other applications, it's often necessary to input an image showing the user's pose into a virtual avatar platform, which then drives the virtual avatar to mimic the user's pose in real-time. This requires accurate prediction of the user's shape and pose parameters within the image. Previous techniques either employed complex networks (such as ResNet50) to extract features from the image and then predicted the user's shape and pose parameters based on these features. This approach is computationally intensive, time-consuming, and unsuitable for real-time operation. Other methods determine the location of key points on the user's limbs in the image, extract features from these key points, and predict the user's shape and pose parameters. However, this method suffers from inaccurate predictions due to the loss of depth information.

[0065] Based on this, this embodiment provides a method that can accurately predict human shape parameters and posture parameters in an image, with low computational load, can be used on ordinary mobile terminals, and has low latency, meeting the real-time requirements in live streaming scenarios.

[0066] In this embodiment, the model structure used to predict human shape parameters and pose parameters in an image is as follows: Figure 4 As shown, the model is mainly divided into three parts: a feature extraction network, a keypoint feature network branch, and an image feature network branch. The feature extraction network is used to extract features from the image, obtaining feature maps. The extracted feature maps can be input into the keypoint feature network branch and the image feature network branch. Both of these network branches will extract features from the feature maps, obtaining keypoint features and image features, and predict shape parameters and pose parameters based on the extracted features, respectively. Both network branches include a Transformer module, which is used to interact the keypoint features and image features extracted by the two network branches, so that each feature can affect the other, thereby improving the accuracy of the final prediction result. Even using a lightweight model to extract features, it can accurately predict human shape parameters and motion parameters, achieving accurate prediction results with relatively low computational cost.

[0067] The following sections introduce the structure and function of the feature extraction network, the key point feature network branch, the image feature network branch, and the Transformer module.

[0068] 1. Feature Extraction Network

[0069] To reduce computational cost and achieve real-time performance, the feature extraction network can employ the lightweight MobileNet V3 network. Since the features extracted by the keypoint feature network branch and the image feature network branch interact, the final features obtained are more accurate. Therefore, even with the lighter MobileNet V3 network, accurate shape and pose parameters can still be obtained.

[0070] 2. Key Point Feature Network Branches

[0071] The keypoint branch can be divided into two parts. The first part is the decoder, which predicts the location information of 2D keypoints of the human body based on the feature map output by the feature extraction network. The second part is the Graph Convolutional Network (GCN), which predicts pose parameters based on the location information and connection relationships of the 2D keypoints. and shape parameters .

[0072] Before training the entire model, the decoder can be pre-trained to make the location information of the 2D key points predicted by the decoder more accurate, thereby speeding up the training process when training the entire model.

[0073] The decoder can use three deconvolutional network modules. During pre-training, the input is a feature map, and the output is a 56×56 resolution keypoint prediction heatmap. Then, a loss function is calculated using the actual keypoint data. :

[0074]

[0075] in Indicates the number of 2D key points. This represents the location information of real 2D key points. This represents the predicted location information of 2D keypoints. Through continuous training with a loss function, the decoder's predicted keypoint heatmap becomes increasingly accurate. For the keypoint heatmap, the coordinates of the keypoints can be quickly obtained by taking the coordinates of the maximum response value.

[0076] After obtaining the location information of 2D keypoints of the human body using a decoder, the connections between these keypoints are pre-defined. Therefore, a keypoint connection map of the human limbs can be determined based on the location information and connections of these 2D keypoints. Graph Neural Networks (GCNs) can extract features from the keypoints of the human limbs, obtaining keypoint features. These features are then interacted with image features extracted by another network branch, and further feature extraction is performed on the interacting features. The extracted features are then used to predict the human body's pose and shape parameters. Specifically, a graph convolutional network can include three graph convolutional modules and one Transformer module. Two graph convolutional modules are placed before the Transformer module to extract features from the keypoint connection map, and one is placed after the Transformer module to further extract features from the interacting features. These features are then passed through a global pooling layer and a fully connected network to obtain the human body's pose parameters predicted by the keypoint feature network branch. and shape parameters .

[0077] 3. Image Feature Network Branch

[0078] The input to the image feature network branch is the feature map extracted by the feature extraction network, and the output is the predicted human pose parameters. and shape parameters The image feature map branch is also divided into two sub-modules by the Transformer module, one before and one after. The first module reshapes the feature map to unify the dimensions of the image features and the keypoint features, facilitating the interaction and fusion of the two features by the Transformer module. The second sub-module further extracts features from the interacted features, then processes them through batch normalization and global average pooling layers, and finally passes them through a fully connected network to predict the human pose parameters. and shape parameters .

[0079] 4. Transformer module

[0080] Considering that the features obtained from different network branches can be further improved through interaction, this embodiment designs a Transformer module in each of the two network branches of the model to interact the features extracted by the two network branches, so as to obtain image features after being affected by key point features, and key point features after being affected by image features.

[0081] The Transformer module can obtain keypoint features extracted from the keypoint feature network branch. Image features extracted by the image feature network branch For the Transformer module in the image feature network branch, it can transform image features Keypoint features as Q-query vectors As the K-key vector and V-value vector, for the Transformer module in the keypoint feature network branch, it can represent keypoint features. Image features as Q-query vectors As K-bond vectors and V-value vectors, then based on image features and key features The attention matrix is ​​determined to obtain new features indicating "attention drawn by the attention mechanism of the other branch". Then, a residual and normalization layer is added, followed by a fully connected layer and another normalization layer to obtain the final image features output by the module. Key features of module output The features influenced by the other party's features can be determined using the following formula:

[0082]

[0083]

[0084] in, Indicates key feature Image features after influence , Indicates the image features Keypoint features after feature influence . , A vector representing the image features. This represents the transpose of the vector corresponding to the image feature. , This represents the vector corresponding to the keypoint features. This represents the transpose of the vector corresponding to the key feature.

[0085] The softmax function scales the attention signal to the range of 0-1, where C is a scaling exponent. Softmax function:

[0086]

[0087] After determining the shape and pose parameters using two network branches, the pose and shape parameters of the two branches can be fused to produce a more accurate and stable result. The specific calculation formula for fusion is as follows:

[0088]

[0089]

[0090] in, Indicates the shape parameters after fusion. This represents the fused attitude parameters. , These represent the shape and pose parameters output by the branches of the image feature network, respectively. , These represent the shape parameters and pose parameters output by the keypoint feature network branches, respectively. This represents the weights corresponding to the output of the image feature network. Extensive experiments have shown... The best results can be achieved at that time.

[0091] Corresponding to the above method, this application also provides an apparatus for detecting morphological parameters of a target object in an image. The morphological parameters include the shape parameters and / or pose parameters of the target object. The apparatus is as follows: Figure 5 As shown, it includes:

[0092] Acquisition module 51 is used to acquire the image to be detected, including the target object;

[0093] The first feature extraction module 52 is used to extract features from the image to be detected to obtain a first feature;

[0094] The second feature extraction module 53 determines the key point map data of the target object based on the image to be detected, and obtains a second feature based on the key point map data, wherein the key point map data includes the location information of the key points and the connection relationship between the key points;

[0095] The first prediction module 54 is used to fuse the second feature into the first feature to obtain an updated first feature, and to determine the first morphological parameter of the target object based on the updated first feature.

[0096] The second prediction module 55 is used to fuse the first feature into the second feature to obtain an updated second feature, and to determine the second morphological parameter of the target object based on the updated second feature, so as to obtain the target morphological parameter according to the first morphological parameter and the second morphological parameter.

[0097] Furthermore, embodiments of this application also provide an electronic device, such as... Figure 6As shown, the electronic device includes a processor 61, a memory 62, and a computer program stored in the memory 62 that can be executed by the processor 61. When the processor 61 executes the computer program, it implements the method described in any of the above embodiments.

[0098] Accordingly, this application also provides a computer storage medium storing a program, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0099] The embodiments of this specification may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0100] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0101] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the description disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0102] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0103] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for detecting morphological parameters of a target object in an image, characterized in that, The morphological parameters include the shape parameters and / or pose parameters of the target object, and the method includes: Acquire the image to be detected, which includes the target object; Feature extraction is performed on the image to be detected to obtain the first feature; Based on the image to be detected, key point map data of the target object is determined, and a second feature is obtained based on the key point map data, wherein the key point map data includes the location information of the key points and the connection relationship between the key points; A first weight coefficient group is determined based on the first feature and the second feature, and the first weight coefficient group is used to characterize the degree of influence of the second feature on the first feature. The second feature is weighted using the first set of weight coefficients, and the weighted second feature is summed with the first feature to obtain the updated first feature. The first morphological parameter of the target object is determined based on the updated first feature; A second weight coefficient group is determined based on the first feature and the second feature, and the second weight coefficient group is used to characterize the degree of influence of the first feature on the second feature. The first feature is weighted using the second weight coefficient group, and the weighted first feature is summed with the second feature to obtain the updated second feature. The second morphological parameter of the target object is determined based on the updated second feature, so as to obtain the target morphological parameter according to the first morphological parameter and the second morphological parameter.

2. The method according to claim 1, characterized in that, The first feature and the second feature are represented by vectors, and the first set of weight coefficients is determined based on the product of the vector representing the first feature and the transpose of the vector representing the second feature; and / or The second set of weight coefficients is determined based on the product of the vector representing the second feature and the transpose of the vector representing the first feature.

3. The method according to claim 1, characterized in that, The determination of key point map data of the target object based on the image to be detected includes: Feature extraction is performed on the image to be detected to obtain a feature map of the image to be detected; The location information of the key points in the image to be detected is determined based on the feature map; The key point map data is obtained based on the location information of the key points and the preset connection relationship of the key points.

4. The method according to claim 1, characterized in that, Determining the first morphological parameter of the target object based on the updated first feature includes: After obtaining the updated first feature, further feature extraction is performed on the updated first feature, and the first morphological parameter of the target object is determined based on the extracted features; and / or Determining the second morphological parameters of the target object based on the updated second feature includes: After obtaining the updated second feature, further feature extraction is performed on the updated second feature, and the second morphological parameters of the target object are determined based on the extracted features.

5. The method according to claim 1, characterized in that, Before obtaining the updated first feature or the updated second feature, the process also includes: The representation information of the first feature and / or the representation information of the second feature are deformed to make the dimension of the representation information of the first feature consistent with the dimension of the representation information of the second feature.

6. The method according to claim 1, characterized in that, The step of obtaining the target morphological parameters based on the first morphological parameter and the second morphological parameter includes: Determine the first weight corresponding to the first morphological parameter and the second weight corresponding to the second morphological parameter; The first morphological parameter and the second morphological parameter are weighted and summed based on the first weight and the second weight to obtain the target morphological parameter.

7. The method according to claim 1, characterized in that, After obtaining the target shape parameters based on the first shape parameters and the second shape parameters, the method further includes: A driving signal is generated based on the target morphological parameters; The driving signal is used to drive a preset three-dimensional model to present a shape that matches the target object in the image to be detected.

8. The method according to claim 1, characterized in that, The method is implemented using a pre-trained neural network, which includes: A feature extraction subnetwork is used to extract features from the image to be detected, thereby obtaining a feature map of the image to be detected; The first network branch is used to obtain the first feature based on the feature map, determine the first weight coefficient group based on the first feature and the second feature, the first weight coefficient group is used to characterize the degree of influence of the second feature on the first feature, perform weighted processing on the second feature using the first weight coefficient group, sum the weighted second feature with the first feature to obtain the updated first feature, and determine the first morphological parameter of the target object based on the updated first feature. The second network branch is used to determine the key point map data of the target object based on the feature map, and to obtain the second feature based on the key point map data. Based on the first feature and the second feature, a second weight coefficient group is determined. The second weight coefficient group is used to characterize the degree of influence of the first feature on the second feature. The first feature is weighted using the second weight coefficient group. The weighted first feature is summed with the second feature to obtain the updated second feature. The second morphological parameter of the target object is determined based on the updated second feature.

9. The method according to claim 1 or 8, characterized in that, The method is implemented using a pre-trained neural network, which is trained in the following manner: Acquire a sample image carrying label information, wherein the sample image includes the target object, and the label information is used to indicate the true value of the morphological parameters of the target object; The sample image is input into a preset initial neural network to determine the first location information of the key points of the target object, the predicted value of the morphological parameters of the target object, and the parameters of the image acquisition device for acquiring the sample image. Based on the predicted value, a three-dimensional model corresponding to the target object is determined, and the three-dimensional model is projected based on the parameters of the image acquisition device to obtain the second position information of the key points after projection. A target loss is constructed based on the deviation between the first position information and the second position information, and the initial neural network is trained using the target loss to obtain the neural network.

10. An apparatus for detecting morphological parameters of a target object in an image, characterized in that, The morphological parameters include the shape parameters and / or posture parameters of the target object, and the device includes: The acquisition module is used to acquire the image to be detected, including the target object; The first feature extraction module is used to extract features from the image to be detected to obtain a first feature; The second feature extraction module determines the key point map data of the target object based on the image to be detected, and obtains the second feature based on the key point map data, wherein the key point map data includes the location information of the key points and the connection relationship between the key points; The first prediction module is used to determine a first weight coefficient group based on the first feature and the second feature. The first weight coefficient group is used to characterize the degree of influence of the second feature on the first feature. The second feature is weighted using the first weight coefficient group. The weighted second feature is summed with the first feature to obtain the updated first feature. The first morphological parameter of the target object is determined based on the updated first feature. The second prediction module is used to determine a second weight coefficient group based on the first feature and the second feature. The second weight coefficient group is used to characterize the degree of influence of the first feature on the second feature. The first feature is weighted using the second weight coefficient group. The weighted first feature is summed with the second feature to obtain an updated second feature. The second morphological parameter of the target object is determined based on the updated second feature, so as to obtain the target morphological parameter according to the first morphological parameter and the second morphological parameter.

11. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a computer program stored in the memory that can be executed by the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1-9.

Images