Method, device and equipment for determining key points of object

Abstract

The invention provides a method, a device and equipment for determining key points of an object. The specific implementation scheme is as follows: acquiring an original image; intercepting a target image at least comprising a part of the target object from the original image; determining a global feature vector and a heat distribution diagram of the target image; predicting a first key point coordinate set of the target object based on the popularity distribution map; wherein the first key point coordinate set comprises a first prediction coordinate of each key point of the target object; predicting a second key point coordinate set of the target object based on the global feature vector; wherein the second key point coordinate set comprises a second prediction coordinate of each key point of the target object; and fusing the first key point coordinate set and the second key point coordinate set to obtain coordinates of each key point of the target object.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, device and apparatus for determining key points of an object. Background Art

[0002] In the fields of motion analysis, human-computer interaction, motion capture, etc., two-dimensional (2D) key point detection of objects such as human body, hands, animals, and faces plays an important role. Summary of the invention

[0003] In a first aspect, an embodiment of the present disclosure provides a method for determining key points of an object, comprising: acquiring an original image; intercepting a target image including at least a portion of a target object from the original image; determining a global feature vector and a heat distribution map of the target image; based on the heat distribution map, predicting a first key point coordinate set of the target object; wherein the first key point coordinate set includes first predicted coordinates of each key point of the target object; based on the global feature vector, predicting a second key point coordinate set of the target object; wherein the second key point coordinate set includes second predicted coordinates of each key point of the target object; and fusing the first key point coordinate set and the second key point coordinate set to obtain the coordinates of each key point of the target object.

[0004] In a second aspect, an embodiment of the present disclosure provides a method for training a key point regression model, comprising: obtaining an image sample set; wherein the image sample set includes multiple training samples, each training sample includes a sample image, a sample screenshot taken from the sample image and including at least a portion of a preset object, and the true coordinates of each key point of the preset object; performing the following operations through the model to be trained until the first stage of training of the model to be trained is completed, and a heat map regression model is obtained: for any training sample in any first batch, determining a first sample heat distribution map of the sample screenshot in the training sample; wherein each first batch includes at least one training sample in the image sample set; using the true coordinates of each key point of the preset object in the training sample, generating a true heat distribution map corresponding to the sample screenshot; based on the true heat distribution map corresponding to the sample screenshot and the first sample heat distribution map, determining a first loss of the training sample; based on the first loss of each training sample in the first batch, obtaining a heat map loss of the first batch; based on the heat map loss of the first batch, updating the network parameters of the model to be trained to obtain an updated model to be trained; performing the following operations through the model to be trained until the second stage of training of the model to be trained is completed , a key point regression model is obtained, and the model to be trained is initially a heat map regression model: for any training sample in any second batch, a sample global feature vector and a second sample heat distribution map of the sample screenshot in the training sample are determined; wherein each second batch includes at least one training sample in the image sample set; based on the second sample heat distribution map, a first sample key point coordinate set of a preset object is predicted; wherein the first sample key point coordinate set includes the first sample predicted coordinates of each key point of the preset object; based on the sample global feature vector, a second sample key point coordinate set of the preset object is predicted; wherein the second sample key point coordinate set includes the second sample predicted coordinates of each key point of the preset object; the first sample key point coordinate set and the second sample key point coordinate set are fused to obtain the sample coordinates of each key point of the preset object; based on the first sample key point coordinate set, the second sample key point coordinate set, the sample coordinates of each key point of the preset object and the true value coordinates, a second loss of the training sample is determined; based on the second loss of each training sample in the second batch, a model loss of the second batch is obtained; based on the model loss of the second batch, the network parameters of the model to be trained are updated to obtain an updated model to be trained.

[0005] In a third aspect, an embodiment of the present disclosure provides a device for determining key points of an object, comprising: an image acquisition module for acquiring an original image; a first processing module for intercepting a target image including at least a portion of a target object from the original image; a second processing module for determining a global feature vector and a heat distribution map of the target image; a third processing module for predicting a first key point coordinate set of the target object based on the heat distribution map; wherein the first key point coordinate set includes first predicted coordinates of each key point of the target object; a fourth processing module for predicting a second key point coordinate set of the target object based on the global feature vector; wherein the second key point coordinate set includes second predicted coordinates of each key point of the target object; and a fifth processing module for fusing the first key point coordinate set and the second key point coordinate set to obtain the coordinates of each key point of the target object.

[0006] In a fourth aspect, an embodiment of the present disclosure provides an apparatus for training a key point regression model, comprising: a sample acquisition module, for acquiring an image sample set; wherein the image sample set comprises a plurality of training samples, each training sample comprises a sample image, a sample screenshot taken from the sample image and comprising at least a portion of a preset object, and true coordinates of each key point of the preset object; a first training module, for performing the following operations through the model to be trained until the first stage of training of the model to be trained is completed to obtain a heat map regression model: for any training sample in any first batch, determining a first sample heat score of the sample screenshot in the training sample; wherein each first batch includes at least one training sample in the image sample set; using the true value coordinates of each key point of the preset object in the training sample, generating a true value heat distribution map corresponding to the sample screenshot; based on the true value heat distribution map corresponding to the sample screenshot and the first sample heat distribution map, determining the first loss of the training sample; based on the first loss of each training sample in the first batch, obtaining the heat map loss of the first batch; based on the heat map loss of the first batch, updating the network parameters of the to-be-trained model, obtaining the updated to-be-trained model; a second training module, used to perform the following operations through the to-be-trained model, until When the second stage of training of the model to be trained is completed, a key point regression model is obtained, and the model to be trained is initially a heat map regression model: for any training sample in any second batch, a sample global feature vector and a second sample heat distribution map of the sample screenshot in the training sample are determined; wherein each second batch includes at least one training sample in the image sample set; based on the second sample heat distribution map, a first sample key point coordinate set of a preset object is predicted; wherein the first sample key point coordinate set includes the first sample predicted coordinates of each key point of the preset object; based on the sample global feature vector, a second sample key point coordinate set of the preset object is predicted; wherein the second sample key point coordinate set includes the second sample predicted coordinates of each key point of the preset object; the first sample key point coordinate set and the second sample key point coordinate set are fused to obtain the sample coordinates of each key point of the preset object; based on the first sample key point coordinate set, the second sample key point coordinate set, the sample coordinates of each key point of the preset object and the true value coordinates, the second loss of the training sample is determined; based on the second loss of each training sample in the second batch, the model loss of the second batch is obtained; based on the model loss of the second batch, the network parameters of the model to be trained are updated to obtain an updated model to be trained.

[0007] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium storing a computer program for executing any one of the methods for determining object key points provided in any of the above embodiments of the present disclosure and any one of the methods for training a key point regression model provided in any of the above embodiments.

[0008] In a sixth aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; a memory for storing processor executable instructions; a processor, configured to read executable instructions from the memory and execute the instructions to implement any of the methods for determining object key points provided in any of the above embodiments of the present disclosure and any of the methods for training a key point regression model provided in any of the above embodiments.

[0009] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The embodiments of the present disclosure are described in more detail with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 is an exemplary system architecture that can be applied to embodiments of the method or apparatus for determining key points of an object disclosed herein;

[0012] Figure 2 is a flowchart of a method for determining key points of an object provided by some exemplary embodiments of the present disclosure;

[0013] Figure 3 is a schematic diagram of a process of determining a heat distribution map of a target image provided by some exemplary embodiments of the present disclosure;

[0014] Figure 4 is a schematic diagram of a process for determining a target feature map provided by some exemplary embodiments of the present disclosure;

[0015] Figure 5 is a schematic diagram of a network structure of a key point regression model provided by some example embodiments of the present disclosure;

[0016] Figure 6 is a flowchart of a method for training a key point regression model provided by some exemplary embodiments of the present disclosure;

[0017] Figure 7 is a schematic diagram of the training process of a heat map regression model provided by some exemplary embodiments of the present disclosure;

[0018] Figure 8 is a flowchart for training a key point regression model provided by some exemplary embodiments of the present disclosure;

[0019] Fig. 9 is a schematic diagram of a network structure of a key point regression model provided by other exemplary embodiments of the present disclosure;

[0020] Fig.10 is a schematic diagram of a process of obtaining an image sample set provided by some exemplary embodiments of the present disclosure;

[0021] Fig.11 is a flowchart of a method for training a key point regression model provided by some other exemplary embodiments of the present disclosure;

[0022] Fig.12 is a flowchart for training a key point regression model provided by some other exemplary embodiments of the present disclosure;

[0023] Fig.13 is a schematic diagram of the structure of an apparatus for determining key points of an object provided by some exemplary embodiments of the present disclosure;

[0024] Fig.14 is a schematic diagram of the structure of an apparatus for training a key point regression model provided by some exemplary embodiments of the present disclosure;

[0025] Fig.15 is a schematic diagram of the structure of an apparatus for training a key point regression model provided by some other exemplary embodiments of the present disclosure;

[0026] Fig.16 It is a structural schematic diagram of an application embodiment of the electronic device disclosed in the present invention. DETAILED DESCRIPTION

[0027] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0028] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0029] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0030] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0031] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0032] The embodiments of the present disclosure can be applied to electronic devices such as head-mounted display devices, terminal devices, and servers, which can operate with many other general or special computing system environments or configurations. Electronic devices such as head-mounted display devices, terminal devices, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. The computer system / server can be implemented in a distributed cloud computing environment, in which tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0033] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0034] Figure 1 It is an exemplary system architecture that can be applied to an embodiment of the method or apparatus for determining key points of an object disclosed in the present invention. The system architecture may include a head mounted display device 1, a network 2, and a server 3. The network 2 may be a medium that provides a communication link between the head mounted display device 1 and the server 3. The network 2 may include various connection types, such as wired or wireless communication links. The wired communication link may be, for example, an optical fiber, a cable, etc. The wireless communication link may be, for example, WiFi, Bluetooth, etc.

[0035] The head-mounted display device 1 may be an electronic device with an image display function, including but not limited to AR (Augmented Reality) smart glasses, VR (Virtual Reality) smart glasses, and the like. The head-mounted display device 1 may be an all-in-one device, in which the function of a computing unit is integrated, and the computing unit may provide processing capabilities for the head-mounted display device, and various client applications installed on the head-mounted display device are actually installed in the computing unit. Alternatively, the head-mounted display device may also be a split device, in which the split head-mounted display device includes a head-mounted display and a computing unit that is separately arranged from the head-mounted display, and the display screen of the head-mounted display is used to display the image processed by the computing unit. The computing unit in the split head-mounted display device may be a terminal device, and the terminal device may be a mobile phone, a handle, an adapter, a computer, a tablet, a car computer, and the like. Here, for the convenience of representation, in the case where the execution subject is a head-mounted display device, the following text will introduce the scheme of the present disclosure using the all-in-one head-mounted display device as the action execution subject, and the head-mounted display device 1 is used to represent it in the figure. Those skilled in the art should understand that, for a split-type head-mounted display device, the execution entity is the computing unit.

[0036] The head mounted display device 1 can provide various services, for example, the method for determining the key points of an object disclosed in the present invention. When the original image is acquired, a target image including at least a part of the target object can be intercepted from the original image; the global feature vector and the heat distribution map of the target image can be determined; based on the heat distribution map, a first key point coordinate set of the target object is predicted, wherein the first key point coordinate set includes the first predicted coordinates of each key point of the target object; based on the global feature vector, a second key point coordinate set of the target object is predicted, wherein the second key point coordinate set includes the second predicted coordinates of each key point of the target object; the first key point coordinate set and the second key point coordinate set are fused to obtain the coordinates of each key point of the target object. The method for determining the key points of an object provided in the embodiment of the present invention is usually performed by the head mounted display device 1 mentioned above, and accordingly, the device for determining the key points of the object is usually provided in the head mounted display device 1.

[0037] It should also be pointed out that the method for determining the key points of an object provided by the present disclosure can also be applied to the server 3, which can be a background server. When the method of the present disclosure is applied to the server 3, the server 3 can obtain the original image from the head-mounted display device, and can obtain the coordinates of each key point of the target object based on the original image. Afterwards, the head-mounted display device 1 can perform subsequent processing based on the coordinates of each key point of the target object, such as performing posture estimation on the target object based on the coordinates of each key point of the target object. Accordingly, the device for determining the key points of an object can also be set in the server 3. It is also possible that the server performs subsequent processing based on the coordinates of each key point of the target object, which is not specifically limited.

[0038] In addition to the server and the head mounted display device, the method for determining the key points of an object can also be applied to other terminal devices (not shown in the figure) connected to the head mounted display device. The other terminal devices here can be mobile phones, tablets, computers, etc.

[0039] Exemplary Methods

[0040] Figure 2 is a flowchart of a method for determining key points of an object provided by some exemplary embodiments of the present disclosure. The embodiments of the present disclosure can be applied to electronic devices, such as Figure 2 As shown, the method comprises the following steps:

[0041] Step 210, obtaining an original image.

[0042] In some optional embodiments of the present disclosure, the original image can be acquired by collecting through a terminal device with a camera, and the terminal device can be a head-mounted display device, a mobile phone, a tablet, etc. with a camera.

[0043] In some optional embodiments of the present disclosure, the original image may include at least one target object, and for each target object, the original image may include all or part of each key point of the target object. The target object may be any one of objects such as a human body, a hand, an animal, a human face, etc.

[0044] The execution body of the method for determining the key points of the object (e.g., Figure 1 When the key points of the target object in the image need to be identified, the original image is obtained. Figure 1 In the case where the head mounted display device 1 is used as the execution subject of the method for determining the key points of the object, the original image can be acquired by the head mounted display device 1 itself. Figure 1 In the case where the server 3 in the method is used as the execution subject of the method for determining the key points of the object, the original image required by the server 3 can be obtained from the head-mounted display device 1. In the case where other terminal devices connected to the head-mounted display device are used as the execution subject of the method for determining the key points of the object, the original image required by the other terminal devices can be obtained from the head-mounted display device.

[0045] Step 220: extracting a target image including at least a portion of the target object from the original image.

[0046] In some optional embodiments of the present disclosure, the target object can be detected by performing target detection on the original image to obtain a detection frame corresponding to the target object, and then the image of the detection frame area can be intercepted from the original image as the target image based on the detection frame. The target image includes at least a part of the target object, which means that at least a part of the pixels occupied by the target object in the original image is circled in the target image, and there may be pixels occupied by the target object outside the target image. The completeness of the target object included in the target image may be related to the accuracy of the detection frame. For example, if the error of the detection frame is large, the intercepted target image may only include a part of the pixels occupied by the target object in the original image. If the detection frame is more accurate, the intercepted target image may include all the pixels occupied by the target object in the original image. The target object may not be complete in the original image, such as half of a face, the upper body of a human body, etc. in the original image.

[0047] Step 230, determining the global feature vector and heat distribution map of the target image.

[0048] In some optional embodiments of the present disclosure, the global feature vector may be a vector including global features of the target image. For example, the extracted feature map of the target image is subjected to any of global pooling processing, convolution processing, etc. to obtain the global feature vector of the target image.

[0049] In some optional embodiments of the present disclosure, the heat distribution map is a distribution probability map of key points generated based on a Gaussian distribution function generation method. The heat distribution map may include probability maps corresponding to each key point of the target object. For example, for human body key points, 16 key points may be included to generate heat distribution maps of 16 channels, each channel corresponding to a key point. For each key point, the heat distribution map of the channel corresponding to the key point generates a probability area with a Gaussian distribution at the position corresponding to the key point, and the central probability value of the area is the largest and closest to 1, and the probability is smaller as it is around.

[0050] It should be noted that determining the global feature vector of the target image and determining the heat distribution map of the target image in the above step 230 have no execution order dependency and can be executed simultaneously or successively without any restriction on the execution order.

[0051] Step 240 , predicting a first key point coordinate set of the target object based on the heat distribution map.

[0052] In some optional embodiments of the present disclosure, the first key point coordinate set includes first predicted coordinates of each key point of the target object.

[0053] In some optional embodiments of the present disclosure, since the heat distribution map represents the distribution probability of each key point of the target object, the first key point coordinate set of the target object can be predicted based on the heat distribution map.

[0054] In some optional embodiments of the present disclosure, the first predicted coordinates of each key point of the target object included in the first key point coordinate set may be the pixel coordinates of each key point in the pixel coordinate system of the target image, or may be the pixel coordinates of each key point in the pixel coordinate system of the original image, without specific limitation. The pixel coordinate system of the target image and the pixel coordinate system of the original image may be converted to each other. For example, when capturing the target image from the original image, the positional relationship between the target image and the original image may be recorded, and the conversion between the pixel coordinate systems of the two may be achieved based on the positional relationship.

[0055] In some optional embodiments of the present disclosure, for key points in the first key point coordinate set that are located in the target image, an accurate and effective heat distribution map can be generated based on the features of the key points in the target image, so the coordinates of the key points located in the target image can have a higher accuracy. For key points located outside the target image, since there is no relevant feature information of the key point in the target image, the center probability of the heat distribution map of the channel corresponding to the key point is small, so that the regressed coordinates of the key point may be preset values, for example, the coordinates of the key point obtained are (0,0).

[0056] In some optional embodiments of the present disclosure, when a heat distribution map has been obtained, the first key point coordinate set of the target object can be predicted based on the heat distribution map by any feasible prediction method. For example, the heat distribution map can be processed by any key point regression network to obtain the first key point coordinate set. The key point regression network can be, for example, an IPR (Integral Pose Regression) network.

[0057] Step 250: predict a second key point coordinate set of the target object based on the global feature vector.

[0058] In some optional embodiments of the present disclosure, the second key point coordinate set includes second predicted coordinates of each key point of the target object.

[0059] In some optional embodiments of the present disclosure, the second key point coordinate set of the target object can be obtained by prediction through any feasible prediction network. For example, the key point coordinates of the target object can be predicted based on a linear prediction network to obtain the second key point coordinate set.

[0060] It should be noted that the above steps 240 and 250 have no dependency on the execution order and can be executed simultaneously or successively without any restriction on the execution order.

[0061] Step 260: Fusing the first key point coordinate set and the second key point coordinate set to obtain the coordinates of each key point of the target object.

[0062] In some optional embodiments of the present disclosure, the fusion method of fusing the first key point coordinate set and the second key point coordinate set can adopt any feasible method. For example, for the key points located in the target image, the coordinates in the first key point coordinate set are adopted, and for the key points located outside the target image, the coordinates in the second key point coordinate set are adopted to obtain the coordinates of each key point of the target object. The first key point coordinate set and the second key point coordinate set can also be spliced, and the splicing result can be further fused through any of the output mapping networks such as linear layer, squeeze and excitation network (Squeeze-and-Excitation Networks, SE-NET for short) to obtain the coordinates of each key point of the target object. The present disclosure does not make specific limitations on this.

[0063] The method for determining the key points of an object provided by the embodiment of the present disclosure can determine the global feature vector and heat distribution map of the target image for a target image that includes at least a part of the target object, so that the first key point coordinate set of the target object can be predicted based on the heat distribution map, and the second key point coordinate set of the target object can be predicted based on the global feature vector. By fusing the first key point coordinate set and the second key point coordinate set, the coordinates of each key point of the target object are obtained. Among them, the coordinates of the key points of the target object in the target image and the coordinates of the key points of the target object that are not in the target image can be predicted by the global feature vector, and the coordinates of the key points of the target object in the target image can be at least predicted by the heat distribution map, and with high accuracy. Therefore, it can be achieved that when the target image includes a part of the target object, the coordinates of each key point of the target object can also be robustly and accurately estimated, which helps to improve the overall accuracy of the determined key point coordinates and improve the versatility of the method for determining the key points of the object.

[0064] Figure 3 It is a schematic diagram of a process of determining a heat distribution map of a target image provided by some exemplary embodiments of the present disclosure.

[0065] In some optional embodiments of the present disclosure, Figure 3 As shown, determining the heat distribution map of the target image in step 230 includes:

[0066] Step 2310, based on the target image, determine a target feature map of a preset resolution.

[0067] In some optional embodiments of the present disclosure, the preset resolution is lower than the resolution of the target image, and the scale of the target feature map is smaller than the scale of the target image.

[0068] In some optional embodiments of the present disclosure, the resolution of the target image represents the number of pixels in the height and width directions of the target image. For example, the resolution of the target image is represented by H*W, and the preset resolution can be represented by H / n*W / m, where n and m are both positive integers, and at least one of n and m is an integer greater than 1. For example, the preset resolution is H / 2*W / 2, H / 4*W / 4, H / 8*W / 8, etc.

[0069] In some optional embodiments of the present disclosure, the preset resolution may be a higher resolution that is smaller than the resolution of the target image. For example, considering the amount of calculation, if the feature map scale of H / 2*W / 2 is too large, the preset resolution may be set to H / 4*W / 4.

[0070] In some optional embodiments of the present disclosure, the scale of the target feature map is 1 / s of the scale of the target image, for example, the scale of the target feature map is 1 / 4, 1 / 8, 1 / 16, 1 / 32 of the scale of the target image. s can be any positive integer. s can also be 2 n , where n is a positive integer.

[0071] In some optional embodiments of the present disclosure, a target feature map of a preset resolution can be realized by extracting and fusing features of a target image using a feature extraction network combined with a feature fusion network. The feature extraction network can adopt any implementable network. For example, the feature extraction network can adopt any backbone network (Backbone) such as ResNet (residual network), DenseNet (densely connected network), Xception Model (a convolutional neural network), MobileNet (mobile network), etc. The feature fusion network can adopt FPN (Feature Pyramid Network), and fuse the extracted multi-scale features based on FPN to obtain a fused target feature map of a preset resolution. Feature extraction and fusion can also be implemented based on U-Net (U-type network) to obtain a target feature map of a preset resolution.

[0072] 2320, based on the target feature map, generate a heat distribution map of the target image.

[0073] In some optional embodiments of the present disclosure, the heat map can be generated by a pre-trained heat map regression model. The heat map regression model includes a feature extraction network and a feature fusion network for generating a target feature map, and a Gaussian distribution function for regressing the heat map from the target feature map. The distribution probability of the target feature map is calculated by the Gaussian distribution function to obtain the heat map of the target image.

[0074] This embodiment generates a heat distribution map of a target image through a target feature map of a preset resolution of the target image, and can obtain a higher resolution target feature map of the target image, which helps to improve the accuracy of the generated heat distribution map of the target image, and further helps to improve the accuracy of the key point coordinates of the determined target object.

[0075] In some optional embodiments of the present disclosure, determining a target feature map of a preset resolution based on the target image in step 2310 includes:

[0076] The target image is subjected to step-by-step feature extraction at multiple resolutions from high to low resolutions to obtain feature maps corresponding to the multiple resolutions. The preset resolution is the largest resolution among the multiple resolutions. The scales of the feature maps corresponding to the multiple resolutions decrease as the resolution decreases. The feature maps corresponding to the multiple resolutions are fused to obtain the target feature map of the preset resolution.

[0077] In some optional embodiments of the present disclosure, the number of resolutions included in the multiple resolutions is not limited. For example, the multiple resolutions may include 2 resolutions, 3 resolutions, 4 resolutions, etc.

[0078] In some optional embodiments of the present disclosure, feature extraction may be performed on the target image first to obtain a basic feature map corresponding to the target image, and the resolution of the basic feature map may be, for example, H / 2*W / 2. Downsample and feature extract the basic feature map to obtain a feature map corresponding to the first resolution among multiple resolutions. The first resolution may be, for example, H / 4*W / 4. The first resolution is used as the preset resolution. Downsample and feature extract the feature map corresponding to the first resolution to obtain a feature map corresponding to the second resolution. The second resolution may be, for example, H / 8*W / 8. Downsample and feature extract the feature map corresponding to the second resolution to obtain a feature map corresponding to the third resolution. The third resolution may be, for example, H / 16*W / 16. By analogy, feature maps corresponding to multiple resolutions are obtained, thereby realizing step-by-step feature extraction of multiple resolutions of the target image from high to low resolutions, and obtaining feature maps corresponding to multiple resolutions.

[0079] In some optional embodiments of the present disclosure, as the resolution decreases, the scale of the feature map corresponding to the obtained resolution decreases. For example, the resolution of the target image is H*W, and the multiple resolutions include H / 4*W / 4, H / 8*W / 8, H / 16*W / 16, and H / 32*W / 32. As the resolution decreases, the scale of the feature map is 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the scale of the target image, respectively.

[0080] In some optional embodiments of the present disclosure, the fusion of feature maps corresponding to multiple resolutions can be achieved based on upsampling and feature fusion. For example, in the order of resolution from low to high, the feature map corresponding to the lowest resolution is upsampled to obtain the upsampled feature map of the second lowest resolution. The upsampled feature map of the second lowest resolution is fused with the feature map corresponding to the second lowest resolution to obtain the fused feature map of the second lowest resolution. The fused feature map of the second lowest resolution is upsampled to obtain the upsampled feature map of the third lowest resolution. The upsampled feature map of the third lowest resolution is fused with the feature map corresponding to the third lowest resolution to obtain the fused feature map of the third lowest resolution. The fused feature map of the third lowest resolution is upsampled to obtain the upsampled feature map of the fourth lowest resolution. By analogy, the fused feature map corresponding to the third lowest resolution is completed. The fused feature map of the preset resolution is used as the target feature map, that is, the target feature map of the preset resolution is obtained.

[0081] In some optional embodiments of the present disclosure, feature maps corresponding to multiple resolutions can be implemented using a feature fusion network. The feature fusion network can be a neural network based on upsampling and summation, or a neural network based on upsampling, splicing and convolution. Wherein upsampling can be achieved, for example, by interpolation or deconvolution. For example, the FPN network can be used to continuously upsample the low-resolution feature map and add it with the adjacent feature map of a higher resolution until a target feature map of a preset resolution is obtained. The feature maps corresponding to each resolution are fused, and the feature map corresponding to the resolution and the upsampled feature map of the resolution can be added, or the feature map corresponding to the resolution and the upsampled feature map of the resolution can be spliced ​​and then convolved, etc.

[0082] This embodiment can obtain feature maps corresponding to multiple resolutions by performing step-by-step feature extraction on the target image according to the resolution from high to low, so as to obtain shallow features and deep features of the target image. Shallow features can provide more underlying information such as texture and color, and deep features can provide more semantic information, which helps to improve the richness of features. By fusing the feature maps corresponding to multiple resolutions, a target feature map with the highest preset resolution among multiple resolutions is obtained, and the feature maps corresponding to multiple resolutions are fused into a high-resolution target feature map, so that the target feature map integrates the shallow information and deep information of the target image and has a higher resolution. The target feature map is used to generate a heat distribution map of the target image, which helps to improve the precision and accuracy of the heat distribution map.

[0083] Figure 4 It is a schematic diagram of a flow chart of determining a target feature map provided by some exemplary embodiments of the present disclosure.

[0084] In some optional embodiments of the present disclosure, Figure 4 As shown, the feature maps corresponding to multiple resolutions are fused to obtain a target feature map of a preset resolution, including:

[0085] Step 310: Use the feature map corresponding to the lowest resolution among multiple resolutions as the initial feature map.

[0086] In some optional embodiments of the present disclosure, the number of channels of the feature map corresponding to the lowest resolution may be transformed to obtain the initial feature map. For example, a 1*1 convolution operation is performed on the feature map corresponding to the lowest resolution through a convolutional network layer, the number of channels of the feature map corresponding to the lowest resolution is transformed, and the convolution result is used as the initial feature map.

[0087] In some optional embodiments of the present disclosure, the feature map corresponding to the lowest resolution may also be used as the initial feature map.

[0088] Step 320: Select the second lowest resolution from the multiple resolutions as the current resolution.

[0089] In some optional embodiments of the present disclosure, feature maps corresponding to multiple resolutions may be sorted in order from low to high resolution, and the lowest resolution and the second lowest resolution may be determined based on the sorting.

[0090] Step 330 , upsampling the feature map to be upsampled at the previous resolution to obtain an upsampled feature map at the current resolution.

[0091] In some optional embodiments of the present disclosure, the current resolution is one level higher than the previous resolution among the multiple resolutions. The feature map to be upsampled of the previous resolution is initially an initial feature map.

[0092] In some optional embodiments of the present disclosure, the multiple resolutions include a first resolution, a second resolution, a third resolution, and a fourth resolution in order from low to high, and the first resolution is the lowest resolution. If the current resolution is the third resolution, the previous resolution is the second resolution. The upsampled feature map of the third resolution is obtained by upsampling the feature map to be upsampled of the second resolution. The feature map to be upsampled of the second resolution can be a fused feature map of the second resolution, and the fused feature map of the second resolution is obtained by fusing the upsampled feature map of the second resolution with the feature map corresponding to the second resolution.

[0093] Step 340, the upsampled feature map of the current resolution is fused with the feature map corresponding to the current resolution to obtain a fused feature map of the current resolution.

[0094] In some optional embodiments of the present disclosure, the fusion method for obtaining the fused feature map of the current resolution may be any one of concatenation (concat) + convolution, corresponding element addition (add), etc. The concatenation may be to concatenate the upsampled feature map of the current resolution with the feature map corresponding to the current resolution along the feature map channel direction. For example, the number of channels of the upsampled feature map of the current resolution is 256, and the number of channels of the feature map corresponding to the current resolution is 256. Then, the number of channels of the concatenated feature map of the current resolution obtained by concatenating the two is 512, the feature map resolution remains unchanged, and the number of channels is superimposed. Convolution can compress the concatenated feature map with a channel number of 512 into a fused feature map with a channel number of 256, and the scale of the fused feature map remains unchanged from before the fusion. The corresponding element addition refers to the addition of the features of the same pixel position of the upsampled feature map of the current resolution and the feature map corresponding to the current resolution, and the result of the addition is used as the fused feature corresponding to the pixel position in the fused feature map of the current resolution.

[0095] Step 350, determining whether the current resolution is a preset resolution.

[0096] If yes, in step 360, the fused feature map of the current resolution is used as the target feature map.

[0097] In some optional embodiments of the present disclosure, if the current resolution is a preset resolution, it means that the fusion of feature maps corresponding to multiple resolutions has been completed, and the fused feature map of the current resolution is used as the target feature map.

[0098] If not, in step 370, the current fused feature map is used as a new feature map to be upsampled at the previous resolution, a resolution one level higher than the current resolution is selected from multiple resolutions as the new current resolution, and step 330 is repeated to upsample the feature map to be upsampled at the previous resolution to obtain the upsampled feature map at the current resolution.

[0099] In some optional embodiments of the present disclosure, if the current resolution is not a preset resolution, it means that the fusion of feature maps corresponding to multiple resolutions has not been completed, and the current fused feature map can be used as a new feature map to be upsampled at the previous resolution. For example, if the current resolution is the third resolution, the current fused feature map is used as the feature map to be upsampled at the third resolution, and a resolution one level higher than the current resolution (for example, a fourth resolution one level higher than the third resolution) is selected from multiple resolutions as the new current resolution, then the current resolution becomes the fourth resolution, and step 330 is repeated to perform upsampling processing on the feature map to be upsampled at the previous resolution to obtain the steps and subsequent steps of the upsampled feature map of the current resolution. For example, the feature map to be upsampled at the new previous resolution, i.e., the third resolution, is upsampled to obtain a new feature map after upsampling at the current resolution, i.e., the fourth resolution, and the upsampled feature map of the fourth resolution is fused with the feature map corresponding to the fourth resolution to obtain a fused feature map of the fourth resolution. In response to the fourth resolution being the preset resolution, the fused feature map of the fourth resolution is used as the target feature map. In response to the fourth resolution not being the preset resolution, the fused feature map of the fourth resolution (i.e., the current fused feature map) is used as the new feature map to be upsampled at the previous resolution, a resolution one level higher than the current resolution (e.g., a fifth resolution one level higher than the fourth resolution) is selected from multiple resolutions as the new current resolution, and step 330 of upsampling the feature map to be upsampled at the previous resolution is repeatedly performed to obtain the upsampled feature map at the current resolution. This process is repeated until the current resolution is the preset resolution, and the fused feature map at the current resolution is used as the target feature map.

[0100] This embodiment performs feature fusion step by step from low to high according to multiple resolutions, so that shallow information and deep information in feature maps corresponding to multiple resolutions can be fused into the target feature map, which helps to further improve the accuracy of the heat distribution map.

[0101] In some optional embodiments of the present disclosure, determining the global feature vector of the target image in step 230 includes:

[0102] Step 2330, determining a global feature vector based on a feature map corresponding to the lowest resolution among multiple resolutions.

[0103] In some optional embodiments of the present disclosure, a global feature vector can be obtained by pooling, convolution, full connection, expansion, etc. For expansion, assuming that the feature map is C*h*w (three-dimensional), after expansion it is hwC*1*1, and k 1*1 convolution kernels are used to transform the dimension to obtain a k-dimensional global feature vector. Where h is the height of the feature map, w is the width of the feature map, and C represents the number of channels of the feature map. The present disclosure does not limit the specific mapping method.

[0104] This embodiment determines the global feature vector through the feature map corresponding to the lowest resolution among multiple resolutions. The feature map corresponding to the lowest resolution includes deep information of the target image, can provide more semantic information, and help improve the accuracy and effectiveness of the global feature vector, providing more accurate and effective global features for determining each key point of the target object.

[0105] In some optional embodiments of the present disclosure, the global feature vector in step 2330 is determined by any of the following:

[0106] Use the global pooling network to perform global pooling on the feature map corresponding to the lowest resolution to obtain a global feature vector. Use the convolution network to perform convolution on the feature map corresponding to the lowest resolution to obtain a global feature vector. Use the fully connected network to process the feature map corresponding to the lowest resolution to obtain a global feature vector. Expand the feature map corresponding to the lowest resolution to obtain a global feature vector.

[0107] In some optional embodiments of the present disclosure, a global pooling network (Global Pooling Layer) is used to perform global pooling processing on the feature map corresponding to the lowest resolution, pooling the height and width of the feature map of the lowest resolution to 1*1, and obtaining a global feature vector.

[0108] In some optional embodiments of the present disclosure, the global pooling network may include any one of a global average pooling network, a global maximum pooling network, and the like. The global average pooling network is used to perform global average pooling processing on the feature map corresponding to the lowest resolution to obtain a global feature vector. The global average pooling process is to average the h*w eigenvalues ​​of each channel in the feature map corresponding to the lowest resolution, where h and w respectively represent the height and width of the feature map corresponding to the lowest resolution, as the global eigenvalue corresponding to the channel in the global feature vector, thereby converting the C*h*w feature map into a C*1*1 tensor to obtain a C-dimensional global feature vector. The global maximum pooling network is used to perform global maximum pooling processing on the feature map corresponding to the lowest resolution to obtain a global feature vector. The global maximum pooling process refers to taking the maximum value of the h*w eigenvalues ​​of each channel in the feature map corresponding to the lowest resolution as the global eigenvalue of the channel in the global feature vector.

[0109] In some optional embodiments of the present disclosure, the role of convolution in the convolutional network includes changing the size of the convolved feature map. Based on this, the number of convolutional layers of the convolutional network, the size of the convolution kernel of each convolutional layer, the step size, the padding method and the number of channels can be set so that the convolutional network can map the feature map corresponding to the lowest resolution to a global feature vector. For example, the feature map corresponding to the lowest resolution can be globally convolved based on the convolution kernel of h*w to obtain a global feature vector.

[0110] In some optional embodiments of the present disclosure, the function of convolution also includes changing the number of channels of the convolved feature map. For example, the feature map corresponding to the lowest resolution is C*h*w, which is convolved with k convolution kernels of C*h*w (C is the number of channels of the convolution kernel, and h*w represents the convolution kernel size) to obtain k global eigenvalues, and a k-dimensional global eigenvector is obtained based on the k global eigenvalues.

[0111] In some optional embodiments of the present disclosure, the fully connected network obtains the global eigenvalue corresponding to the channel in the global eigenvector by performing weighted summation on the h*w eigenvalues ​​of each channel of the feature map corresponding to the lowest resolution.

[0112] This embodiment achieves the acquisition of global feature vectors through global pooling, convolution, full connection and other processing methods, which helps to provide effective global feature vectors for the prediction of each key point of the target object, thereby improving the accuracy of each key point obtained by prediction.

[0113] In some optional embodiments of the present disclosure, predicting a first key point coordinate set of the target object based on the heat distribution map in step 240 includes:

[0114] The heat distribution map is processed using a key point regression network to obtain the first key point coordinate set of the target object.

[0115] In some optional embodiments of the present disclosure, the key point regression network may be any feasible regression network, such as a convolution-based regression network, a fully connected-based regression network, and the like.

[0116] In some optional embodiments of the present disclosure, the key point regression network can be implemented using an IPR network.

[0117] This embodiment performs key point regression processing on the heat distribution map through a key point regression network to obtain a first key point coordinate set of the target object, which helps to improve the accuracy of the first key point coordinate set.

[0118] In some optional embodiments of the present disclosure, predicting a second key point coordinate set of the target object based on the global feature vector in step 250 includes:

[0119] The global feature vector is predicted using a linear prediction network to obtain a second key point coordinate set.

[0120] In some optional embodiments of the present disclosure, the linear prediction network may include at least one linear layer (Linear Layer), and the global feature vector is predicted by the linear prediction network to obtain a second key point coordinate set of the target object.

[0121] In some optional embodiments of the present disclosure, after the linear prediction network, the obtained tensor may be reshaped to obtain a second key point coordinate set. For example, the number of output channels of the linear prediction network is set to 2N, and the output tensor of the shape [2N] is reshaped to obtain a tensor of the shape [N, 2].

[0122] This embodiment performs prediction processing on the global feature vector through a linear prediction network to obtain a second key point coordinate set of the target object, which helps to improve the accuracy of the second key point coordinate set.

[0123] In some optional embodiments of the present disclosure, the step 260 of fusing the first key point coordinate set and the second key point coordinate set to obtain the coordinates of each key point of the target object includes:

[0124] Step 2610, splicing the first key point coordinate set and the second key point coordinate set to obtain a splicing result.

[0125] In some optional embodiments of the present disclosure, the first key point coordinate set and the second key point coordinate set may be spliced ​​into a larger key point coordinate set as a splicing result.

[0126] In some optional embodiments of the present disclosure, the first key point coordinate set can be represented as an N*2 tensor, where N represents the number of key points of the target object. For example, if the target object is a human body, and the human body key points include 16 key points, the first key point coordinate set can be represented as a 16*2 tensor, and the coordinates of each of the 16 key points include two coordinates, a horizontal coordinate and a vertical coordinate. Similarly, the second key point coordinate set can be represented as an N*2 tensor, and the first key point coordinate set and the second key point coordinate set are spliced ​​along the N direction to obtain a 2N*2 splicing result.

[0127] Step 2620, using the output mapping network to fuse the stitching results, to obtain the coordinates of each key point of the target object.

[0128] In some optional embodiments of the present disclosure, the output mapping network may include any one of a linear network and a channel weighted network (SE-NET). The splicing result may be linearly weighted by a linear network to obtain the coordinates of each key point of the target object. SE-NET is not a network model of a specific structure. Any network model can become SE-NET after introducing SE Block. SE Block is essentially an attention module, which may include a compression module and an excitation module.

[0129] This embodiment helps to improve the generalization ability of the fusion by splicing the first key point coordinate set and the second key point coordinate set, and then improves the accuracy of the coordinates of each key point obtained by the fusion.

[0130] In some optional embodiments of the present disclosure, the step 260 of fusing the first key point coordinate set and the second key point coordinate set to obtain the coordinates of each key point of the target object includes:

[0131] For any key point of the target object: in response to the key point being in the target image, the first predicted coordinates corresponding to the key point in the first key point coordinate set are used as the coordinates of the key point. In response to the key point not being in the target image, the second predicted coordinates corresponding to the key point in the second key point coordinate set are used as the coordinates of the key point.

[0132] In some optional embodiments of the present disclosure, when the first key point coordinate set and the second key point coordinate set are fused, the advantages of the key point coordinates of the heat map regression and the key point coordinates of the global eigenvector regression are comprehensively considered. For the key point located in the target image, the first predicted coordinates of the key point of the heat map regression are used as the coordinates of the key point. For the key point not in the target image, the second predicted coordinates of the key point of the global eigenvector regression are used as the coordinates of the key point. Since the first predicted coordinates of each key point of the heat map regression in the target image have a higher accuracy, and the first predicted coordinates of the key point not in the target image have a poorer accuracy, and the second predicted coordinates of each key point of the global eigenvector regression, the accuracy of the second predicted coordinates of the key point in the target image is relatively lower than the corresponding first predicted coordinates, and the accuracy of the second predicted coordinates of the key point not in the target image of the global eigenvector regression is relatively higher than the corresponding first predicted coordinates. Therefore, the coordinates of the key points in the target image and outside the target image respectively can be combined to obtain the coordinates of each key point with a higher overall accuracy.

[0133] In some optional embodiments of the present disclosure, the target object includes any one of a human body, a hand, an animal, and a human face.

[0134] In some optional embodiments of the present disclosure, the human body, hand, animal and face are different objects, and different objects focus on different key points. For example, the human body may include any one of 21 key points, 16 key points, etc., and the human body key points are used to describe the key points distributed on the human skeleton. For example, it may include key points of the head, key points of the neck, key points on the arms, key points of the abdomen, key points of the legs, etc. The hand may include key points distributed on the hand skeleton, for example, the hand key points may be represented as 21 key points, which may include key points at the base of the palm, key points at each joint of the fingers, key points at the fingertips, etc. The face may include key points distributed on the face of a person, for example, the key points of the face may include eye key points, mouth key points, nose key points, eyebrow key points, ear key points, etc. The key points specifically included in different objects are not limited.

[0135] In practical applications, a certain key point regression model can be used to predict the coordinates of the key points of a certain object. For example, the human body key point regression model is used to predict the coordinates of the human body key points, the face key point regression model is used to predict the coordinates of the face key points, and the hand key point regression model is used to predict the coordinates of the hand key points.

[0136] In some optional embodiments of the present disclosure, Figure 5 Schematic diagram of the structure of the key point regression model provided by some exemplary embodiments of the present disclosure. Figure 5The key point regression model shown can be used to implement the method for determining the key points of an object in an embodiment of the present disclosure. Among them, the backbone network (Backbone) is used as a feature extraction network to perform step-by-step feature extraction on the target image to obtain feature maps corresponding to multiple resolutions. The stem layer (stem layer) may include at least one convolution layer for performing preliminary feature extraction on the target image to obtain a basic feature map. The first stage layer to the fifth stage layer (stage layer 1~stage layer5) are used to perform step-by-step feature extraction on the basic feature map. In order to reduce the amount of calculation, the feature map obtained by the second stage layer (stage layer 2) is used as a feature map of a preset resolution. Here, the multiple resolutions can be 4 resolutions. The step size is stride, for example, a step size of 4 (stride 4) indicates that the scale of the feature map corresponding to the output resolution is 1 / 4 of the scale of the target image. The neck network (Neck) is used as a feature fusion network, and FPN is formed based on 4 bottom-up layers (BottomUpLayer) to fuse the feature maps corresponding to multiple resolutions to obtain a fused target feature map of a preset resolution. The neck network also performs global pooling processing on the lowest resolution feature map extracted by the fifth stage layer (stage layer5) based on the global pooling network (Global Pooling Layer) to obtain a global feature vector. The head network (Head) performs heat map prediction on the target feature map of preset resolution based on the heat map regression network (Heatmap Output Layer) to obtain the heat distribution map of the target image, and then performs key point regression on the heat distribution map based on the integral posture regression network (IPR) to obtain the first key point coordinate set. On the other hand, the head network performs key point regression prediction on the global feature vector through the linear prediction network (Linear Layer) to obtain the second key point coordinate set. The output network (Output) fuses the first key point coordinate set and the second key point coordinate set through the output result fusion network (output fusion) to obtain the coordinates of each key point of the target object.

[0137] The above-mentioned embodiments or optional examples of the present disclosure may be implemented separately or in any combination without conflict. The specific configuration may be based on actual needs and the present disclosure does not limit this.

[0138] Any method for determining key points of an object provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any method for determining key points of an object provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any method for determining key points of an object mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0139] Figure 6 to Figure 8 is a flowchart of a method for training a key point regression model provided by some exemplary embodiments of the present disclosure. The embodiments of the present disclosure can be applied to electronic devices, such as Figure 6 As shown, the method comprises the following steps:

[0140] Step 410: Obtain an image sample set.

[0141] In some optional embodiments of the present disclosure, the image sample set includes multiple training samples, each training sample includes a sample image, a sample screenshot taken from the sample image and including at least a portion of a preset object, and the true value coordinates of each key point of the preset object.

[0142] In some optional embodiments of the present disclosure, the sample images in the image sample set correspond to the initial images, and the sample screenshots correspond to the target images. The sample images can be acquired based on a terminal device equipped with a camera.

[0143] In some optional embodiments of the present disclosure, the preset object may include any one of objects such as human body, animal, face, hand, etc.

[0144] In some optional embodiments of the present disclosure, the true value coordinates of each key point of the preset object can be obtained by annotation, for example, by manual annotation, semi-automatic annotation, etc.

[0145] Step 420, the operations of steps 4210 to 4250 are performed by the model to be trained until the first stage of training of the model to be trained is completed, and a heat map regression model is obtained (see Figure 7 ).

[0146] Step 4210: for any training sample in any first batch, determine a first sample heat distribution map of the sample screenshot in the training sample.

[0147] In some optional embodiments of the present disclosure, each first batch includes at least one training sample in the image sample set.

[0148] In some optional embodiments of the present disclosure, the first sample heat distribution map may be a first sample heat map distribution map generated by the model to be trained according to the sample screenshot.

[0149] Step 4220, using the true value coordinates of each key point of the preset object in the training sample, generate a true value heat distribution map corresponding to the sample screenshot.

[0150] In some optional embodiments of the present disclosure, for the true value coordinates of each key point of the preset object in the training sample, a probability map centered on the true value coordinates of the key point can be generated based on the Gaussian distribution function, and then the probability maps corresponding to the key points are fused to obtain the true value heat distribution map corresponding to the sample screenshot. The fusion method can be to use the probability map corresponding to each key point as a channel of the true value heat distribution map, splice the probability maps of each channel, and crop the spliced ​​probability map according to the area of ​​the sample screenshot to obtain the first sample heat distribution map of the sample screenshot. The first sample heat distribution map includes the probability values ​​of each pixel in the sample screenshot belonging to each key point, and each probability value in the channel corresponding to each key point represents the probability that the pixel position belongs to the key point.

[0151] Step 4230, determining the first loss of the training sample based on the true value heat distribution map corresponding to the sample screenshot and the first sample heat distribution map.

[0152] In some optional embodiments of the present disclosure, the first loss can be determined based on any one of the mean absolute error (MAE) loss function (L1 loss function), the mean square error (MSE) loss function (L2 loss function), etc.

[0153] In some optional examples of the present disclosure, the first loss of the training sample can be expressed as follows:

[0154]

[0155] Wherein, N represents the number of key points included in the preset object, H represents the true value heat distribution map corresponding to the sample screenshot, and H′ represents the first sample heat distribution map. i represents the i-th channel component of the true value heat distribution map, H i ′ represents the i-th channel component of the first sample heat distribution map.

[0156] Step 4240, based on the first loss of each training sample in the first batch, obtain the heat map loss of the first batch.

[0157] In some optional embodiments of the present disclosure, the absolute values ​​of the first losses of each training sample in the first batch may be weighted summed to obtain the heat map loss of the first batch. The absolute values ​​of the first losses of each training sample in the first batch may be averaged, and the average value may be used as the heat map loss of the first batch. The median or quantile value of the first loss of each training sample in the first batch may also be used as the heat map loss of the first batch.

[0158] Step 4250: Update the network parameters of the model to be trained based on the heat map loss of the first batch to obtain an updated model to be trained.

[0159] In some optional embodiments of the present disclosure, the update gradient of the network parameters can be calculated based on the heat map loss of the first batch, and then the network parameters are updated based on the update gradient to obtain an updated model to be trained.

[0160] In some optional embodiments of the present disclosure, the update gradient can be determined using any implementable gradient descent algorithm, such as a stochastic gradient descent algorithm, a batch gradient descent algorithm, a learning rate adaptive gradient descent algorithm, and the like.

[0161] Step 430, the following steps 4310 to 4370 are performed by the model to be trained until the second stage of training of the model to be trained is completed, and a key point regression model is obtained (see Figure 8 ).

[0162] In some optional embodiments of the present disclosure, the model to be trained in step 430 is initially a heat map regression model. That is, the heat map regression model is obtained after the first stage training of step 420 is completed. The network parameters of the heat map regression model are used as the initial network parameters of the model to be trained, and the second stage training of step 430 is performed.

[0163] Step 4310: for any training sample in any second batch, determine the sample global feature vector and the second sample heat distribution map of the sample screenshot in the training sample.

[0164] In some optional embodiments of the present disclosure, each second batch includes at least one training sample in the image sample set. The number of training samples included in the second batch may be the same as or different from the number of training samples included in the first batch.

[0165] It should be noted that the specific operation principle of determining the sample global feature vector and the second sample heat distribution map of the sample screenshot is the same as that of determining the global feature vector and heat distribution map of the target image in the aforementioned embodiment. Figure 1 I will not elaborate on this.

[0166] Step 4320: predict a first sample key point coordinate set of a preset object based on the second sample heat distribution map.

[0167] In some optional embodiments of the present disclosure, the first sample key point coordinate set includes first sample predicted coordinates of each key point of a preset object.

[0168] It should be noted that the specific operating principle of step 4320 is consistent with the above-mentioned embodiment of predicting the first key point coordinate set of the target object based on the heat distribution map, and will not be elaborated here.

[0169] Step 4330: predict a second sample key point coordinate set of the preset object based on the sample global feature vector.

[0170] In some optional embodiments of the present disclosure, the second sample key point coordinate set includes second sample predicted coordinates of each key point of the preset object.

[0171] It should be noted that the above steps 4320 and 4330 have no dependency on the execution order and can be executed simultaneously or successively without any restriction on the execution order.

[0172] It should be noted that the specific operating principle of step 4330 is consistent with the prediction of the second key point coordinate set of the target object based on the global feature vector in the aforementioned embodiment, and will not be elaborated here.

[0173] Step 4340: fuse the first sample key point coordinate set and the second sample key point coordinate set to obtain sample coordinates of each key point of the preset object.

[0174] It should be noted that the specific operating principle of step 4340 is consistent with the above-mentioned embodiment of fusing the first key point coordinate set and the second key point coordinate set to obtain the coordinates of each key point of the target object, and will not be elaborated here.

[0175] Step 4350, based on the first sample key point coordinate set, the second sample key point coordinate set, the sample coordinates and true value coordinates of each key point of the preset object, determine the second loss of the training sample.

[0176] In some optional embodiments of the present disclosure, the second loss may be determined based on any one of an L1 loss function, an L2 loss function, and the like.

[0177] Step 4360, based on the second loss of each training sample in the second batch, obtain the model loss of the second batch.

[0178] In some optional embodiments of the present disclosure, any one of the weighted loss, average loss, median, quantile value, etc. of the second loss of each training sample in the second batch can be calculated as the model loss of the second batch.

[0179] Step 4370: Update the network parameters of the model to be trained based on the model loss of the second batch to obtain an updated model to be trained.

[0180] In some optional embodiments of the present disclosure, the update gradient of the network parameters can be calculated based on the model loss of the second batch, and then the network parameters are updated based on the update gradient to obtain an updated model to be trained.

[0181] In some optional embodiments of the present disclosure, the update gradient can be determined using any implementable gradient descent algorithm, such as a stochastic gradient descent algorithm, a batch gradient descent algorithm, a learning rate adaptive gradient descent algorithm, and the like.

[0182] In some optional embodiments of the present disclosure, Fig. 9 Schematic diagram of the network structure of the model to be trained provided by some other exemplary embodiments of the present disclosure. Fig. 9 As shown, the network structure of the model to be trained may include a feature extraction network, a feature fusion network, a global feature vector generation network, a first regression network, a second regression network, and a key point fusion network. Among them, the feature extraction network is used to perform step-by-step feature extraction of multiple resolutions on the sample screenshots according to the resolution from high to low, and obtain sample feature maps corresponding to multiple resolutions. The feature fusion network is used to fuse the sample feature maps corresponding to multiple resolutions to obtain the target sample feature map of the preset resolution. The first regression network may include a heat map generation subnetwork and a key point regression subnetwork, the heat map generation subnetwork is used to generate a sample heat distribution map based on the target sample feature map, and the key point regression subnetwork is used to perform key point regression on the sample heat distribution map to obtain a first sample key point coordinate set. The global feature vector generation network is used to perform any one of global pooling, convolution, full connection and other operations on the sample feature map corresponding to the lowest resolution to obtain a sample global feature vector. The second regression network is used to perform key point regression on the sample global feature vector to obtain a second sample key point coordinate set. The key point fusion network is used to fuse the first sample key point coordinate set and the second sample key point coordinate set to obtain the sample coordinates of each key point of the preset object.

[0183] In some optional embodiments of the present disclosure, during the first stage of training, the feature extraction network, feature fusion network and heat map generation sub-network in the model to be trained participate in model training. After the first stage of training, the model to be trained becomes a heat map regression model. During the second stage of training, all networks in the model to be trained whose initial model parameters are the heat map regression model participate in model training to obtain the final key point regression model.

[0184] The method for training a key point regression model provided in this embodiment obtains a heat map regression model through a first stage of training, and then performs a second stage of overall model training on the basis of the heat map regression model to obtain a key point regression model, which helps to quickly converge the model and improve the efficiency of model training. The obtained key point regression model can combine the advantages of heat map regression key points and global eigenvector regression key points, and predicts the coordinates of each key point of a preset object by fusing the two methods, which helps to improve the overall accuracy of the predicted coordinates of each key point.

[0185] In some optional embodiments of the present disclosure, determining the first sample heat distribution map of the sample screenshots in the training sample in step 4210 includes:

[0186] Based on the sample screenshot, a first target sample feature map of a preset resolution is determined. The preset resolution is lower than the resolution of the sample screenshot, and the scale of the first target sample feature map is smaller than the scale of the sample screenshot. Based on the first target sample feature map, a first sample heat distribution map of the sample screenshot is generated.

[0187] In some optional embodiments of the present disclosure, determining a first target sample feature map of a preset resolution based on the sample screenshot includes:

[0188] The sample screenshot is subjected to step-by-step feature extraction at multiple resolutions from high to low resolutions to obtain first sample feature maps corresponding to the multiple resolutions. The preset resolution is the highest resolution among the multiple resolutions. The scale of the first sample feature maps corresponding to the multiple resolutions decreases as the resolution decreases. The first sample feature maps corresponding to the multiple resolutions are fused to obtain a first target sample feature map with the preset resolution.

[0189] In some optional embodiments of the present disclosure, first sample feature maps corresponding to multiple resolutions are fused to obtain a first target sample feature map of a preset resolution, including:

[0190] The first sample feature map corresponding to the lowest resolution among multiple resolutions is used as the initial sample feature map. The second lowest resolution is selected from the multiple resolutions as the current resolution. The sample feature map to be upsampled of the previous resolution is upsampled to obtain the upsampled sample feature map of the current resolution. Among the multiple resolutions, the current resolution is one level higher than the resolution of the previous resolution, and the sample feature map to be upsampled of the previous resolution is initially the initial sample feature map. The upsampled sample feature map of the current resolution is fused with the first sample feature map corresponding to the current resolution to obtain the fused sample feature map of the current resolution. In response to the current resolution being the preset resolution, the fused sample feature map of the current resolution is used as the first target sample feature map. In response to the current resolution not being the preset resolution, the current fused sample feature map is used as the new sample feature map to be upsampled of the previous resolution, and a resolution one level higher than the current resolution is selected from the multiple resolutions as the new current resolution, and the step of upsampling the sample feature map to be upsampled of the previous resolution is repeated to obtain the upsampled sample feature map of the current resolution.

[0191] In some optional embodiments of the present disclosure, determining the second sample heat distribution map of the sample screenshot in the training sample in step 4310 includes:

[0192] Based on the sample screenshot, a second target sample feature map of a preset resolution is determined. The preset resolution is lower than the resolution of the sample screenshot, and the scale of the second target sample feature map is smaller than the scale of the sample screenshot. Based on the second target sample feature map, a second sample heat distribution map of the sample screenshot is generated.

[0193] In some optional embodiments of the present disclosure, determining a second target sample feature map of a preset resolution based on the sample screenshot includes:

[0194] The sample screenshot is subjected to step-by-step feature extraction at multiple resolutions from high to low resolutions to obtain second sample feature maps corresponding to multiple resolutions. The preset resolution is the highest resolution among the multiple resolutions. The scale of the second sample feature maps corresponding to the multiple resolutions decreases as the resolution decreases. The second sample feature maps corresponding to the multiple resolutions are fused to obtain a second target sample feature map with the preset resolution.

[0195] In some optional embodiments of the present disclosure, the second sample feature maps corresponding to the multiple resolutions are fused to obtain a second target sample feature map of a preset resolution, including:

[0196] The second sample feature map corresponding to the lowest resolution among the multiple resolutions is used as the initial sample feature map. The second lowest resolution is selected from the multiple resolutions as the current resolution. The sample feature map to be upsampled of the previous resolution is upsampled to obtain the upsampled sample feature map of the current resolution. Among the multiple resolutions, the current resolution is one level higher than the resolution of the previous resolution, and the sample feature map to be upsampled of the previous resolution is initially the initial sample feature map. The upsampled sample feature map of the current resolution is fused with the second sample feature map corresponding to the current resolution to obtain the fused sample feature map of the current resolution. In response to the current resolution being the preset resolution, the fused sample feature map of the current resolution is used as the second target sample feature map. In response to the current resolution not being the preset resolution, the current fused sample feature map is used as the new sample feature map to be upsampled of the previous resolution, and a resolution one level higher than the current resolution is selected from the multiple resolutions as the new current resolution, and the step of upsampling the sample feature map to be upsampled of the previous resolution is repeated to obtain the upsampled sample feature map of the current resolution.

[0197] In some optional embodiments of the present disclosure, determining the sample global feature vector of the sample screenshot in the training sample in step 4310 includes:

[0198] Based on the sample feature map corresponding to the lowest resolution among multiple resolutions, a sample global feature vector is determined.

[0199] In some optional embodiments of the present disclosure, the sample global feature vector is determined by any of the following:

[0200] Use the global pooling network to perform global pooling processing on the sample feature map corresponding to the lowest resolution to obtain the sample global feature vector. Use the convolution network to perform convolution processing on the sample feature map corresponding to the lowest resolution to obtain the sample global feature vector. Use the fully connected network to process the sample feature map corresponding to the lowest resolution to obtain the sample global feature vector. Expand the sample feature map corresponding to the lowest resolution to obtain the sample global feature vector.

[0201] In some optional embodiments of the present disclosure, predicting a first sample key point coordinate set of a preset object based on the second sample heat distribution map in step 4320 includes:

[0202] The second sample heat distribution map is processed using a key point regression network to obtain a first sample key point coordinate set of a preset object.

[0203] In some optional embodiments of the present disclosure, the step 4330 of predicting a second sample key point coordinate set of a preset object based on the sample global feature vector includes:

[0204] The linear prediction network is used to predict the global feature vector of the sample to obtain the coordinate set of the second sample key points.

[0205] In some optional embodiments of the present disclosure, the fusing of the first sample key point coordinate set and the second sample key point coordinate set in step 4340 to obtain the sample coordinates of each key point of the preset object includes:

[0206] The first sample key point coordinate set and the second sample key point coordinate set are spliced ​​to obtain a sample splicing result. The sample splicing result is fused using an output mapping network to obtain sample coordinates of each key point of the preset object.

[0207] In some optional embodiments of the present disclosure, the output mapping network includes any one of a linear network and a channel weighted network.

[0208] In some optional embodiments of the present disclosure, the fusing of the first sample key point coordinate set and the second sample key point coordinate set in step 4340 to obtain the sample coordinates of each key point of the preset object includes:

[0209] For any key point of a preset object: in response to the key point being in the sample screenshot, the first sample predicted coordinates corresponding to the key point in the first sample key point coordinate set are used as the sample coordinates of the key point. In response to the key point not being in the sample screenshot, the second sample predicted coordinates corresponding to the key point in the second sample key point coordinate set are used as the sample coordinates of the key point.

[0210] In some optional embodiments of the present disclosure, the preset object includes any one of a human body, a hand, an animal, and a human face.

[0211] It should be noted that the specific operating principles of the processing of each step of the sample screenshots in the above training process are consistent with the operating principles of the corresponding steps of the reasoning process of the target image in the above embodiment, and will not be repeated here.

[0212] In some optional embodiments of the present disclosure, step 4350 determines the second loss of the training sample based on the first sample key point coordinate set, the second sample key point coordinate set, the sample coordinates of each key point of the preset object, and the true value coordinates, including:

[0213] Step 43510, determining a first regression loss based on the first sample predicted coordinates and true value coordinates of each key point of the preset object in the training sample.

[0214] In some optional embodiments of the present disclosure, the first regression loss may be calculated based on any one of an L1 loss function, an L2 loss function, and the like.

[0215] Step 43520, determining a second regression loss based on the second sample predicted coordinates and true value coordinates of each key point of the preset object in the training sample.

[0216] In some optional embodiments of the present disclosure, the second regression loss may be calculated based on any one of an L1 loss function, an L2 loss function, and the like.

[0217] Step 43530, determining a third regression loss based on the sample coordinates and true value coordinates of each key point of the preset object in the training sample.

[0218] In some optional embodiments of the present disclosure, the third regression loss can be calculated based on any one of the L1 loss function, the L2 loss function, etc.

[0219] It should be noted that the above steps 43510, 43520 and 43530 have no execution order dependency and can be executed simultaneously or sequentially without any restriction on the execution order.

[0220] Step 43540, based on the first regression loss, the second regression loss and the third regression loss, determine the second loss of the training sample.

[0221] In some optional embodiments of the present disclosure, the first regression loss, the second regression loss, and the third regression loss may be weightedly summed to obtain the second loss of the training sample.

[0222] In some optional embodiments of the present disclosure, the second loss of the training sample can be expressed as follows:

[0223] Loss2=w1*L1(P′ ipr -P)*M+w2*L1(P′ global -P)+w3*L1(P′ out -P)

[0224] Among them, P represents the true value coordinates of each key point of the preset object, P′ ipr represents the coordinate set of the first sample key point, P′ global Represents the coordinate set of the second sample key points, P′ out Indicates the fusion P′ ipr and P′ global The sample coordinates of each key point of the preset object are obtained. L1 represents the L1 loss function. w1, w2 and w3 represent the weights of the first regression loss, the second regression loss and the third regression loss respectively. The L1 loss function is defined as:

[0225]

[0226] Where N represents the number of key points of the preset object, and P′ can be P′ ipr , P′ global and P′ out In any of the above, abs() represents the absolute value function.

[0227] It should be noted that for each training sample, when calculating L1(P′ ipr -P), the key point P′ located in the sample screenshot of the training sample is used. ipr and P. For the first sample prediction coordinates in the first sample key point coordinate set that are located in the sample screenshot, the M value is, and for the first sample prediction coordinates in the first sample key point coordinate set that are located outside the sample screenshot, the M value is 0. This is because the accuracy of the first sample prediction coordinates in the first sample key point coordinate set that are located outside the sample screenshot is not high enough, so for the key points that are not in the sample screenshot, the influence of the key points on the first regression loss is adjusted to 0 through M to avoid the heat map loss of the key points outside the sample screenshot affecting the model convergence speed.

[0228] This embodiment jointly supervises model training by combining the first regression loss, the second regression loss and the third regression loss, which helps to further improve the accuracy of the prediction results of the key point regression model.

[0229] In some optional embodiments of the present disclosure, Fig.10 FIG. 1 is a flow chart of obtaining an image sample set provided by some exemplary embodiments of the present disclosure. Fig.10 As shown, the method for training a key point regression model in an embodiment of the present disclosure further includes:

[0230] Step 510: Obtain an initial image sample set.

[0231] In some optional embodiments of the present disclosure, the initial image sample set includes multiple initial samples, each initial sample includes an initial image, a complete initial screenshot including a preset object captured from the initial image, and the true value coordinates of each key point of the preset object in the initial image.

[0232] In some optional embodiments of the present disclosure, the initial image may be acquired based on a terminal device equipped with a camera.

[0233] In some optional embodiments of the present disclosure, the initial screenshot that completely includes the preset object can be obtained by cropping the initial image based on the enclosing frame that completely surrounds the preset object on the initial image. For example, the initial image includes multiple preset objects, and the detection frames corresponding to each preset object in the initial image are obtained through target detection. For each detection frame corresponding to the preset object, an initial screenshot that completely surrounds the preset object is captured from the initial image based on the detection frame, so that the initial screenshots corresponding to each preset object in the initial image can be obtained.

[0234] It should be noted that the initial screenshot completely includes the preset object, which means that all pixels occupied by the preset object in the initial image are circled in the initial screenshot, rather than the initial screenshot needing to include a complete preset object. For example, for an initial image with half a face, the initial screenshot needs to include all pixels occupied by the face in the initial image, rather than necessarily including the entire face.

[0235] Step 520 , for each initial sample in the cropped image sample set, crop the initial screenshot of the initial sample to obtain a cropped screenshot.

[0236] In some optional embodiments of the present disclosure, the cropped image sample set is a subset of the initial image sample set. The cropped image sample set may be a complete initial image sample set, or may include a portion of the initial samples in the initial image sample set. In the case where the cropped image sample set is a complete initial image sample set, the initial image sample set is directly used as the cropped image sample set. In the case where the cropped image sample set includes a portion of the initial samples in the initial image sample set, the initial samples in the initial image sample set are extracted to obtain the cropped image sample set. The extraction may be random extraction, sequential extraction, or extraction according to a preset rule (for example, according to an arithmetic progression or a geometric progression), etc. The cropped screenshot includes a portion of a preset object.

[0237] It should be noted that the cropped screenshot includes a part of the preset object, which means that a part of the pixels occupied by the preset object in the initial image is circled in the cropped screenshot, and the cropped screenshot may also include pixels occupied by the preset object.

[0238] In some optional embodiments of the present disclosure, the initial screenshot of the initial sample may be cropped along at least one of the horizontal and vertical directions to crop a portion of the initial screenshot, and the remaining portion of the initial screenshot after cropping is used as the cropped screenshot. For example, if the size of the initial screenshot is H1*W1, H1 represents the height of the initial screenshot, and W1 represents the width of the initial screenshot, the initial screenshot may be cropped into a cropped screenshot of H2*W1, where H2 is smaller than H1, so that the cropped screenshot includes a portion of the preset object.

[0239] Step 530: Use the cropped screenshot to replace the initial screenshot in the initial sample to obtain an enhanced sample corresponding to the initial sample.

[0240] In some optional embodiments of the present disclosure, each enhanced sample constitutes an enhanced image sample set.

[0241] Step 540: Obtain an image sample set based on the initial image sample set and the enhanced image sample set.

[0242] In some optional embodiments of the present disclosure, the corresponding initial samples in the cropped image sample set may be removed from the initial image sample set to obtain a remaining image sample set. The remaining image sample set and the enhanced image sample set are then merged to obtain an image sample set. If the cropped image sample set is a complete initial image sample set, the enhanced image sample set may be used as the image sample set.

[0243] In some optional embodiments of the present disclosure, the initial image sample set and the enhanced image sample set may be merged to obtain an image sample set.

[0244] The disclosed embodiment performs data enhancement on training samples by the above-mentioned means, so as to facilitate the model to learn sample screenshots that completely include preset objects and sample screenshots that incompletely include preset objects, thereby improving the richness of training samples.

[0245] In this embodiment, the initial screenshot in the initial sample is replaced by a cropped screenshot, so that the sample screenshot in the obtained training sample includes a part of the preset object, and the training sample is then used to train the key point regression model, so that when the obtained key point regression model is performing inference, even if the image only includes a part of the target object, it can still accurately predict the coordinates of each key point of the target object.

[0246] The above-mentioned embodiments or optional examples of the present disclosure may be implemented separately or in any combination without conflict. The specific configuration may be based on actual needs and the present disclosure does not limit this.

[0247] Any method for training a key point regression model provided in the embodiments of the present disclosure may be performed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any method for training a key point regression model provided in the embodiments of the present disclosure may be performed by a processor, such as the processor executing any method for training a key point regression model mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0248] Figure 11-12is a flowchart of a method for training a key point regression model provided by some other exemplary embodiments of the present disclosure. The embodiments of the present disclosure can be applied to electronic devices, such as Fig.11 As shown, the method comprises the following steps:

[0249] Step 610: Obtain an image sample set.

[0250] In some optional embodiments of the present disclosure, the image sample set includes multiple training samples, each training sample includes a sample image, a sample screenshot taken from the sample image and including at least a portion of a preset object, and the true value coordinates of each key point of the preset object.

[0251] Step 620, perform the following steps 6210 to 6270 through the model to be trained until the training of the model to be trained is completed, and a key point regression model is obtained (see Fig.11 ):

[0252] Step 6210, for any training sample in any batch, determine the sample global feature vector and sample heat distribution map of the sample screenshot in the training sample.

[0253] In some optional embodiments of the present disclosure, each batch includes at least one training sample in the image sample set.

[0254] Step 6220: predict a first sample key point coordinate set of a preset object based on the sample heat distribution map.

[0255] In some optional embodiments of the present disclosure, the first sample key point coordinate set includes first sample predicted coordinates of each key point of a preset object.

[0256] Step 6230: predict a second sample key point coordinate set of the preset object based on the sample global feature vector.

[0257] In some optional embodiments of the present disclosure, the second sample key point coordinate set includes second sample predicted coordinates of each key point of the preset object.

[0258] It should be noted that the above-mentioned step 6220 and step 6230 have no dependency on the execution order and can be executed simultaneously or successively without any restriction on the execution order.

[0259] Step 6240: Fusing the first sample key point coordinate set and the second sample key point coordinate set to obtain sample coordinates of each key point of the preset object.

[0260] Step 6250, based on the first sample key point coordinate set, the second sample key point coordinate set, the sample coordinates and true value coordinates of each key point of the preset object, determine the loss of the training sample.

[0261] Step 6260, based on the loss of each training sample in the batch, obtain the model loss of the batch.

[0262] Step 6270: Update the network parameters of the model to be trained based on the model loss of the batch to obtain an updated model to be trained.

[0263] It should be noted that the specific operating principles of step 610 and step 6210 to step 6270 in step 620 are consistent with the operating principles of each step of the second stage training in the aforementioned embodiment, and will not be repeated here.

[0264] In some optional embodiments of the present disclosure, determining the loss of the training sample based on the first sample key point coordinate set, the second sample key point coordinate set, the sample coordinates of each key point of the preset object, and the true value coordinates includes:

[0265] Step 62510, determining a first regression loss based on the first sample predicted coordinates and true value coordinates of each key point of the preset object in the training sample.

[0266] Step 62520, determining a second regression loss based on the second sample predicted coordinates and true value coordinates of each key point of the preset object in the training sample.

[0267] Step 62530, determining a third regression loss based on the sample coordinates and true value coordinates of each key point of the preset object in the training sample.

[0268] It should be noted that the above steps 62510, 62520 and 62530 have no execution order dependency and can be executed simultaneously or sequentially without any restriction on the execution order.

[0269] Step 62540, based on the first regression loss, the second regression loss and the third regression loss, determine the model loss of the training sample.

[0270] It should be noted that the specific operating principles of the above steps 62510 to 62540 are consistent with the determination of the second loss of the second stage training in the aforementioned embodiment, and will not be repeated here.

[0271] In some optional embodiments of the present disclosure, the method for training a key point regression model in the embodiment of the present disclosure further includes:

[0272] Step 710: Obtain an initial image sample set.

[0273] In some optional embodiments of the present disclosure, the initial image sample set includes multiple initial samples, each initial sample includes an initial image, a complete initial screenshot including a preset object captured from the initial image, and the true value coordinates of each key point of the preset object in the initial image.

[0274] Step 720 , for each initial sample in the cropped image sample set, crop the initial screenshot of the initial sample to obtain a cropped screenshot.

[0275] In some optional embodiments of the present disclosure, the cropped image sample set is a subset of the initial image sample set. The cropped screenshot includes a portion of the preset object.

[0276] Step 730: Use the cropped screenshot to replace the initial screenshot in the initial sample to obtain an enhanced sample corresponding to the initial sample.

[0277] In some optional embodiments of the present disclosure, each enhanced sample constitutes an enhanced image sample set.

[0278] Step 740: Obtain an image sample set based on the initial image sample set and the enhanced image sample set.

[0279] It should be noted that the specific operation principle of determining the image sample set in the above steps 710 to 740 is consistent with the above embodiment and will not be elaborated here.

[0280] The above-mentioned embodiments or optional examples of the present disclosure may be implemented separately or in any combination without conflict. The specific configuration may be based on actual needs and the present disclosure does not limit this.

[0281] Any method for training a key point regression model provided in the embodiments of the present disclosure may be performed by any appropriate device with data processing capabilities, including but not limited to: a terminal device and a server, etc. Alternatively, any method for training a key point regression model provided in the embodiments of the present disclosure may be performed by a processor, such as the processor executing any method for training a key point regression model mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0282] Exemplary Devices

[0283] Fig.13 is a schematic diagram of the structure of an apparatus for determining key points of an object provided by some exemplary embodiments of the present disclosure. The apparatus of this embodiment can be used to implement the corresponding method embodiment for determining key points of an object of the present disclosure, such as Fig.13 The device shown includes: an image acquisition module 810 , a first processing module 820 , a second processing module 830 , a fourth processing module 840 and a fifth processing module 850 .

[0284] The image acquisition module 810 is used to acquire an original image.

[0285] The first processing module 820 is used to capture a target image including at least a part of the target object from the original image.

[0286] The second processing module 830 is used to determine the global feature vector and heat distribution map of the target image.

[0287] The third processing module 840 is used to predict a first key point coordinate set of the target object based on the heat distribution map.

[0288] In some optional embodiments of the present disclosure, the first key point coordinate set includes first predicted coordinates of each key point of the target object.

[0289] The fourth processing module 850 is used to predict a second key point coordinate set of the target object based on the global feature vector.

[0290] In some optional embodiments of the present disclosure, the second key point coordinate set includes second predicted coordinates of each key point of the target object.

[0291] The fifth processing module 860 is used to fuse the first key point coordinate set and the second key point coordinate set to obtain the coordinates of each key point of the target object.

[0292] In some optional embodiments of the present disclosure, the second processing module 830 is specifically used for:

[0293] Based on the target image, a target feature map of a preset resolution is determined, wherein the preset resolution is lower than the resolution of the target image, and the scale of the target feature map is smaller than the scale of the target image. Based on the target feature map, a heat distribution map of the target image is generated.

[0294] In some optional embodiments of the present disclosure, the second processing module 830 is specifically used for:

[0295] The target image is subjected to step-by-step feature extraction at multiple resolutions from high to low resolutions to obtain feature maps corresponding to the multiple resolutions. The preset resolution is the largest resolution among the multiple resolutions. The scales of the feature maps corresponding to the multiple resolutions decrease as the resolution decreases. The feature maps corresponding to the multiple resolutions are fused to obtain the target feature map of the preset resolution.

[0296] In some optional embodiments of the present disclosure, the second processing module 830 is specifically used to determine a global feature vector based on a feature map corresponding to the lowest resolution among multiple resolutions.

[0297] In some optional embodiments of the present disclosure, the second processing module 830 is specifically configured to determine the global feature vector by any of the following methods:

[0298] Use the global pooling network to perform global pooling on the feature map corresponding to the lowest resolution to obtain a global feature vector. Use the convolution network to perform convolution on the feature map corresponding to the lowest resolution to obtain a global feature vector. Use the fully connected network to process the feature map corresponding to the lowest resolution to obtain a global feature vector. Expand the feature map corresponding to the lowest resolution to obtain a global feature vector.

[0299] In some optional embodiments of the present disclosure, the second processing module 830 is specifically used for:

[0300] The feature map corresponding to the lowest resolution among multiple resolutions is used as the initial feature map. The second lowest resolution is selected from multiple resolutions as the current resolution. The feature map to be upsampled of the previous resolution is upsampled to obtain the upsampled feature map of the current resolution. Among the multiple resolutions, the current resolution is one level higher than the resolution of the previous resolution, and the feature map to be upsampled of the previous resolution is initially the initial feature map. The upsampled feature map of the current resolution and the feature map corresponding to the current resolution are fused to obtain the fused feature map of the current resolution. In response to the current resolution being the preset resolution, the fused feature map of the current resolution is used as the target feature map. In response to the current resolution not being the preset resolution, the current fused feature map is used as the new upsampled feature map of the previous resolution, and a resolution one level higher than the current resolution is selected from multiple resolutions as the new current resolution, and the step of upsampling the feature map to be upsampled of the previous resolution is repeated to obtain the upsampled feature map of the current resolution.

[0301] In some optional embodiments of the present disclosure, the third processing module 840 is specifically used to:

[0302] The heat distribution map is processed using a key point regression network to obtain the first key point coordinate set of the target object.

[0303] In some optional embodiments of the present disclosure, the fourth processing module 850 is specifically used to:

[0304] The global feature vector is predicted using a linear prediction network to obtain a second key point coordinate set.

[0305] In some optional embodiments of the present disclosure, the fifth processing module 860 is specifically used to:

[0306] The first key point coordinate set and the second key point coordinate set are spliced ​​to obtain a splicing result. The splicing result is fused using an output mapping network to obtain the coordinates of each key point of the target object.

[0307] In some optional embodiments of the present disclosure, the output mapping network includes any one of a linear network and a channel weighted network.

[0308] In some optional embodiments of the present disclosure, the fifth processing module 860 is specifically used to:

[0309] For any key point of the target object: in response to the key point being in the target image, the first predicted coordinates corresponding to the key point in the first key point coordinate set are used as the coordinates of the key point. In response to the key point not being in the target image, the second predicted coordinates corresponding to the key point in the second key point coordinate set are used as the coordinates of the key point.

[0310] In some optional embodiments of the present disclosure, the target object includes any one of a human body, a hand, an animal, and a human face.

[0311] It should be noted that the specific implementation of the device for determining the key points of an object in the embodiment of the present disclosure is similar to the specific implementation of the method for determining the key points of an object in the embodiment of the present disclosure. Please refer to the part of the method for determining the key points of an object for details. In order to reduce redundancy, it will not be repeated here.

[0312] Fig.14 is a schematic diagram of the structure of an apparatus for training a key point regression model provided by some exemplary embodiments of the present disclosure. The apparatus of this embodiment can be used to implement the corresponding method embodiment for training a key point regression model of the present disclosure, such as Fig.14 The device shown includes: a sample acquisition module 910 , a first training module 920 and a second training module 930 .

[0313] The sample acquisition module 910 is used to acquire an image sample set.

[0314] In some optional embodiments of the present disclosure, the image sample set includes multiple training samples, each training sample includes a sample image, a sample screenshot taken from the sample image and including at least a portion of a preset object, and the true value coordinates of each key point of the preset object.

[0315] The first training module 920 is used to perform the following operations on the model to be trained until the first stage of training of the model to be trained is completed to obtain a heat map regression model:

[0316] For any training sample in any first batch, determine the first sample heat distribution map of the sample screenshot in the training sample. Each first batch includes at least one training sample in the image sample set. Generate a true heat distribution map corresponding to the sample screenshot using the true value coordinates of each key point of the preset object in the training sample. Determine the first loss of the training sample based on the true value heat distribution map corresponding to the sample screenshot and the first sample heat distribution map. Based on the first loss of each training sample in the first batch, obtain the heat map loss of the first batch. Update the network parameters of the model to be trained based on the heat map loss of the first batch to obtain an updated model to be trained.

[0317] The second training module 930 is used to perform the following operations on the model to be trained until the second stage of training of the model to be trained is completed to obtain a key point regression model. The model to be trained is initially a heat map regression model:

[0318] For any training sample in any second batch, determine the sample global feature vector and the second sample heat distribution map of the sample screenshot in the training sample. Each second batch includes at least one training sample in the image sample set. Based on the second sample heat distribution map, predict the first sample key point coordinate set of the preset object. The first sample key point coordinate set includes the first sample predicted coordinates of each key point of the preset object. Based on the sample global feature vector, predict the second sample key point coordinate set of the preset object. The second sample key point coordinate set includes the second sample predicted coordinates of each key point of the preset object. The first sample key point coordinate set and the second sample key point coordinate set are fused to obtain the sample coordinates of each key point of the preset object. Based on the first sample key point coordinate set, the second sample key point coordinate set, the sample coordinates of each key point of the preset object and the true value coordinates, determine the second loss of the training sample. Based on the second loss of each training sample in the second batch, obtain the model loss of the second batch. Based on the model loss of the second batch, update the network parameters of the model to be trained to obtain the updated model to be trained.

[0319] In some optional embodiments of the present disclosure, the first training module 920 is specifically used for:

[0320] Based on the sample screenshot, a first target sample feature map of a preset resolution is determined. The preset resolution is lower than the resolution of the sample screenshot, and the scale of the first target sample feature map is smaller than the scale of the sample screenshot. Based on the first target sample feature map, a first sample heat distribution map of the sample screenshot is generated.

[0321] In some optional embodiments of the present disclosure, the first training module 920 is specifically used for:

[0322] The sample screenshot is subjected to step-by-step feature extraction at multiple resolutions from high to low resolutions to obtain first sample feature maps corresponding to the multiple resolutions. The preset resolution is the highest resolution among the multiple resolutions. The scale of the first sample feature maps corresponding to the multiple resolutions decreases as the resolution decreases. The first sample feature maps corresponding to the multiple resolutions are fused to obtain a first target sample feature map with the preset resolution.

[0323] In some optional embodiments of the present disclosure, the first training module 920 is specifically used for:

[0324] The first sample feature map corresponding to the lowest resolution among multiple resolutions is used as the initial sample feature map. The second lowest resolution is selected from the multiple resolutions as the current resolution. The sample feature map to be upsampled of the previous resolution is upsampled to obtain the upsampled sample feature map of the current resolution. Among the multiple resolutions, the current resolution is one level higher than the resolution of the previous resolution, and the sample feature map to be upsampled of the previous resolution is initially the initial sample feature map. The upsampled sample feature map of the current resolution is fused with the first sample feature map corresponding to the current resolution to obtain the fused sample feature map of the current resolution. In response to the current resolution being the preset resolution, the fused sample feature map of the current resolution is used as the first target sample feature map. In response to the current resolution not being the preset resolution, the current fused sample feature map is used as the new sample feature map to be upsampled of the previous resolution, and a resolution one level higher than the current resolution is selected from the multiple resolutions as the new current resolution, and the step of upsampling the sample feature map to be upsampled of the previous resolution is repeated to obtain the upsampled sample feature map of the current resolution.

[0325] In some optional embodiments of the present disclosure, the second training module 930 is specifically used for:

[0326] Based on the sample screenshot, a second target sample feature map of a preset resolution is determined. The preset resolution is lower than the resolution of the sample screenshot, and the scale of the second target sample feature map is smaller than the scale of the sample screenshot. Based on the second target sample feature map, a second sample heat distribution map of the sample screenshot is generated.

[0327] In some optional embodiments of the present disclosure, the second training module 930 is specifically used for:

[0328] The sample screenshot is subjected to step-by-step feature extraction at multiple resolutions from high to low resolutions to obtain second sample feature maps corresponding to multiple resolutions. The preset resolution is the highest resolution among the multiple resolutions. The scale of the second sample feature maps corresponding to the multiple resolutions decreases as the resolution decreases. The second sample feature maps corresponding to the multiple resolutions are fused to obtain a second target sample feature map with the preset resolution.

[0329] In some optional embodiments of the present disclosure, the second training module 930 is specifically used for:

[0330] The second sample feature map corresponding to the lowest resolution among the multiple resolutions is used as the initial sample feature map. The second lowest resolution is selected from the multiple resolutions as the current resolution. The sample feature map to be upsampled of the previous resolution is upsampled to obtain the upsampled sample feature map of the current resolution. Among the multiple resolutions, the current resolution is one level higher than the resolution of the previous resolution, and the sample feature map to be upsampled of the previous resolution is initially the initial sample feature map. The upsampled sample feature map of the current resolution is fused with the second sample feature map corresponding to the current resolution to obtain the fused sample feature map of the current resolution. In response to the current resolution being the preset resolution, the fused sample feature map of the current resolution is used as the second target sample feature map. In response to the current resolution not being the preset resolution, the current fused sample feature map is used as the new sample feature map to be upsampled of the previous resolution, and a resolution one level higher than the current resolution is selected from the multiple resolutions as the new current resolution, and the step of upsampling the sample feature map to be upsampled of the previous resolution is repeated to obtain the upsampled sample feature map of the current resolution.

[0331] In some optional embodiments of the present disclosure, the second training module 930 is specifically used for:

[0332] Based on the sample feature map corresponding to the lowest resolution among multiple resolutions, a sample global feature vector is determined.

[0333] In some optional embodiments of the present disclosure, the second training module 930 is specifically configured to determine the sample global feature vector by any of the following methods:

[0334] Use the global pooling network to perform global pooling processing on the sample feature map corresponding to the lowest resolution to obtain the sample global feature vector. Use the convolution network to perform convolution processing on the sample feature map corresponding to the lowest resolution to obtain the sample global feature vector. Use the fully connected network to process the sample feature map corresponding to the lowest resolution to obtain the sample global feature vector. Expand the sample feature map corresponding to the lowest resolution to obtain the sample global feature vector.

[0335] In some optional embodiments of the present disclosure, the second training module 930 is specifically used for:

[0336] The second sample heat distribution map is processed using a key point regression network to obtain a first sample key point coordinate set of a preset object.

[0337] In some optional embodiments of the present disclosure, the second training module 930 is specifically used for:

[0338] The linear prediction network is used to predict the global feature vector of the sample to obtain the coordinate set of the second sample key points.

[0339] In some optional embodiments of the present disclosure, the second training module 930 is specifically used for:

[0340] The first sample key point coordinate set and the second sample key point coordinate set are spliced ​​to obtain a sample splicing result. The sample splicing result is fused using an output mapping network to obtain sample coordinates of each key point of the preset object.

[0341] In some optional embodiments of the present disclosure, the output mapping network includes any one of a linear network and a channel weighted network.

[0342] In some optional embodiments of the present disclosure, the second training module 930 is specifically used for:

[0343] For any key point of a preset object: in response to the key point being in the sample screenshot, the first sample predicted coordinates corresponding to the key point in the first sample key point coordinate set are used as the sample coordinates of the key point. In response to the key point not being in the sample screenshot, the second sample predicted coordinates corresponding to the key point in the second sample key point coordinate set are used as the sample coordinates of the key point.

[0344] In some optional embodiments of the present disclosure, the preset object includes any one of a human body, a hand, an animal, and a human face.

[0345] In some optional embodiments of the present disclosure, the second training module 930 is specifically used for:

[0346] Based on the first sample prediction coordinates and the true value coordinates of each key point of the preset object in the training sample, a first regression loss is determined. Based on the second sample prediction coordinates and the true value coordinates of each key point of the preset object in the training sample, a second regression loss is determined. Based on the sample coordinates and the true value coordinates of each key point of the preset object in the training sample, a third regression loss is determined. Based on the first regression loss, the second regression loss and the third regression loss, a second loss of the training sample is determined.

[0347] In some optional embodiments of the present disclosure, the sample acquisition module 910 may also be used to:

[0348] Obtain an initial image sample set. The initial image sample set includes multiple initial samples, each of which includes an initial image, a complete initial screenshot including a preset object captured from the initial image, and the true coordinates of each key point of the preset object in the initial image. For each initial sample in the cropped image sample set, crop the initial screenshot of the initial sample to obtain a cropped screenshot. The cropped image sample set is a subset of the initial image sample set. The cropped image sample set can be a complete initial image sample set, or it can include a part of the initial samples in the initial image sample set. The cropped screenshot includes a part of the preset object. Use the cropped screenshot to replace the initial screenshot in the initial sample to obtain an enhanced sample corresponding to the initial sample. Based on the initial image sample set and the enhanced image sample set, an image sample set is obtained.

[0349] It should be noted that the specific implementation of the device for determining the key points of an object in the embodiment of the present disclosure is similar to the specific implementation of the method for determining the key points of an object in the embodiment of the present disclosure. Please refer to the part of the method for determining the key points of an object for details. In order to reduce redundancy, it will not be repeated here.

[0350] Fig.15 is a schematic diagram of the structure of an apparatus for training a key point regression model provided by some other exemplary embodiments of the present disclosure. The apparatus of this embodiment can be used to implement the corresponding method embodiment for training a key point regression model of the present disclosure, such as Fig.15 The device shown includes: a sample acquisition module 910 and a model training module 940.

[0351] The sample acquisition module 910 is used to acquire an image sample set.

[0352] In some optional embodiments of the present disclosure, the image sample set includes multiple training samples, each training sample includes a sample image, a sample screenshot taken from the sample image and including at least a portion of a preset object, and the true value coordinates of each key point of the preset object.

[0353] The model training module 940 is used to perform the following operations on the model to be trained until the training of the model to be trained is completed to obtain a key point regression model:

[0354] For any training sample in any batch, determine the sample global feature vector and sample heat distribution map of the sample screenshot in the training sample. Each batch includes at least one training sample in the image sample set. Based on the sample heat distribution map, predict the first sample key point coordinate set of the preset object. The first sample key point coordinate set includes the first sample predicted coordinates of each key point of the preset object. Based on the sample global feature vector, predict the second sample key point coordinate set of the preset object. The second sample key point coordinate set includes the second sample predicted coordinates of each key point of the preset object. The first sample key point coordinate set and the second sample key point coordinate set are fused to obtain the sample coordinates of each key point of the preset object. Based on the first sample key point coordinate set, the second sample key point coordinate set, the sample coordinates of each key point of the preset object and the true value coordinates, determine the loss of the training sample. Based on the loss of each training sample in the batch, obtain the model loss of the batch. Based on the model loss of the batch, update the network parameters of the model to be trained to obtain the updated model to be trained.

[0355] In some optional embodiments of the present disclosure, the model training module 940 is specifically used to:

[0356] Based on the first sample prediction coordinates and the true value coordinates of each key point of the preset object in the training sample, a first regression loss is determined. Based on the second sample prediction coordinates and the true value coordinates of each key point of the preset object in the training sample, a second regression loss is determined. Based on the sample coordinates and the true value coordinates of each key point of the preset object in the training sample, a third regression loss is determined. Based on the first regression loss, the second regression loss, and the third regression loss, a second loss of the training sample is determined.

[0357] In some optional embodiments of the present disclosure, the sample acquisition module 910 may also be used to:

[0358] Obtain an initial image sample set. The initial image sample set includes multiple initial samples, each of which includes an initial image, an initial screenshot including a preset object captured from the initial image, and the true coordinates of each key point of the preset object in the initial image. For each initial sample in the cropped image sample set, crop the initial screenshot of the initial sample to obtain a cropped screenshot. The cropped image sample set is a subset of the initial image sample set. The cropped screenshot includes a part of the preset object. Replace the initial screenshot in the initial sample with the cropped screenshot to obtain an enhanced sample corresponding to the initial sample. Each enhanced sample constitutes an enhanced image sample set. Based on the initial image sample set and the enhanced image sample set, obtain an image sample set.

[0359] It should be noted that the specific implementation of the device for determining the key points of an object in the embodiment of the present disclosure is similar to the specific implementation of the method for determining the key points of an object in the embodiment of the present disclosure. Please refer to the part of the method for determining the key points of an object for details. In order to reduce redundancy, it will not be repeated here.

[0360] Exemplary Electronic Devices

[0361] An embodiment of the present disclosure further provides an electronic device, including: a processor, and a memory for storing instructions executable by the processor.

[0362] A processor is used to read executable instructions from a memory and execute the executable instructions to implement any of the methods for determining object key points provided in any of the above embodiments of the present disclosure and any of the methods for training a key point regression model provided in any of the above embodiments.

[0363] Fig.16 FIG. 1 is a schematic diagram of a structure of an application embodiment of the electronic device disclosed in the present invention. In this embodiment, the electronic device 100 includes one or more processors 110 and a memory 120 .

[0364] The processor 110 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0365] The memory 120 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 110 may execute the program instructions to implement the methods of the various embodiments of the present disclosure described above and / or other desired functions.

[0366] In one example, the electronic device 100 may further include: an input device 130 and an output device 140 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0367] For example, the input device 130 may be the microphone or microphone array mentioned above, which is used to capture the input signal of the sound source.

[0368] In addition, the input device 130 may also include, for example, a keyboard, a mouse, and the like.

[0369] The output device 140 can output various information to the outside, including the determined distance information, direction information, etc. The output device 140 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0370] Of course, to simplify, Fig.16 Only some of the components related to the present disclosure in the electronic device 100 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 100 may also include any other appropriate components.

[0371] Exemplary computer program products and computer-readable storage media

[0372] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.

[0373] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0374] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used to execute the steps in the method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0375] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0376] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0377] The above description has been given for the purpose of illustration and description, and those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof based on the above description, and these variations are all within the scope of protection of the present disclosure.

Claims

1. A method for determining key points of an object, comprising: Get the original image; intercepting a target image including at least a portion of the target object from the original image; Determining a global feature vector and a heat distribution map of the target image; Based on the heat distribution map, predict a first key point coordinate set of the target object; wherein the first key point coordinate set includes first predicted coordinates of each key point of the target object; Based on the global feature vector, predict a second key point coordinate set of the target object; wherein the second key point coordinate set includes second predicted coordinates of each key point of the target object; The first key point coordinate set and the second key point coordinate set are fused to obtain coordinates of each key point of the target object.

2. The method according to claim 1, wherein: Determining the heat distribution map of the target image includes: Based on the target image, determining a target feature map of a preset resolution; wherein the preset resolution is lower than the resolution of the target image, and the scale of the target feature map is smaller than the scale of the target image; Based on the target feature map, a heat distribution map of the target image is generated.

3. The method according to claim 2, wherein: The step of determining a target feature map with a preset resolution based on the target image includes: Performing step-by-step feature extraction of multiple resolutions of the target image from high to low resolutions to obtain feature maps corresponding to the multiple resolutions; wherein the preset resolution is the highest resolution among the multiple resolutions; and the scales of the feature maps corresponding to the multiple resolutions decrease as the resolution decreases; The feature maps corresponding to the multiple resolutions are fused to obtain a target feature map with the preset resolution.

4. The method according to claim 3, wherein: Determining the global feature vector of the target image includes: The global feature vector is determined based on a feature map corresponding to a lowest resolution among the multiple resolutions.

5. The method according to claim 4, wherein: The global feature vector is determined by any of the following: Performing global pooling processing on the feature map corresponding to the lowest resolution using a global pooling network to obtain the global feature vector; Using a convolutional network to perform convolution processing on the feature map corresponding to the lowest resolution to obtain the global feature vector; Processing the feature map corresponding to the lowest resolution using a fully connected network to obtain the global feature vector; The feature map corresponding to the lowest resolution is expanded to obtain the global feature vector.

6. The method according to claim 3, wherein: The step of fusing the feature maps corresponding to the multiple resolutions to obtain the target feature map of the preset resolution includes: Using a feature map corresponding to the lowest resolution among the multiple resolutions as an initial feature map; selecting a second lowest resolution from the plurality of resolutions as the current resolution; Performing upsampling processing on the feature map to be upsampled of the previous resolution to obtain an upsampled feature map of the current resolution; wherein the current resolution is one level higher than the previous resolution among the multiple resolutions, and the feature map to be upsampled of the previous resolution is initially the initial feature map; The upsampled feature map of the current resolution is fused with the feature map corresponding to the current resolution to obtain a fused feature map of the current resolution; In response to the current resolution being the preset resolution, using the fused feature map of the current resolution as the target feature map; In response to the current resolution not being the preset resolution, the current fused feature map is used as a new feature map to be upsampled at the previous resolution, a resolution one level higher than the current resolution is selected from the multiple resolutions as the new current resolution, and the steps of upsampling the feature map to be upsampled at the previous resolution to obtain the upsampled feature map at the current resolution are repeated.

7. The method according to any one of claims 1 to 6, wherein: The fusing the first key point coordinate set and the second key point coordinate set to obtain the coordinates of each key point of the target object includes: Splicing the first key point coordinate set and the second key point coordinate set to obtain a splicing result; The stitching results are fused using an output mapping network to obtain the coordinates of each key point of the target object.

8. The method according to any one of claims 1 to 6, wherein: The fusing the first key point coordinate set and the second key point coordinate set to obtain the coordinates of each key point of the target object includes: For any key point of the target object: In response to the key point being located in the target image, taking a first predicted coordinate corresponding to the key point in the first key point coordinate set as the coordinate of the key point; In response to the key point not being in the target image, taking the second predicted coordinates corresponding to the key point in the second key point coordinate set as the coordinates of the key point.

9. The method according to any one of claims 1 to 6, wherein: The step of predicting a first key point coordinate set of the target object based on the heat distribution map includes: The heat distribution map is processed using a key point regression network to obtain a first key point coordinate set of the target object.

10. The method according to any one of claims 1 to 6, wherein: The step of predicting a second key point coordinate set of the target object based on the global feature vector comprises: The global feature vector is predicted using a linear prediction network to obtain the second key point coordinate set.

11. The method according to any one of claims 1 to 6, wherein: The target object includes any one of a human body, a hand, an animal, and a human face.

12. A method for training a keypoint regression model, comprising: Acquire an image sample set; wherein the image sample set includes a plurality of training samples, each training sample includes a sample image, a sample screenshot taken from the sample image and including at least a portion of a preset object, and true value coordinates of each key point of the preset object; Perform the following operations on the model to be trained until the first stage of training of the model to be trained is completed to obtain the heat map regression model: For any training sample in any first batch, determine a first sample heat distribution map of the sample screenshots in the training sample; wherein each of the first batches includes at least one training sample in the image sample set; Using the true value coordinates of each key point of the preset object in the training sample, a true value heat distribution map corresponding to the sample screenshot is generated; Determine a first loss of the training sample based on a true value heat distribution map corresponding to the sample screenshot and a first sample heat distribution map; Based on the first loss of each training sample in the first batch, a heat map loss of the first batch is obtained; The network parameters of the model to be trained are updated based on the heat map loss of the first batch to obtain an updated model to be trained; The following operations are performed on the model to be trained until the second stage of training of the model to be trained is completed to obtain a key point regression model. The model to be trained is initially the heat map regression model: For any training sample in any second batch, determine a sample global feature vector and a second sample heat distribution map of the sample screenshot in the training sample; wherein each second batch includes at least one training sample in the image sample set; Based on the second sample heat distribution map, predict a first sample key point coordinate set of the preset object; wherein the first sample key point coordinate set includes the first sample predicted coordinates of each key point of the preset object; Based on the sample global feature vector, predict a second sample key point coordinate set of the preset object; wherein the second sample key point coordinate set includes the second sample predicted coordinates of each key point of the preset object; Fusing the first sample key point coordinate set and the second sample key point coordinate set to obtain sample coordinates of each key point of the preset object; Determine a second loss of the training sample based on the first sample key point coordinate set, the second sample key point coordinate set, and the sample coordinates and true value coordinates of each key point of the preset object; Obtaining a model loss for the second batch based on a second loss of each training sample in the second batch; The network parameters of the model to be trained are updated based on the model losses of the second batch to obtain an updated model to be trained.

13. The method according to claim 12, wherein: Also includes: Acquire an initial image sample set; wherein the initial image sample set includes a plurality of initial samples, each of which includes an initial image, an initial screenshot taken from the initial image and including the preset object, and true value coordinates of each key point of the preset object in the initial image; For each initial sample in the cropped image sample set, cropping the initial screenshot of the initial sample to obtain a cropped screenshot; wherein the cropped image sample set is a subset of the initial image sample set; and the cropped screenshot includes a part of the preset object; The cropped screenshot is used to replace the initial screenshot in the initial sample to obtain an enhanced sample corresponding to the initial sample; each of the enhanced samples constitutes an enhanced image sample set; The image sample set is obtained based on the initial image sample set and the enhanced image sample set.

14. A device for determining key points of an object, comprising: An image acquisition module, used for acquiring original images; A first processing module, used for capturing a target image including at least a part of the target object from the original image; A second processing module, used to determine the global feature vector and heat distribution map of the target image; A third processing module is used to predict a first key point coordinate set of the target object based on the heat distribution map; wherein the first key point coordinate set includes first predicted coordinates of each key point of the target object; A fourth processing module, configured to predict a second key point coordinate set of the target object based on the global feature vector; wherein the second key point coordinate set includes second predicted coordinates of each key point of the target object; The fifth processing module is used to fuse the first key point coordinate set and the second key point coordinate set to obtain the coordinates of each key point of the target object.

15. A device for training a key point regression model, comprising: A sample acquisition module, used to acquire an image sample set; wherein the image sample set includes a plurality of training samples, each training sample includes a sample image, a sample screenshot taken from the sample image and including at least a portion of a preset object, and true value coordinates of each key point of the preset object; The first training module is used to perform the following operations on the model to be trained until the first stage of training of the model to be trained is completed to obtain a heat map regression model: For any training sample in any first batch, determine a first sample heat distribution map of the sample screenshots in the training sample; wherein each of the first batches includes at least one training sample in the image sample set; Using the true value coordinates of each key point of the preset object in the training sample, a true value heat distribution map corresponding to the sample screenshot is generated; Determine a first loss of the training sample based on a true value heat distribution map corresponding to the sample screenshot and a first sample heat distribution map; Based on the first loss of each training sample in the first batch, a heat map loss of the first batch is obtained; The network parameters of the model to be trained are updated based on the heat map loss of the first batch to obtain an updated model to be trained; The second training module is used to perform the following operations through the model to be trained until the second stage of training of the model to be trained is completed to obtain a key point regression model, and the model to be trained is initially the heat map regression model: For any training sample in any second batch, determine a sample global feature vector and a second sample heat distribution map of the sample screenshot in the training sample; wherein each second batch includes at least one training sample in the image sample set; Based on the second sample heat distribution map, predict a first sample key point coordinate set of the preset object; wherein the first sample key point coordinate set includes the first sample predicted coordinates of each key point of the preset object; Based on the sample global feature vector, predict a second sample key point coordinate set of the preset object; wherein the second sample key point coordinate set includes the second sample predicted coordinates of each key point of the preset object; Fusing the first sample key point coordinate set and the second sample key point coordinate set to obtain sample coordinates of each key point of the preset object; Determine a second loss of the training sample based on the first sample key point coordinate set, the second sample key point coordinate set, and the sample coordinates and true value coordinates of each key point of the preset object; Obtaining a model loss for the second batch based on a second loss of each training sample in the second batch; The network parameters of the model to be trained are updated based on the model losses of the second batch to obtain an updated model to be trained.

16. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute any one of the methods for determining key points of an object described in any one of claims 1-11 and the methods for training a key point regression model described in any one of claims 12-13.

17. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement any one of the methods for determining object key points described in any one of claims 1-11 and the methods for training a key point regression model described in any one of claims 12-13.

Images