Attitude estimation method, device, computer equipment and storage medium

Abstract

The present application relates to a posture estimation method, device, computer equipment and storage medium. The method comprises: obtaining a target image to be subjected to posture estimation; the target image includes a target object to be processed; performing feature extraction based on the target image to obtain a first extracted feature; performing feature expansion on the first extracted feature through an image feature expansion network to obtain an expanded image feature; performing feature extraction on the expanded image feature to obtain a second extracted feature; performing feature compression on the second extracted feature through an image feature compression network to obtain a compressed image feature; determining the key point position information corresponding to the target object in the target image based on the compressed image feature, and performing posture estimation on the target object based on the key point position information. This method can improve the efficiency of posture estimation.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a posture estimation method, apparatus, computer device and storage medium. Background Art

[0002] With the development of computer vision technology, posture estimation, as one of the important applications in computer vision, has also developed rapidly and has been widely used in the fields of object activity analysis, video surveillance or object interaction. For example, human posture estimation in posture estimation can detect the key points of the human body in an image containing the human body. For example, the facial features, limbs or joints of the human body can be obtained through human posture estimation. Due to its functions, it is widely used in scenes such as stop-motion animation, collage dance, transparent people, walking splicing or action classification.

[0003] However, current posture estimation methods suffer from low efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a posture estimation method, device, computer equipment and storage medium that can improve the efficiency of posture estimation in order to solve the above technical problems.

[0005] A posture estimation method, the method comprising: obtaining a target image to be subjected to posture estimation; the target image includes a target object to be processed; performing feature extraction based on the target image to obtain a first extracted feature; performing feature expansion on the first extracted feature through an image feature expansion network to obtain an expanded image feature; performing feature extraction on the expanded image feature to obtain a second extracted feature; performing feature compression on the second extracted feature through an image feature compression network to obtain a compressed image feature; determining key point position information corresponding to the target object in the target image based on the compressed image feature, and performing posture estimation on the target object based on the key point position information.

[0006] In one embodiment, the image feature expansion network includes multiple feature convolution channels, and the feature expansion of the first extracted feature through the image feature expansion network to obtain the expanded image feature includes: inputting the first extracted feature into the multiple feature convolution channels corresponding to the image feature expansion network respectively, each of the feature convolution channels convolves the first extracted feature with a feature dimension preserving convolution kernel to obtain the convolution feature output by each of the feature convolution channels; and integrating the convolution features output by each of the feature convolution channels to obtain the expanded image feature.

[0007] In one embodiment, determining the key point position information corresponding to the target object in the target image based on the compressed image feature includes: amplifying the compressed image feature to obtain an amplified image feature; convolving the amplified image feature to obtain a third extracted feature; and determining the key point position information corresponding to the target object in the target image based on the third extracted feature.

[0008] In one of the embodiments, obtaining a target image for posture estimation includes: obtaining an initial image; performing object detection on the initial image to obtain probabilities that multiple candidate image regions in the initial image respectively include the target object; selecting an object image region including the target object from the candidate image regions based on the probability that the candidate image region includes the target object; and extracting a captured image region from the initial image based on the object image region, and using the captured image as the target image for posture estimation.

[0009] In one of the embodiments, extracting a cropped image area from the initial image based on the object image area and using the cropped image as a target image for posture estimation includes: obtaining the center coordinates in the object image area; obtaining the area size corresponding to the object image area, and obtaining an area extension value based on the area size and a size expansion coefficient; extending in an extension direction corresponding to the area extension value based on the center coordinates and the area extension value to obtain extension coordinates; using the image area within the extension coordinates as an image cropped area, and using the cropped image as a target image for posture estimation.

[0010] In one embodiment, there are multiple key point position information, and the method further includes: converting each key point position information into corresponding target point position information according to a mapping relationship between the key point position information and the target point position information; the target point position information is the position information of the key point position information in the initial image; and estimating the posture of the target object based on each target point position information to obtain a target posture corresponding to the target image.

[0011] A method for generating a target video, the method further comprising: obtaining a target action, determining a posture sequence corresponding to the target action, and executing the postures in the posture sequence in sequence to obtain the target action; executing the above-mentioned posture estimation method to obtain the target posture corresponding to each target image in a target image set; obtaining an image corresponding to each target posture in the posture sequence from the target image set as a video frame image; arranging the obtained video frame images according to the order of the postures in the posture sequence to obtain a target video corresponding to the target action.

[0012] A posture estimation device, the device comprising: a target image acquisition module, used to acquire a target image to be subjected to posture estimation; the target image comprises a target object to be processed; a first feature extraction module, used to perform feature extraction based on the target image to acquire a first extracted feature; an expanded image feature acquisition module, used to perform feature expansion on the first extracted feature through an image feature expansion network to obtain an expanded image feature; a second extracted feature acquisition module, used to perform feature extraction on the expanded image feature to obtain a second extracted feature; a compressed image feature acquisition module, used to perform feature compression on the second extracted feature through an image feature compression network to obtain a compressed image feature; and a key point position information determination module, used to determine the key point position information corresponding to the target object in the target image based on the compressed image feature, and perform posture estimation on the target object based on the key point position information.

[0013] In one embodiment, the expanded image feature obtaining module is used to input the first extracted features into multiple feature convolution channels corresponding to the image feature expansion network respectively, and each of the feature convolution channels uses a feature dimension preserving convolution kernel to convolve the first extracted features to obtain the convolution features output by each of the feature convolution channels; the expanded image features are obtained by integrating the convolution features output by each of the feature convolution channels.

[0014] In one of the embodiments, the key point position information determination module is used to amplify the compressed image feature to obtain an amplified image feature; convolve the amplified image feature to obtain a third extracted feature; and determine the key point position information corresponding to the target object in the target image based on the third extracted feature.

[0015] In one of the embodiments, the target image acquisition module is used to acquire an initial image; perform object detection on the initial image to obtain the probability that multiple candidate image areas in the initial image respectively include the target object; based on the probability that the candidate image area includes the target object, select an object image area including the target object from the candidate image area; based on the object image area, extract a captured image area from the initial image, and use the captured image as the target image to be subjected to posture estimation.

[0016] In one of the embodiments, the target image acquisition module is used to obtain the central coordinates in the object image area; obtain the area size corresponding to the object image area, and obtain the area extension value based on the area size and the size expansion coefficient; extend in the extension direction corresponding to the area extension value based on the central coordinates and the area extension value to obtain the extension coordinates; use the image area located within the extension coordinates as the captured image area, and use the captured image as the target image to be subjected to posture estimation.

[0017] In one of the embodiments, the target image acquisition module is used to convert each key point position information into corresponding target point position information according to the mapping relationship between the key point position information and the target point position information; the target point position information is the position information of the key point position information in the initial image; based on each target point position information, the target object is estimated in posture to obtain the target posture corresponding to the target image.

[0018] A target video generation device, the device is used to obtain a target action, determine a posture sequence corresponding to the target action, and execute the postures in the posture sequence in sequence to obtain the target action; obtain the target posture corresponding to each target image in a target image set; obtain the image corresponding to each target posture in the posture sequence from the target image set as a video frame image; arrange the obtained video frame images according to the order of the postures in the posture sequence to obtain a target video corresponding to the target action.

[0019] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program: obtaining a target image to be subjected to posture estimation; the target image comprises a target object to be processed; performing feature extraction based on the target image to obtain a first extracted feature; performing feature expansion on the image extracted feature through an image feature expansion network to obtain an expanded image feature; performing feature extraction on the expanded image feature to obtain a second extracted feature; performing feature compression on the second extracted feature through an image feature compression network to obtain a compressed image feature; determining key point position information corresponding to the target object in the target image based on the compressed image feature, and performing posture estimation on the target object based on the key point position information.

[0020] A computer-readable storage medium stores a computer program, which implements the following steps when executed by a processor: obtaining a target image to be subjected to posture estimation; the target image includes a target object to be processed; performing feature extraction based on the target image to obtain a first extracted feature; performing feature expansion on the image extracted feature through an image feature expansion network to obtain an expanded image feature; performing feature extraction on the expanded image feature to obtain a second extracted feature; performing feature compression on the second extracted feature through an image feature compression network to obtain a compressed image feature; determining key point position information corresponding to the target object in the target image based on the compressed image feature, and performing posture estimation on the target object based on the key point position information.

[0021] The above-mentioned posture estimation method, device, computer equipment and storage medium are as follows: obtaining a target image to be subjected to posture estimation; the target image includes a target object to be processed; extracting features based on the target image to obtain a first extracted feature; performing feature expansion on the first extracted feature through an image feature expansion network to obtain an expanded image feature; extracting features from the expanded image feature to obtain a second extracted feature; performing feature compression on the second extracted feature through an image feature compression network to obtain a compressed image feature; determining the key point position information corresponding to the target object in the target image based on the compressed image feature, and performing posture estimation on the target object based on the key point position information. First, the extracted first extracted feature is feature expanded using an image feature expansion network so that as many image features as possible can be input at the input end of the network for posture estimation, and then the second extracted feature is feature compressed through an image feature compression network. Combining the above features can achieve the purpose of improving the efficiency and accuracy of posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A diagram showing an application environment of a posture estimation method in an embodiment;

[0023] Figure 2 1 is a flow chart of a posture estimation method in one embodiment;

[0024] Figure 3 is a flow chart of a posture estimation method in another embodiment;

[0025] Figure 4 is a flow chart of a posture estimation method in another embodiment;

[0026] Figure 5 is a flow chart of a posture estimation method in another embodiment;

[0027] Figure 6 is a flow chart of a posture estimation method in another embodiment;

[0028] Figure 7 is a flow chart of a posture estimation method in another embodiment;

[0029] Figure 8 is a flow chart of a method for generating a target video in one embodiment;

[0030] Fig. 9 is a schematic diagram of a panoramic image including an object in one embodiment;

[0031] Fig.10 is a schematic diagram of detecting an object in one embodiment;

[0032] Fig.11 A schematic diagram with an object subgraph is captured in one embodiment;

[0033] Fig.12 is a schematic diagram of key points of an object in one embodiment;

[0034] Fig.13 is a schematic diagram of a human body posture model in one embodiment;

[0035] Fig.14 is a structural block diagram of a posture estimation device in one embodiment;

[0036] Fig.15 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0038] The posture estimation method provided in this application can be applied to Figure 1In the application environment shown, it is specifically applied to a posture estimation system. The posture estimation system includes an image acquisition device 102 and a terminal 104, wherein the image acquisition device 102 is connected to the terminal 104 in communication. The terminal 104 executes a posture estimation method, specifically, the terminal 104 obtains a target image to be subjected to posture estimation transmitted from the image acquisition device 102; the target image includes a target object to be processed; the terminal 104 extracts features based on the target image to obtain a first extracted feature; the first extracted feature is expanded by an image feature expansion network to obtain an expanded image feature; the expanded image feature is extracted to obtain a second extracted feature; the second extracted feature is compressed by an image feature compression network to obtain a compressed image feature; the key point position information corresponding to the target object in the target image is determined based on the compressed image feature, and the posture of the target object is estimated based on the key point position information. Among them, the image acquisition device 102 can be, but is not limited to, various devices with image acquisition functions, which can be distributed outside the terminal 104 or inside the terminal 104. For example: various cameras, scanners, various cameras, and image acquisition cards distributed outside the terminal 104. The terminal 104 may be, but is not limited to, various cameras, personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. It is understood that the method provided in the embodiment of the present application may also be executed by a server.

[0039] In one embodiment, Figure 2 As shown, a posture estimation method is provided, which is applied to Figure 1 The terminal in is used as an example to illustrate, including the following steps:

[0040] Step 202: Acquire a target image for which posture estimation is to be performed; the target image includes a target object to be processed.

[0041] Among them, posture estimation refers to the process of estimating the posture of the target object by detecting key points in the target object and describing one or more key points. Key points refer to feature points that can describe the structural features of the target object. For example, the facial features, leg joints or hand joints of the target object. The target object refers to the object for posture estimation. For example, the human body or animals.

[0042] Specifically, the terminal may obtain the target image to be subjected to posture estimation in a direct or indirect manner.

[0043] In one embodiment, the terminal receives the image transmitted from the image acquisition device and including the target object to be processed as the target image.

[0044] In one embodiment, the terminal receives an image transmitted from an image acquisition device, preprocesses the image, and uses the preprocessed image as a target image.

[0045] In one embodiment, the image acquisition device is a panoramic camera. After the panoramic camera acquires a panoramic image, the panoramic image is used as a target image to be subjected to posture estimation, and the target image includes a target object to be processed. The target object may be complete, or may only contain a part of it or be blocked.

[0046] In one embodiment, the terminal extracts frames from a panoramic video to obtain a panoramic image, and directly or preprocesses the panoramic image to obtain a target image for posture estimation. The preprocessing includes normalizing the panoramic image or cropping the target object in the panoramic image.

[0047] Step 204: extract features based on the target image to obtain first extracted features.

[0048] The feature is information representing a property unique to the target image, through which an object in the target image can be identified or the target image can be classified.

[0049] Specifically, feature extraction can be performed on the target image through a feature extraction network to obtain a first extracted feature.

[0050] In one embodiment, a lightweight deep neural network may be used to perform feature extraction on the target image to obtain a first extracted feature.

[0051] Step 206: perform feature expansion on the first extracted feature through an image feature expansion network to obtain an expanded image feature.

[0052] The image feature expansion network refers to a network that can increase the number of image features. The expanded image features refer to image features after the image features are expanded.

[0053] Specifically, the channel of the first extracted feature is expanded through point-by-point convolution to enrich the number of features and obtain expanded image features.

[0054] In one embodiment, after the terminal obtains the first extracted feature, the image feature expansion network uses 1*1 point-by-point convolution to expand the image extracted feature to obtain an expanded image feature.

[0055] Step 208: extract the dilated image features to obtain second extracted features.

[0056] Specifically, after the dilated image features are obtained, feature extraction can be performed on the dilated image features through convolution with fewer parameters to obtain second extracted features.

[0057] In one embodiment, the dilated image features can be downsampled by a preset convolution and activation function, and the dilated image features can be feature extracted to obtain a second extracted feature. For example, the preset convolution can be a 3*3 convolution and a ReLU (Rectified Linear Unit) activation function to extract the dilated image features to obtain a second extracted feature. Based on the adaptability of each application scenario, the above activation function can be replaced by a Sigmoid function (Sigmoid function, S-shaped growth curve), ELU (Exponential Linear Unit), GELU (Gaussian Error Linear Unit), etc.

[0058] Step 210: compress the second extracted features through an image feature compression network to obtain compressed image features.

[0059] The image feature compression network refers to a network that can reduce the number of image features. The compressed image features refer to image features after the image features are compressed.

[0060] Specifically, after obtaining the second extracted features, the terminal may perform feature compression on the second extracted features so as to increase the speed at which the terminal performs posture estimation.

[0061] In one embodiment, after the terminal obtains the second extracted feature, the image feature compression network uses 1*1 point-by-point convolution to compress the second extracted feature, and obtains the compressed image feature after linear transformation.

[0062] Step 212: determine key point position information corresponding to the target object in the target image based on the compressed image features, and perform posture estimation on the target object based on the key point position information.

[0063] The key point position information refers to information that can determine the position of the key point in the target image, for example, the coordinates, name or direction of the key point in the target image.

[0064] Specifically, after obtaining the compressed image features, the terminal may obtain the key point position information corresponding to the target object through the correspondence between the compressed image features and the key point position information corresponding to the target object.

[0065] In one embodiment, the terminal stores a matching relationship table between compressed image features and key point position information. After obtaining the compressed image features, the terminal obtains the corresponding key point position information by traversing the matching relationship table. According to the position coordinates and names in the key point position information, the posture of the target object is estimated. For example, the key point position information obtained by the terminal after traversing the matching relationship table is (200, 200, wrist joint), indicating that the position coordinates of the key point are (200, 200), and the key point is the wrist joint. The posture is estimated through the description of multiple key point position information.

[0066] In the above-mentioned posture estimation method, a target image to be subjected to posture estimation is obtained; the target image includes a target object to be processed; feature extraction is performed based on the target image to obtain a first extracted feature; feature expansion is performed on the first extracted feature through an image feature expansion network to obtain an expanded image feature; feature extraction is performed on the expanded image feature to obtain a second extracted feature; feature compression is performed on the second extracted feature through an image feature compression network to obtain a compressed image feature; key point position information corresponding to the target object in the target image is determined based on the compressed image feature, and posture estimation is performed on the target object based on the key point position information. The first extracted feature is firstly feature expanded through an image feature expansion network so that as many image features as possible can be input at the input end of the network for posture estimation, and then the second extracted feature is feature compressed through an image feature compression network. Combining the above features can achieve the purpose of improving the efficiency and accuracy of posture estimation.

[0067] In one embodiment, Figure 3 As shown, the image feature expansion network includes multiple feature convolution channels. The first extracted feature is expanded through the image feature expansion network to obtain expanded image features including:

[0068] Step 302, input the first extracted features into multiple feature convolution channels corresponding to the image feature expansion network respectively, and each feature convolution channel convolves the first extracted features using a feature dimension preserving convolution kernel to obtain the convolution features output by each feature convolution channel.

[0069] The feature dimension preserving convolution kernel refers to a convolution kernel that can keep the dimension of the image unchanged, and the dimension of the image refers to the number of channels of the image. For example, a convolution kernel with a size of 1*1.

[0070] Specifically, the terminal can maintain the convolution kernel by setting the feature dimension and convolve the first extracted feature from each feature convolution channel. The convolution feature output by the feature convolution channel can be obtained using fewer parameters while keeping the scale of the first extracted feature unchanged.

[0071] In one embodiment, the terminal maintains the number and size of convolution kernels by setting the feature dimension, convolves the image extraction features of each feature convolution channel, and obtains the convolution features output by the feature convolution channel. For example, on a 3*3 convolution network with 64 channels, after adding a convolution kernel with a size of 1*1 and a number of channels of 256, the number of channels of the original network can be expanded from 64 to 256 through 64*256 parameters.

[0072] Step 304, the convolution features output by each feature convolution channel are integrated to obtain the dilated image features.

[0073] Specifically, after obtaining the convolution features output by the feature convolution channel, the feature dimension-preserving convolution kernel can linearly combine each pixel in the first extracted feature in the image on different channels to obtain the dilated image features. For example, the composition of the dilated network is to add a 1*1, 28-channel convolution kernel after the 3*3, 64-channel convolution kernel, transforming it into a 3*3, 28-channel convolution kernel. The original 64 channels are linearly combined across channels to become 28 channels, realizing information interaction between channels, and obtaining the dilated image features through the convolution features output by each feature convolution channel.

[0074] In this embodiment, by using a feature dimension preserving convolution kernel to convolve the first extracted feature in the feature convolution channel in the image feature expansion network, the convolution feature output by the feature convolution channel is obtained, and the convolution features output by each feature convolution channel are combined to obtain the expanded image feature, which can achieve the purpose of obtaining the expanded image feature with a smaller number of parameters, thereby improving the efficiency of posture estimation.

[0075] In one embodiment, Figure 4 As shown, determining the key point position information corresponding to the target object in the target image based on the compressed image features includes:

[0076] Step 402: amplify the compressed image feature to obtain an amplified image feature.

[0077] Specifically, after obtaining the compressed image features, the features are up-sampled to obtain the enlarged image features.

[0078] In one embodiment, the terminal amplifies the compressed image features by setting a three-layer sampling network, and setting the number of input and output channels of the three-layer sampling network to (256, 128), (128, 64), and (64, 64), respectively, which can reduce the amount of network parameters and the amount of calculation.

[0079] In one embodiment, the terminal interpolates the compressed image features by an interpolation method to obtain the enlarged image features. For example, based on the compressed image features, a new element is inserted between pixels using a suitable interpolation algorithm such as linear interpolation or bilinear interpolation.

[0080] Step 404, convolve the enlarged image feature to obtain a third extracted feature.

[0081] Specifically, after obtaining the enlarged image feature, in order to compensate for the reduction of nonlinear units in the process of enlarging the compressed image feature, the enlarged image feature is convolved to obtain the third extracted feature.

[0082] Step 406: Determine key point position information corresponding to the target object in the target image based on the third extracted feature.

[0083] Specifically, after the third extracted feature is obtained, the third extracted feature is searched and filtered to determine the key point position information corresponding to the target object in the target image.

[0084] In one embodiment, a matching list of image features and key point position information is stored in the terminal. After the third extracted feature is obtained, the matching list is traversed to obtain the key point position information corresponding to the third extracted feature, that is, the key point position information corresponding to the target object in the target image.

[0085] In this embodiment, the compressed image features are amplified to obtain enlarged image features, and the enlarged image features are convolved to obtain a third extracted feature. Based on the third extracted feature, the key point position information corresponding to the target object in the target image is determined, which can achieve the purpose of improving the image output quality, thereby achieving the purpose of more accurately obtaining the key point position information corresponding to the target object.

[0086] In one embodiment, Figure 5 As shown, obtaining the target image to be subjected to posture estimation includes:

[0087] Step 502: Acquire an initial image.

[0088] The initial image refers to an original image that has not been processed. The original image is an image directly obtained by an image acquisition device or terminal.

[0089] In one embodiment, the terminal can capture the initial image through a connected image capture device, and the capture device transmits the captured initial image to the terminal in real time; or the capture device temporarily stores the captured initial image locally, and when receiving an image acquisition instruction from the terminal, the locally stored initial image is transmitted to the terminal. Accordingly, the terminal can acquire the initial image.

[0090] In one embodiment, the terminal collects the initial image through an internal image collection module, and stores the collected image in a terminal memory. When the terminal needs to obtain the initial image, the initial image is obtained from the memory.

[0091] Step 504: perform object detection on the initial image to obtain the probability that a plurality of candidate image regions in the initial image respectively include the target object.

[0092] Specifically, after acquiring the initial image, the initial image is segmented into multiple image sub-regions as candidate image regions, and the probability of the target object in each candidate image region is detected. For example, the image is segmented into sub-region A, sub-region B, and sub-region C, the probability of the target object in sub-region A is 0%, the probability of the target object in sub-region B is 10%, and the probability of the target object in sub-region C is 90%.

[0093] In one embodiment, the terminal obtains the probability that each image sub-region includes the target object by gradually reducing the size of the image sub-region.

[0094] Step 506: Select an object image region including the target object from the candidate image regions based on the probability that the candidate image regions include the target object.

[0095] Specifically, after obtaining the probabilities that multiple candidate image areas in the initial image respectively include the target object, the terminal can compare the probabilities of each candidate image area to obtain a candidate image area whose probability is within a preset probability threshold range, and use the candidate image area as the object image area including the target object.

[0096] In one embodiment, the terminal traverses the probability that the candidate image region includes the target object, obtains the maximum probability value among the probabilities, and uses the candidate image region corresponding to the maximum probability value as the object image region of the target object.

[0097] Step 508: extract a captured image region from the initial image according to the object image region, and use the captured image as a target image for posture estimation.

[0098] Specifically, after obtaining the object image area including the target object, the terminal can intercept the image of the obtained image area as the target image to be subjected to posture estimation based on the position information of the object image area, so as to reduce the amount of calculation during posture estimation and improve the efficiency of posture estimation.

[0099] In one embodiment, the terminal may extract coordinate information of the object image region and use the coordinate information to capture a target image for posture estimation.

[0100] In one embodiment, by receiving a target object selection operation from a user, the selected image area can be used as the object image area, and the selected image area can be captured in the initial image as the target image to be subjected to posture estimation.

[0101] In this embodiment, an initial image is acquired, object detection is performed on the initial image, and the probabilities that multiple candidate image areas in the initial image respectively include the target object are obtained. Based on the probabilities that the candidate image areas include the target object, object image areas including the target object are selected from the candidate image areas, and the object image areas are extracted from the initial image. The captured image is used as the target image to be subjected to posture estimation, thereby achieving the purpose of accurately obtaining the target image from the initial image.

[0102] In one embodiment, Figure 6 As shown, according to the object image area, a cut-off image area is extracted from the initial image, and the cut-off image is used as the target image to be subjected to posture estimation, including:

[0103] Step 602, obtaining the center coordinates of the object image area.

[0104] The center coordinates refer to the coordinates of the pixel at the center of the object image area. The coordinates are based on the coordinates of the pixel at the center of the object image area of ​​the target object in the initial image, and can be determined based on the length and width of the initial image.

[0105] Specifically, after determining the object image region, the terminal obtains the coordinates of the pixel point by obtaining the pixel point at the center of the object image region through a pixel point coordinate acquisition tool.

[0106] Step 604 , obtaining a region size corresponding to the object image region, and obtaining a region extension value based on the region size and the size expansion coefficient.

[0107] The region size refers to the region length and region width of the object image region. For example, if the region length of the object image region is h and the region width of the object image region is w, then the region size is w*h. The size expansion coefficient refers to the coefficient that can increase the region size. The region extension value refers to the increase value of the region size obtained by correcting the region size using the size expansion coefficient.

[0108] Specifically, after obtaining the center coordinates of the object image area, the terminal may obtain the area size corresponding to the object image area, and obtain the area extension value through the functional relationship between the area size and the size expansion coefficient.

[0109] In one embodiment, the terminal obtains the region size corresponding to the object image region through an image size measurement tool, and obtains the region extension value by using the product relationship between the region size and the size expansion coefficient. For example, the region width in the region size is w, the size expansion coefficient is exp_ratio, and the region extension value of the region width is w*exp_ratio*1.2 / 2; similarly, the region extension value of the region length in the region size can also be obtained through the corresponding size expansion coefficient.

[0110] Step 606: Extend in an extension direction corresponding to the region extension value based on the center coordinates and the region extension value to obtain extension coordinates.

[0111] The extension direction refers to the direction in which the width and length increase corresponding to the region extension value. The extension coordinates refer to the coordinates of the object image region obtained by extending the object image region in the extension direction with the center coordinates as the reference point. The coordinates can be represented by the upper left corner coordinates and the lower right corner coordinates of the object image region.

[0112] Specifically, after obtaining the region extension value, the terminal can use the central coordinate as a reference point and use the region extension value to expand the object image region to obtain the extension coordinates corresponding to the object image region, so that the object image region can include a more complete target object.

[0113] In one embodiment, the center coordinates are expressed as (x, y), and the extended coordinates are (x0, y0) and (x1, y1), where x0 and x1 are the coordinates of the object image area extension value in the width extension direction of the image, and y0 and y1 are the coordinates of the object image area extension value in the length extension direction of the image. The extended coordinates can be expressed as a formula:

[0114] x0=int(xw*exp_ratio*1.2 / 2)

[0115] x1=int(x+w*exp_ratio*1.2 / 2)

[0116] y0=int(yh*exp_ratio*0.8 / 2)

[0117] y1=int(y+h*exp_ratio*0.8 / 2)

[0118] In one embodiment, when the region extension value in the width extension direction in the region extension value is less than or equal to 0, the extension value is taken as zero; when the region extension value is greater than or equal to the width of the initial image, the width of the initial image is used as the region extension value. Similarly, when the region extension value in the length extension direction in the region extension value is less than or equal to zero, the region extension value is taken as zero; when the region extension value is greater than or equal to the height of the initial image, the height of the initial image is used as the region extension value.

[0119] Step 608: Use the image area within the extended coordinates as the captured image area, and use the captured image as the target image for posture estimation.

[0120] Specifically, after acquiring the extended coordinates, the terminal may use the image area within the extended coordinates as the captured image area, and use the captured image as the target image to be subjected to posture estimation.

[0121] In this embodiment, by obtaining the center coordinates of the object image area and the area size corresponding to the object image area, the area extension value is obtained based on the area size and the size expansion coefficient, and the area is extended in the extension direction corresponding to the area extension value based on the center coordinates and the area extension value to obtain the extension coordinates. The image area located within the extension coordinates is used as the captured image area, and the captured image is used as the target image to be subjected to posture estimation. This can achieve the purpose of accurately capturing the target image, thereby improving the efficiency of posture estimation.

[0122] In one embodiment, Figure 7 As shown, there are multiple key point position information, and the method further includes:

[0123] Step 702: convert each key point position information into corresponding target point position information according to the mapping relationship between the key point position information and the target point position information; the target point position information is the position information of the key point position information in the initial image.

[0124] The position information refers to information related to the position of a certain position point. The position point may be a key point or other point with the same structure or function as the key point. The position information may be the coordinate information of the key point or information describing the position of the key point. For example, the key point position information may be represented as (100, 100, eye).

[0125] Specifically, there is a one-to-one correspondence between the key point position information in the target image and the target point position information in the initial image, and they can be converted to each other. After knowing the key point position information, the position information of the key point in the initial image can be obtained accordingly, so that the position information in the initial image obtained in the target image can be accurately reflected in the initial image.

[0126] In one embodiment, the key point position information of the jth key point is expressed as (x_keypoints_j, y_keypoints_j), the vertex coordinates of the upper left corner of the i-th target image in the initial image are expressed as (x_person_i, y_person_i), and the coordinates of the key point in the initial image are expressed as (x_original_keypoints, y_original_keypoints). Then, the coordinates of the key point in the initial image are expressed as the formula:

[0127] x_original_keypoints=x_person_i+x_keypoints_j

[0128] y_original_keypoints=y_person_i+y_keypoints_j

[0129] Step 704 , estimating the posture of the target object based on the position information of each target point, and obtaining the target posture corresponding to the target image.

[0130] Specifically, after determining the position information of each target point corresponding to the multiple key points, the terminal performs posture estimation based on the correspondence between the specific type of the key point and the position information of the target point to obtain the target posture corresponding to the target image.

[0131] In this embodiment, through the mapping relationship between key point position information and target point position information, each key point position information is converted into corresponding target point position information, and the posture of the target object is estimated through each target point position information to obtain the target posture corresponding to the target image, so as to achieve the purpose of obtaining the target posture in the target image.

[0132] In one embodiment, Figure 8 As shown, the target video generation method includes:

[0133] Step 802, obtaining a target action, determining a posture sequence corresponding to the target action, wherein the postures in the posture sequence are executed in sequence to obtain the target action.

[0134] The target action refers to the action obtained after each posture is executed in sequence. The posture refers to each sub-action that makes up the action. For example, the target action is arm extension, which is composed of multiple sub-actions such as arm flat, arm straight, and arm together and sideways. Multiple postures can form a posture sequence according to the order of front and back, and the target action is obtained when the posture sequence is executed in sequence.

[0135] Step 804: According to the steps in the above method embodiments, the target posture corresponding to each target image in the target image set is obtained.

[0136] Specifically, after acquiring the target action, the terminal can obtain each posture based on the above posture estimation methods. The postures constituting the target action are in different target images, and the corresponding postures can be obtained from each target image. For example, target image E contains target posture F or target image G contains target posture H, and so on.

[0137] In one embodiment, according to the correspondence between the posture and the image, the target posture corresponding to each target image in the target image set is obtained.

[0138] Step 806 , obtaining an image corresponding to each target posture in the posture sequence from the target image set as a video frame image.

[0139] Specifically, after acquiring the target posture, the terminal obtains images corresponding to the target posture according to the target posture, and uses each obtained image as a video frame image.

[0140] In one embodiment, a timestamp corresponding to an image corresponding to the target posture is obtained, and the images carrying the respective timestamps are used as video frame images.

[0141] Step 808 , arranging the obtained video frame images according to the order of the postures in the posture sequence to obtain the target video corresponding to the target action.

[0142] Specifically, there is a one-to-one correspondence between the postures in the posture sequence and the corresponding video frame images. The corresponding video frame images are arranged according to the order of the postures in the posture sequence to obtain the target video corresponding to the target action.

[0143] In one embodiment, there is a binding correspondence between the order of postures in the posture sequence and the timestamps of the video frame images. The postures in the posture sequence are sorted, that is, after the video frame images are obtained, the video frame images are arranged according to the timestamps of the video frame images, and the target video is obtained based on the video frame images.

[0144] In this embodiment, by obtaining the target posture corresponding to the target action and each target image in the target image set, the image corresponding to each target posture in the posture sequence is obtained from the target image set as a video frame image, and the obtained video frame images are arranged according to the order of the postures in the posture sequence to obtain the target video corresponding to the target action. This can achieve the purpose of obtaining the target video corresponding to the target action through posture estimation, so that the posture estimation can be realized and applied in practice.

[0145] In one embodiment, taking the terminal as a panoramic camera and the target object as a human body as an example, Fig. 9 As shown in FIG, a panoramic image is obtained by extracting a video frame from a panoramic image of a panoramic camera or a panoramic video shot by a panoramic camera. The image usually needs to contain a human target object for human posture estimation. The human target object can be complete, or only contain a part of it or be blocked. Fig.10 As shown, after normalizing the panoramic image, the coordinate values ​​of the human body boundary frame B1 are obtained through human body tracking or detection algorithms, and the coordinate values ​​of the expanded frame B2 are obtained through the coordinate values ​​of the human body boundary frame or after the human body boundary frame is expanded. Fig.11 As shown in FIG, the panoramic image is cropped to obtain a sub-panoramic image with a border of B2. After the sub-panoramic image is normalized, the sub-panoramic image is input into the trained human posture estimation model, as shown in FIG. Fig.12 As shown in the figure, the heat map of the first key point C is obtained, and the sub-panoramic image is input into the trained human pose estimation model to obtain the heat map of the second key point. By analogy, by obtaining multiple heat maps, a preset number of heat maps with key points are obtained in sequence. The coordinates of the key points in the heat map are mapped to the sub-panoramic image, and then the positions of the key points in the sub-panoramic image are mapped to the panoramic image, so as to obtain the positions of the key points in the panoramic image, and thus estimate the human pose.

[0146] In one embodiment, the terminal performs normalization processing on a panoramic image or a sub-panoramic image, and can obtain the pixel value of the pixel in the normalized image through the positive proportional relationship between the pixel value of the pixel in the normalized image and the difference between the pixel value of the pixel in the original image and the average value of the pixel value. Assuming that the pixel value of a certain pixel in the normalized image is represented by X_normalization, the pixel value of a certain pixel in the panoramic image or the sub-panoramic image is represented by X, the average value of the pixel values ​​of all pixels in the panoramic image or the sub-panoramic image is represented by mean, and the positive proportionality coefficient is represented by std, then X_normalization is represented by the formula:

[0147] X_normalization=(X-mean) / std

[0148] It can be understood that the std may be the variance of all pixels in the panoramic image or the sub-panoramic image; a certain pixel in the panoramic image or the sub-panoramic image may be a pixel of RGB (red, green and blue) three channels.

[0149] In one embodiment, the terminal may obtain the coordinate values ​​of the human body bounding box through a human body detection algorithm, for example, using Faster RCNN (Faster Region-CNN), YOLO (You Only Look Once) series algorithms, SSD (Single Shot MultiBox Detector) series algorithms, or tracking algorithms such as Siamese (Siamese network) tracking algorithms.

[0150] In one embodiment, Fig.13 As shown, the human posture estimation model can reduce the number of image feature blocks between stages in HRNet, for example, by changing the number of downsampled image feature blocks in the second stage to 1, so that the human posture estimation model can reduce the amount of parameters and calculations, thereby improving the efficiency of human posture estimation.

[0151] It should be understood that although Figure 2-8 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-8 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0152] In one embodiment, Fig.14As shown, a posture estimation device 1400 is provided, including: a target image acquisition module 1402, a first feature extraction module 1404, an expanded image feature acquisition module 1406, a second feature extraction acquisition module 1408, a compressed image feature acquisition module 1410 and a key point position information determination module 1412, wherein: the target image acquisition module 1402 is used to acquire a target image to be subjected to posture estimation; the target image includes a target object to be processed; the first feature extraction module 1404 is used to perform feature extraction based on the target image to obtain a first extracted feature; the expanded image feature acquisition module 1406 is used to perform feature expansion on the first extracted feature through an image feature expansion network to obtain an expanded image feature; the second feature extraction module 1408 is used to perform feature extraction on the expanded image feature to obtain a second extracted feature; the compressed image feature acquisition module 1410 is used to perform feature compression on the second extracted feature through an image feature compression network to obtain a compressed image feature; the key point position information determination module 1412 is used to determine the key point position information corresponding to the target object in the target image based on the compressed image feature, and perform posture estimation on the target object based on the key point position information.

[0153] In one embodiment, the expanded image feature acquisition module 1406 is used to input the first extracted features into multiple feature convolution channels corresponding to the image feature expansion network respectively, and each feature convolution channel uses a feature dimension preserving convolution kernel to convolve the first extracted features to obtain the convolution features output by each feature convolution channel; the expanded image features are obtained by integrating the convolution features output by each feature convolution channel.

[0154] In one embodiment, the key point position information determination module 1412 is used to amplify the compressed image features to obtain amplified image features; convolve the amplified image features to obtain third extracted features; and determine the key point position information corresponding to the target object in the target image based on the third extracted features.

[0155] In one embodiment, the target image acquisition module 1402 is used to acquire an initial image; perform object detection on the initial image to obtain the probability that multiple candidate image areas in the initial image respectively include the target object; based on the probability that the candidate image area includes the target object, select an object image area including the target object from the candidate image area; based on the object image area, extract a captured image area from the initial image, and use the captured image as the target image to be subjected to posture estimation.

[0156] In one embodiment, the target image acquisition module 1402 is used to obtain the central coordinates in the object image area; obtain the area size corresponding to the object image area, and obtain the area extension value based on the area size and the size expansion coefficient; extend in the extension direction corresponding to the area extension value based on the central coordinates and the area extension value to obtain the extension coordinates; use the image area located within the extension coordinates as the captured image area, and use the captured image as the target image to be subjected to posture estimation.

[0157] In one embodiment, the target image acquisition module 1402 is used to convert each key point position information into corresponding target point position information according to the mapping relationship between the key point position information and the target point position information; the target point position information is the position information of the key point position information in the initial image; based on each target point position information, the posture of the target object is estimated to obtain the target posture corresponding to the target image.

[0158] In one embodiment, a target video generating device is used to obtain a target action, determine a posture sequence corresponding to the target action, and execute the postures in the posture sequence in sequence to obtain the target action; obtain the target posture corresponding to each target image in a target image set; obtain the image corresponding to each target posture in the posture sequence from the target image set as a video frame image; arrange the obtained video frame images according to the order of the postures in the posture sequence to obtain a target video corresponding to the target action.

[0159] For the specific definition of the posture estimation device, please refer to the definition of the posture estimation method above, which will not be repeated here. Each module in the above-mentioned posture estimation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0160] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.15As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a posture estimation method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0161] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0163] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0164] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0165] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A posture estimation method, characterized in that: The method comprises: Acquire a target image to be subjected to posture estimation; the target image includes a target object to be processed; Perform feature extraction based on the target image to obtain a first extracted feature; The first extracted features are respectively input into a plurality of feature convolution channels corresponding to an image feature expansion network, and each of the feature convolution channels convolves the first extracted features using a feature dimension preserving convolution kernel to obtain an expanded image feature; the image feature expansion network is a network capable of increasing the number of image features; Performing feature extraction on the dilated image features to obtain second extracted features; Performing feature compression on the second extracted feature through an image feature compression network to obtain a compressed image feature; the image feature compression network is a network that can reduce the number of image features; The key point position information corresponding to the target object in the target image is determined based on the compressed image features, and the posture of the target object is estimated based on the key point position information.

2. The method according to claim 1, characterized in that The step of inputting the first extracted features into a plurality of feature convolution channels corresponding to the image feature expansion network respectively, and each of the feature convolution channels convolving the first extracted features with a feature dimension preserving convolution kernel to obtain an expanded image feature comprises: Inputting the first extracted features into a plurality of feature convolution channels corresponding to the image feature expansion network respectively, and each of the feature convolution channels convolves the first extracted features using a feature dimension preserving convolution kernel to obtain convolution features output by each of the feature convolution channels; The dilated image features are obtained by integrating the convolution features output by each of the feature convolution channels.

3. The method according to claim 1, characterized in that The determining the key point position information corresponding to the target object in the target image based on the compressed image feature comprises: Amplifying the compressed image feature to obtain an amplified image feature; Convolving the enlarged image feature to obtain a third extracted feature; Based on the third extracted feature, key point position information corresponding to the target object in the target image is determined.

4. The method according to claim 1, characterized in that: The acquiring of the target image to be subjected to posture estimation comprises: Get the initial image; Performing object detection on the initial image to obtain probabilities that a plurality of candidate image regions in the initial image respectively include a target object; Selecting an object image region including the target object from the candidate image region based on the probability that the candidate image region includes the target object; According to the object image area, a clipped image area is extracted from the initial image, and the clipped image is used as a target image to be subjected to posture estimation.

5. The method according to claim 4, characterized in that The extracting a captured image region from the initial image according to the object image region and using the captured image as a target image to be subjected to posture estimation comprises: Acquire the center coordinates in the object image area; Acquire a region size corresponding to the object image region, and obtain a region extension value based on the region size and a size expansion coefficient; Extending in an extension direction corresponding to the area extension value based on the center coordinates and the area extension value to obtain an extension coordinate; The image region located within the extended coordinates is used as a clipped image region, and the clipped image is used as a target image to be subjected to posture estimation.

6. The method according to claim 4, characterized in that The key point position information is multiple, and the method further includes: According to the mapping relationship between the key point position information and the target point position information, each of the key point position information is converted into corresponding target point position information; the target point position information is the position information of the key point position information in the initial image; The target object is estimated based on the position information of each target point to obtain the target pose corresponding to the target image.

7. A target video generation method, characterized in that: The method comprises: Acquire a target action, determine a posture sequence corresponding to the target action, and execute the postures in the posture sequence in sequence to obtain the target action; Process the target image based on any one of the methods in weights 1 to 6 to obtain the target posture corresponding to each target image in the target image set; Acquire an image corresponding to each target posture in the posture sequence from the target image set as a video frame image; The obtained video frame images are arranged according to the order of the postures in the posture sequence to obtain the target video corresponding to the target action.

8. A posture estimation device, characterized in that: The device comprises: A target image acquisition module, used to acquire a target image to be subjected to posture estimation; the target image includes a target object to be processed; A first feature extraction module, used to extract features based on the target image to obtain a first extracted feature; An expanded image feature obtaining module, used for inputting the first extracted features into a plurality of feature convolution channels corresponding to the image feature expansion network respectively, each of the feature convolution channels convolving the first extracted features with a feature dimension preserving convolution kernel to obtain an expanded image feature; the image feature expansion network is a network capable of increasing the number of image features; A second feature extraction module is used to extract the dilated image features to obtain second extracted features; A compressed image feature obtaining module, used for compressing the second extracted feature through an image feature compression network to obtain a compressed image feature; the image feature compression network is a network that can reduce the number of image features; A key point position information determination module is used to determine the key point position information corresponding to the target object in the target image based on the compressed image features, and perform posture estimation on the target object based on the key point position information.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 or claim 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 or claim 7 are implemented.

Images