A key point detection method, a computer readable storage medium and an electronic device

Abstract

The present application relates to the technical field of image recognition, and specifically provides a key point detection method, a computer readable storage medium and an electronic device, aiming at solving the problem that the existing heat map method is prone to key point prediction disorder when predicting human key points. To this end, the key point detection method of the present application comprises: implementing a key point detection method based on a key point detection model, the key point detection model comprising a convolutional neural network and a fully connected neural network arranged in parallel, obtaining a first prediction result of key points of a to-be-detected image based on the convolutional neural network; determining a second prediction result of key points of the to-be-detected image based on the fully connected neural network; and determining a key point detection result of the to-be-detected image according to the first prediction result, the second prediction result and a disorder condition distance threshold. The fully connected neural network can pay more attention to overall features, and by combining the convolutional neural network and the fully connected neural network, the problem of key point prediction disorder can be effectively avoided.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, specifically providing a key point detection method, a computer-readable storage medium, and an electronic device. Background Technology

[0002] In applications such as pedestrian video structuring and pedestrian re-identification, after obtaining human detection results, it is necessary to further obtain human pose information for subsequent calculations of human behavior, human integrity, and other related information. Typically, human pose estimation requires determining the positions of multiple joints, such as the head, torso, hands, and feet. Therefore, human pose can be described using human keypoints, where one keypoint represents a joint. Thus, after obtaining the location of a single human region through the human detection module, single-target human keypoint detection can be further performed to estimate human pose.

[0003] Existing technologies typically employ heatmap methods to predict key points in the human body. While this method maintains high accuracy for detecting local areas, it suffers from poor prediction results and the tendency for joint point misalignment in the prediction results, particularly regarding the consistency of the human joint skeleton. Summary of the Invention

[0004] The present invention aims to solve the above-mentioned technical problems, namely, to solve the problem that the existing method of using heatmaps to predict key points of the human body is prone to errors in key point prediction.

[0005] In a first aspect, the present invention provides a keypoint detection method, the keypoint detection method being implemented based on a keypoint detection model, the keypoint detection model comprising a convolutional neural network and a fully connected neural network configured in parallel, the method comprising:

[0006] The first prediction result of key points of the image to be detected is obtained based on the convolutional neural network.

[0007] The second prediction result of key points of the image to be detected is determined based on the fully connected neural network.

[0008] The key point detection result of the image to be detected is determined based on the first prediction result of the key point, the second prediction result of the key point, and the disorder condition distance threshold.

[0009] In some embodiments, determining the keypoint detection result of the image to be detected based on the first keypoint prediction result, the second keypoint prediction result, and the disordered conditional distance threshold includes:

[0010] Calculate the absolute value of the difference between the first prediction result and the second prediction result of the key point;

[0011] The absolute value of the difference is compared with the disorder condition distance threshold, and the key point detection result of the image to be detected is determined based on the comparison result.

[0012] In some embodiments, determining the key point detection result of the image to be detected based on the comparison result includes:

[0013] When the absolute value of the difference is greater than the disordered condition distance threshold, the second prediction result of the key point is output as the key point detection result; when the absolute value of the difference is less than or equal to the disordered condition distance threshold, the first prediction result of the key point is output as the key point detection result.

[0014] In some embodiments, obtaining the first prediction result of key points in the image to be detected based on the convolutional neural network includes:

[0015] The key point heatmap of the image to be detected is obtained based on the convolutional neural network.

[0016] The first prediction result of the key points of the image to be detected is determined based on the key point heatmap.

[0017] In some embodiments, the keypoint detection model further includes a keypoint bias network, and the method further includes:

[0018] The keypoint offset amount of the image to be detected is obtained based on the keypoint offset network.

[0019] The first prediction result of the key point is corrected based on the key point bias.

[0020] In some embodiments, the method further includes at least one preprocessing operation among image scaling, image data type conversion, and image normalization of the image to be detected.

[0021] In some embodiments, the keypoint detection model is trained through the following steps:

[0022] Obtain an image training sample set, which includes multiple image training samples, each of which is labeled with key point information;

[0023] Input the image training samples in the image training sample set into the initial key point detection model to obtain the key point detection results of the image training samples;

[0024] The loss value is calculated based on the keypoint detection results and the keypoint information, and the parameters of the initial keypoint detection model are updated based on the loss value to obtain the trained keypoint detection model.

[0025] In some embodiments, after obtaining the image training sample set, the method further includes: performing data augmentation on the image training samples using at least one of the processing methods of flipping, blurring, occlusion, and truncation.

[0026] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the key point detection method described in any one of the preceding claims.

[0027] In a third aspect, the present invention provides an electronic device including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the key point detection method described in any of the preceding claims.

[0028] By adopting the above technical solution, this invention can implement a keypoint detection method based on a keypoint detection model. The keypoint detection model includes a convolutional neural network and a fully connected neural network configured in parallel. The method obtains a first prediction result of keypoints in the image to be detected based on the convolutional neural network; determines a second prediction result of keypoints in the image to be detected based on the fully connected neural network; and determines the final keypoint detection result of the image to be detected based on the first prediction result, the second prediction result, and a distance threshold for error conditions. The fully connected neural network can focus more on overall features. This method, by combining convolutional neural networks and fully connected neural networks, can effectively avoid the problem of keypoint prediction errors. Attached Figure Description

[0029] The preferred embodiments of the present invention are described below with reference to the accompanying drawings, in which:

[0030] Figure 1 This is a schematic flowchart of a key point detection method provided in an embodiment of the present invention;

[0031] Figure 2 This is a schematic flowchart of a key point detection method provided in another embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the key point detection model training method provided in an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of the electronic device structure provided by the present invention. Detailed Implementation

[0034] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0035] Existing technologies typically use heatmaps to predict key points in the human body. While this method can maintain high accuracy for detecting local areas, it suffers from poor prediction results and the tendency for joint point misalignment in the prediction results, which may affect the consistency of the human joint skeleton.

[0036] In view of this, the present invention provides a keypoint detection method, which can be implemented based on a keypoint detection model, including a convolutional neural network and a fully connected neural network configured in parallel. See also Figure 1 As shown, Figure 1 This is a schematic flowchart of a key point detection method provided in an embodiment of the present invention, which may include:

[0037] Step S11: Obtain the first prediction result of key points in the image to be detected based on the convolutional neural network;

[0038] Step S12: Determine the second prediction result of key points in the image to be detected based on a fully connected neural network;

[0039] Step S13: Determine the key point detection results of the image to be detected based on the first prediction result of key points, the second prediction result of key points, and the disorder condition distance threshold.

[0040] In some embodiments, step S11 may specifically be:

[0041] Key point heatmaps of images to be detected are obtained using convolutional neural networks.

[0042] The first prediction result of key points in the image to be detected is determined based on the key point heatmap.

[0043] The key point heatmap includes the probability that each pixel region in the image to be detected is a corresponding key point, and the position corresponding to the maximum probability is the position of the corresponding key point.

[0044] In some embodiments, the Argmax function can be used to determine the first prediction result of the key points corresponding to the key points in the key point heatmap. When the dimensions of the key point heatmap and the image to be detected are different, the first prediction result of the key points can also be transformed to the coordinate system corresponding to the image to be detected to obtain the final first prediction result of the key points.

[0045] In some embodiments, step S12 may specifically involve obtaining the region of interest in the image to be detected, using a fully connected neural network model to determine the initial second prediction result of key points in the region of interest in the image to be detected, and then transforming the initial second prediction result of key points to the coordinate system corresponding to the image to be detected to obtain the final second prediction result of key points.

[0046] The fully connected neural network model can focus on the overall features of the region of interest, thus avoiding the problem of incorrect key point prediction due to similar local features.

[0047] In some embodiments, step S13 may specifically be:

[0048] Calculate the absolute value of the difference between the first prediction result and the second prediction result of the key point;

[0049] The absolute value of the difference is compared with the distance threshold of the disorder condition, and the key point detection result of the image to be detected is determined based on the comparison result.

[0050] The distance threshold for disorder conditions can be set as needed.

[0051] In some embodiments, when the absolute value of the difference is greater than the disordered conditional distance threshold, a second prediction result is output as the key point detection result; when the absolute value of the difference is less than or equal to the disordered conditional distance threshold, a first prediction result is output as the key point detection result.

[0052] Here, the first prediction result of the keypoint can be the first predicted coordinates corresponding to the keypoint, and the second prediction result of the keypoint can be the second predicted coordinates corresponding to the keypoint. The absolute value of the difference between the first prediction result and the second prediction result of the keypoint can be expressed as |P hmp -P fc |, where P hmp P represents the first predicted coordinate. fc Represents the second predicted coordinate; the disordered condition distance threshold can be expressed as Th, when |P hmp -P fc When |>Th, the keypoint detection result is determined to be P. fc When |P hmp -P fc When |≤Th, the keypoint detection result is determined as P. hmp .

[0053] By comparing the absolute value of the difference between the first and second predicted keypoint results with a threshold for error condition, if the difference is greater than a certain threshold, it can be determined that the first predicted keypoint result obtained from the keypoint heatmap has low prediction accuracy and may contain keypoint prediction errors. Therefore, the second predicted keypoint result obtained based on a fully connected neural network is used as the keypoint detection result. This method can improve the overall accuracy and effectiveness of the keypoint detection method and avoid the problem of keypoint prediction errors.

[0054] In other embodiments, the disordered condition distance threshold may include a first disordered condition distance threshold and a second disordered condition distance threshold, wherein the first disordered condition distance threshold is less than the second disordered condition distance threshold. Step S13 may also involve calculating the ratio of the first prediction result of the key point to the second prediction result of the key point, comparing the ratio with the first disordered condition distance threshold and the second disordered condition distance threshold, and outputting the second prediction result as the key point detection result when the ratio is greater than the second disordered condition distance threshold or less than the first disordered condition distance threshold; and outputting the first prediction result as the key point detection result when the ratio is greater than or equal to the first disordered condition distance threshold and less than or equal to the second disordered condition distance threshold.

[0055] The above describes a keypoint detection method provided by an embodiment of the present invention. This method is based on a keypoint detection model, which includes a convolutional neural network and a fully connected neural network configured in parallel. The method obtains a first prediction result of keypoints in the image to be detected based on the convolutional neural network; determines a second prediction result of keypoints in the image to be detected based on the fully connected neural network; and determines the final keypoint detection result of the image to be detected based on the first prediction result, the second prediction result, and a distance threshold for error conditions. The fully connected neural network can focus more on overall features. This method, by combining convolutional neural networks and fully connected neural networks, can effectively avoid the problem of keypoint prediction errors.

[0056] In some embodiments, the first prediction result of the key points can be corrected to further improve the accuracy of the model, as described below.

[0057] See Figure 2 As shown, Figure 2 This is a schematic flowchart of a key point detection method provided in another embodiment of the present invention, which may include:

[0058] Step S21: Acquire the image to be detected and preprocess it;

[0059] Step S22: Obtain the first prediction result of key points in the preprocessed image to be detected based on the convolutional neural network; obtain the key point bias amount of the preprocessed image to be detected based on the key point bias network; and correct the first prediction result of key points based on the key point bias amount.

[0060] Step S23: Determine the second prediction result of key points in the preprocessed image to be detected based on a fully connected neural network;

[0061] Step S24: Determine the key point detection results of the image to be detected based on the corrected first key point prediction results, the second key point prediction results, and the disordered conditional distance threshold.

[0062] Step S24 can be implemented in the same way as step S13 above. For the sake of brevity, it will not be described again here. For details, please refer to the description above.

[0063] In some embodiments, step S21 may specifically involve acquiring the image to be detected using an image acquisition device or by directly reading pre-stored image data, and performing at least one preprocessing operation on the image to be detected, including image scaling, image data type conversion, and image normalization.

[0064] Specifically, the detection speed is improved by scaling the image to be detected to meet the input size requirements of the keypoint detection model, converting the image data type to meet the data type requirements of the keypoint detection model, and normalizing the image.

[0065] In some embodiments, a region of interest can also be identified from the image to be detected, and the region of interest can be used as the input image for the keypoint detection model.

[0066] It should be noted that the keypoint detection model may also include a backbone network, which is used for feature extraction from the image to be detected. Convolutional neural networks and fully connected neural networks can share the backbone network; the keypoint bias network and the convolutional neural network are configured in parallel. The method may also include inputting the preprocessed image to be detected into the backbone network to obtain a feature image, and then inputting the feature image into the convolutional neural network, the keypoint bias network, and the fully connected neural network respectively.

[0067] As an example, the backbone network may include ResNet (Residual Network), VGG (Visual Geometry Group), HRNet (High-Resolution Net), or MobileNet.

[0068] In some embodiments, step S22 may specifically involve obtaining a keypoint heatmap of the feature image based on a convolutional neural network, using the Argmax function to determine the initial first prediction result of the keypoints corresponding to the keypoints in the keypoint heatmap, obtaining the initial keypoint bias of the feature image based on a keypoint bias network, wherein the feature image and the preprocessed image to be detected have different sizes, and the initial first prediction result of the keypoints may be transformed into the coordinate system corresponding to the preprocessed image to be detected to obtain the first prediction result of the keypoints of the preprocessed image to be detected; and the initial keypoint bias may be transformed into the coordinate system corresponding to the preprocessed image to be detected to obtain the keypoint bias of the preprocessed image to be detected; furthermore, by calculating the sum of the keypoint bias and the first prediction result of the keypoints in the same coordinate system, a corrected first prediction result of the keypoints is obtained, thereby improving the accuracy of the first prediction result of the keypoints.

[0069] In some embodiments, step S23 may specifically involve obtaining the initial second prediction result of key points of the feature image based on a fully connected neural network; transforming the second prediction result of key points into the coordinate system corresponding to the preprocessed image to be detected to obtain the second prediction result of key points of the preprocessed image to be detected.

[0070] The above is another key point detection method provided by the embodiments of the present invention, which can achieve the same results as... Figure 1 In addition to the same beneficial effects as the corresponding embodiments, by setting a key point bias network and correcting the first prediction result of the key points based on the key point bias network, the accuracy of the first prediction result of the key points is effectively improved, thereby improving the overall detection accuracy of the key point detection model.

[0071] In another aspect, the present invention also provides a method for training a keypoint detection model, see [link to relevant documentation]. Figure 3 The above, Figure 3 This is a schematic diagram of the key point detection model training method provided in an embodiment of the present invention, which may include:

[0072] Step S31: Obtain the image training sample set, which includes multiple image training samples, each of which is labeled with key point information;

[0073] Step S32: Input the image training samples from the image training sample set into the initial keypoint detection model to obtain the keypoint detection results of the image training samples;

[0074] Step S33: Calculate the loss value based on the keypoint detection results and keypoint information, and update the parameters of the initial keypoint detection model based on the loss value to obtain the trained keypoint detection model.

[0075] The image training samples can be single-target human images. Key points of the target human in the single-target human image can be labeled in advance. In some embodiments, the labeling information can include the coordinates of the key points and the visibility category corresponding to the key points. The visibility category can include visible and invisible.

[0076] In some embodiments, the image training samples in the image training sample set may be preprocessed before being input into the initial keypoint detection model. This preprocessing may include at least one of image scaling, image data type conversion, and image normalization.

[0077] Specifically, image training samples are scaled to meet the input size requirements of the keypoint detection model, image data type is converted to meet the data type requirements of the keypoint detection model, and image normalization is performed to improve training speed.

[0078] In some embodiments, the initial keypoint detection model may include a backbone network, a parallel convolutional neural network, and a fully connected neural network. The backbone network is used to extract features from the preprocessed image training samples. Step S32 may specifically involve obtaining the third prediction result of keypoints for the preprocessed image training samples based on the convolutional neural network, determining the fourth prediction result of keypoints for the preprocessed image training samples based on the fully connected neural network, and obtaining the keypoint detection result of the image training samples based on the comparison between the absolute value of the difference between the third and fourth prediction results of keypoints and the disordered condition distance threshold.

[0079] Specifically, when the absolute value of the difference is greater than the distance threshold of the disorder condition, the fourth prediction result is output as the key point detection result; when the absolute value of the difference is less than or equal to the distance threshold of the disorder condition, the third prediction result is output as the key point detection result.

[0080] In other embodiments, the initial keypoint detection model may further include a keypoint bias network, with the keypoint bias network and convolutional neural network configured in parallel. Step S32 may further specifically involve obtaining the third prediction result of keypoints for the preprocessed image training samples based on the convolutional neural network, obtaining the keypoint bias amount of the image training samples based on the keypoint bias network, correcting the third prediction result of keypoints for the corresponding image training samples using the keypoint bias amount, determining the fourth prediction result of keypoints for the preprocessed image training samples based on a fully connected neural network, and finally, obtaining the keypoint detection result of the image training samples based on the comparison between the absolute value of the difference between the corrected third prediction result and the fourth prediction result of keypoints and the disordered condition distance threshold. This method improves the accuracy of the third prediction result of keypoints, thereby also improving the overall detection accuracy of the keypoint detection model.

[0081] In some embodiments, step S33 may specifically involve calculating a loss value based on the third or fourth prediction result of the key point and the key point information of the corresponding image training samples, and updating the parameters of the initial key point detection model based on the loss value to obtain a trained key point detection model.

[0082] In other embodiments, in order to increase the learning probability of difficult samples, at least one of the following processing methods, namely flipping, blurring, occlusion and truncation, can be used to augment the image training samples after step S31 and before step S32.

[0083] In some embodiments, Gaussian blur or motion blur can be used to augment the image training samples; random erasure or occlusion can be used to augment the image training samples; random cropping or truncation can be used to augment the image training samples.

[0084] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0085] Another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the key point detection method in any of the above embodiments. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0086] In another aspect, the present invention also provides an electronic device, see [link to relevant documentation]. Figure 4 As shown, Figure 4 This is a schematic diagram of the electronic device structure provided by the present invention, which may include a memory 41 and a processor 42. The memory 41 stores a computer program, and when the computer program is executed by the processor 42, it implements the key point detection method in any of the above embodiments.

[0087] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A key point detection method, characterized in that, The keypoint detection method is implemented based on a keypoint detection model, which includes a parallel convolutional neural network and a fully connected neural network. The method includes: The first prediction result of key points of the image to be detected is obtained based on the convolutional neural network; the first prediction result of key points of the image to be detected is obtained based on the convolutional neural network, which includes: obtaining a key point heatmap of the image to be detected based on the convolutional neural network; and determining the first prediction result of key points of the image to be detected based on the key point heatmap. The second prediction result of key points of the image to be detected is determined based on the fully connected neural network; the determination of the second prediction result of key points of the image to be detected based on the fully connected neural network includes: obtaining the region of interest of the image to be detected, using the fully connected neural network model to determine the initial second prediction result of key points of the region of interest in the image to be detected, transforming the initial second prediction result of key points to the coordinate system corresponding to the image to be detected, and obtaining the final second prediction result of key points. The keypoint detection result of the image to be detected is determined based on the first prediction result of the keypoint, the second prediction result of the keypoint, and the disordered conditional distance threshold. This determination includes: calculating the absolute value of the difference between the first prediction result of the keypoint and the second prediction result of the keypoint; comparing the absolute value of the difference with the disordered conditional distance threshold, and determining the keypoint detection result of the image to be detected based on the comparison result. Further, determining the keypoint detection result of the image to be detected based on the comparison result includes: when the absolute value of the difference is greater than the disordered conditional distance threshold, outputting the second prediction result of the keypoint as the keypoint detection result; when the absolute value of the difference is less than or equal to the disordered conditional distance threshold, outputting the first prediction result of the keypoint as the keypoint detection result.

2. The method according to claim 1, characterized in that, The keypoint detection model further includes a keypoint bias network, and the method further includes: The keypoint offset amount of the image to be detected is obtained based on the keypoint offset network. The first prediction result of the key point is corrected based on the key point bias.

3. The method according to claim 1, characterized in that, The method further includes at least one preprocessing operation among image scaling, image data type conversion, and image normalization of the image to be detected.

4. The method according to claim 1, characterized in that, The keypoint detection model is trained using the following steps: Obtain an image training sample set, which includes multiple image training samples, each of which is labeled with key point information; Input the image training samples in the image training sample set into the initial key point detection model to obtain the key point detection results of the image training samples; The loss value is calculated based on the keypoint detection results and the keypoint information, and the parameters of the initial keypoint detection model are updated based on the loss value to obtain the trained keypoint detection model.

5. The method according to claim 4, characterized in that, After obtaining the image training sample set, the method further includes: performing data augmentation on the image training samples using at least one of the following processing methods: flipping, blurring, occlusion, and truncation.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the key point detection method according to any one of claims 1 to 5.

7. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, implements the key point detection method according to any one of claims 1 to 5.

Images