Key point detection method, electronic equipment and storage medium

Abstract

The invention discloses a key point detection method, electronic equipment and a storage medium. The key point detection method comprises the steps of obtaining image features, wherein the image features are obtained by performing feature extraction on a plurality of detection areas in a to-be-detected image collected by a target object; the image features are classified, an image classification result of the to-be-detected image is obtained, and the image classification result comprises the area where the target object is located in the multiple detection areas or does not comprise the area where the target object is located in the multiple detection areas; and in response to the fact that the image classification result is that the multiple detection areas include the area where the target object is located, key point information obtained by key point detection processing of the image features is obtained, and the key point information includes position information of key points in the multiple detection areas. According to the scheme, the accuracy of the obtained key point information can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a key point detection method, an electronic device and a storage medium. Background Art

[0002] In recent years, the key points of the target object extracted from the image through key point detection can be used to achieve visual perception or other tasks. For example, somatosensory game interaction, target tracking, motion pattern recognition, etc. can be achieved through the key points of the image. However, key point detection strictly depends on the detection frame obtained by target detection. For example, it is necessary to perform motion pattern recognition on the target object belonging to type A. Although the upstream target detection operator outputs the detection frame, the area where the detection frame is located in the image corresponds to the target object of type B, which leads to the key point information output by the key point detection, and then leads to low accuracy of motion pattern recognition for the image. Therefore, for key point detection, if the upstream target detection operator makes a false detection, it will cause the key point detection to output incorrect key point information of the target object, resulting in abnormalities in subsequent visual perception or other tasks.

[0003] In view of the existing technical defects, how to provide a more accurate solution for key point detection is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0004] The present application at least provides a key point detection method, an electronic device and a storage medium.

[0005] The present application provides a key point detection method, including: acquiring image features, the image features are obtained by extracting features from a number of detection areas in an image to be detected that is collected from a target object; performing classification processing on the image features to obtain an image classification result of the image to be detected, the image classification result including areas in the number of detection areas where the target object is located or areas in the number of detection areas where the target object is not located; in response to the image classification result being an area in the number of detection areas where the target object is located, acquiring key point information obtained by key point detection processing of the image features, the key point information including position information of key points in the number of detection areas.

[0006] The present application provides a key point detection device, including: a first acquisition module, a classification module and a second acquisition module; the first acquisition module is used to acquire image features, which are obtained by extracting features from several detection areas in an image to be detected collected from a target object; the classification module is used to classify the image features to obtain an image classification result of the image to be detected, and the image classification result includes several detection areas where the target object is located or several detection areas where the target object is not located; the second acquisition module is used to obtain key point information obtained by key point detection processing of the image features in response to the image classification result being several detection areas where the target object is located, and the key point information includes position information of key points in several detection areas.

[0007] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned key point detection method.

[0008] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the above-mentioned key point detection method is implemented.

[0009] The above scheme obtains image features obtained by extracting features from several detection areas in the image to be detected collected from the target object, and then classifies the image features to obtain the image classification result of the image to be detected. In response to the image classification result being that the image classification result is an area where the target object is located in the several detection areas, key point information obtained by key point detection processing of the image features is obtained, which can ensure that the image features are features corresponding to the target object, avoid the presence of information that does not belong to the target object in the output key point information, thereby improving the accuracy of the output key point information.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0012] Figure 1 This is a schematic diagram of the process of an embodiment of the key point detection method of the present application. Figure 1 ;

[0013] Figure 2 This is a schematic diagram of the process of an embodiment of the key point detection method of the present application. Figure 2 ;

[0014] Figure 3This is a schematic diagram of the process of an embodiment of the key point detection method of the present application. Figure 3 ;

[0015] Figure 4 is a schematic diagram of key points in an embodiment of a key point detection method of the present application;

[0016] Figure 5 It is a structural schematic diagram of an embodiment of a key point detection device of the present application;

[0017] Figure 6 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0018] Figure 7 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0019] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0020] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0021] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0022] The present application provides some key point detection methods and key point detection devices. The application scenarios of the key point detection method include but are not limited to key point detection in posture assessment and key point detection in motion pattern analysis. The executor of the key point detection method may be a key point detection device. For example, the key point detection device may be arranged in a terminal device or a server or other processing device, wherein the terminal device may be a device for key point detection, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, etc. In some possible implementations, the key point detection method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0023] See also Figure 1 , Figure 1 This is a schematic diagram of the process of an embodiment of the key point detection method of the present application. Figure 1 Specifically, the key point detection method may include the following steps:

[0024] Step S11: Acquire image features.

[0025] Image features are obtained by extracting features from several detection areas in the image to be detected collected from the target object.

[0026] The image to be detected may be any image that needs to be detected for key points. The image to be detected includes several detection areas. The several detection areas may be one detection area or multiple detection areas. The several detection areas are associated with several target frames in the image to be detected. Among them, each detection area may be an area where each target frame in the image to be detected is located, or an image area obtained by fine-tuning the area where each target frame in the image to be detected is located. The several target frames may be at least one detection frame. The several target frames may be obtained by performing target detection on the image to be detected. The target object may be any object that needs to be detected for key points, including but not limited to animals and vehicles. In practical applications, the object to which each target frame belongs may be the target object or other objects. Other objects may be other objects except the target object. Exemplarily, when the target object may be an animal and the specific target object may be a cat, the other objects may be a dog or a vehicle.

[0027] The image feature may be a feature obtained by extracting features from the regional images belonging to several detection regions in the image to be detected. The image feature includes image features corresponding to several regional images. Several regional images are respectively input into the feature extraction operator for feature extraction to obtain image features corresponding to each regional image. The feature extraction operator may be a preset feature extraction network, and different feature extraction networks or modules may be set according to feature extraction requirements. In some application scenarios, the preset feature extraction network may be a MobileNet feature extraction network, an Inception feature extraction network, and the like. In other application scenarios, the feature extraction network may adopt the resnet series, such as the resnet34 feature extraction network or the resnet52 feature extraction network, and the transformer structure may also be used to extract features, such as the swin transformer, the vit feature extraction network, and the like. The setting method of the preset feature extraction network is not limited here.

[0028] Step S12: classify the image features to obtain an image classification result of the image to be detected.

[0029] The image classification result includes an area where the target object is located in the plurality of detection areas or an area where the target object is not located in the plurality of detection areas.

[0030] The region including the target object in the several detection regions is used to indicate that at least one detection region is the region where the target object is located. The region not including the target object in the several detection regions is used to indicate that any detection region is not the region where the target object is located. The image features include image features corresponding to at least one regional image. The above step S12 may include: classifying the image features corresponding to at least one regional image to obtain the classification results of the image features corresponding to each regional image, and the classification results of the image features corresponding to each regional image include that the detection region where the regional image is located is the region where the target object is located, or the detection region where the regional image is located is not the region where the target object is located. Based on the classification results of the image features corresponding to each regional image, the image classification results of the image to be detected are determined. Exemplarily, the image features corresponding to at least one regional image are input into a preset classification network for classification processing to obtain the classification features output by the preset classification network. The classification features are input into the fully connected layer for mapping processing to obtain the classification results of the image features corresponding to each regional image.

[0031] In some application scenarios, when several detection areas are one detection area, the classification results of the image features corresponding to each area image are equivalent to the classification results of the image to be detected. In other application scenarios, when several detection areas are multiple detection areas, the classification results of the image features corresponding to each area image are determined to be the classification results of the image to be detected. Specifically, when the classification result of the image features corresponding to at least one area image is that the detection area where the area image is located is the area where the target object is located, the image classification result of the image to be detected is determined to be the area where the target object is located in the several detection areas. Specifically, when the classification result of the image features corresponding to any one area image is that the detection area where the area image is located is not the area where the target object is located, the image classification result of the image to be detected is determined to be the area where the target object is not located in the several detection areas.

[0032] Step S13: In response to the image classification result being that the area where the target object is located is included in the plurality of detection areas, key point information obtained by key point detection processing of the image features is obtained.

[0033] The key point information includes the position information of the key points in the plurality of detection areas. Specifically, the key point information includes the position information of the key points in the target detection area. The target detection area is at least part of the plurality of detection areas. Specifically, when the plurality of detection areas are one detection area, the target detection area is the detection area. When the plurality of detection areas are multiple detection areas, the number of the target detection areas is at least one, and each target detection area is an area in the plurality of detection areas where the target object is located.

[0034] In the case where the image classification result is that the target object is located in a plurality of detection areas, the key point information obtained by the key point detection processing of the image features belonging to the target detection area is obtained. The key detection processing of the image features may be inputting the image features belonging to the target detection area into the key point detection module for key point detection processing. A preset key point detection algorithm may be set on the key point detection module, wherein the preset key point detection algorithm may include but is not limited to an algorithm based on coordinate value regression, an algorithm based on instance segmentation, and an algorithm based on a heat map. Specifically, the algorithm based on coordinate value regression may be to directly regress the coordinate values ​​of the key points in the image features using a fully connected layer. Specifically, the algorithm based on instance segmentation may be that each instance object corresponds to its own key point, avoiding the pairing problem during multi-target key point detection. Detect the object in the image to be detected, and generate a bounding box for each instance object, which is the above-mentioned target box. Segment the image features corresponding to the target detection area and extract the key points therefrom. Specifically, the algorithm based on the heat map can be to represent the coordinates of each key point in each target detection area with a probability map, and give a probability to each pixel position in the image corresponding to each target detection area, indicating the probability that the pixel point at the pixel position belongs to the key point of the corresponding category. Usually, the probability of the pixel point closer to the key point position is closer to 1, and the probability of the pixel point farther from the key point is closer to 0.

[0035] When the image classification result is that the target object is located in a plurality of detection areas, the key point information obtained by key point detection processing of the acquired image features can be at least one key point information corresponding to the target object. In this case, the present application can ensure that the output key point information only belongs to the key point information of the target object.

[0036] After acquiring the above-mentioned key point information, the key point information is sent to the downstream task for identification processing to obtain the identification information of the downstream task. Among them, the downstream tasks include but are not limited to the posture assessment task of the target object, the motion pattern analysis task of the target object, etc. The identification information of the downstream task may include the posture assessment result, and may also include the motion pattern result. By sending only the key point information belonging to the target object to the downstream task, it can be ensured that the key point information sent to the downstream task is correct. Therefore, the present application can filter the key point information that does not belong to the target object by classifying the image features, thereby avoiding the influence of the erroneous key point information on the accuracy corresponding to the identification information of the downstream task.

[0037] In some application scenarios, the step of obtaining key point information obtained by key point detection processing of image features may be: regardless of whether the image features are image features corresponding to the above-mentioned target detection area, the key point information obtained by key point detection processing of all image features is obtained. Then, from the key point information obtained by key point detection processing of all image features, the key point information obtained by key point detection processing of image features belonging to the target detection area is selected as the final output. In other application scenarios, the step of obtaining key point information obtained by key point detection processing of image features may be: only the key point information obtained by key point detection processing of image features corresponding to the above-mentioned target detection area is used as the final output. That is, only the image features corresponding to the above-mentioned target detection area are subjected to key point detection processing. It can be considered that if the image features corresponding to the target detection area are target image features, not performing key point detection processing on other image features except the target image features among all image features can reduce the amount of calculation.

[0038] The above scheme obtains image features obtained by extracting features from several detection areas in the image to be detected collected from the target object, and then classifies the image features to obtain the image classification result of the image to be detected. In response to the image classification result being that the image classification result is an area where the target object is located in the several detection areas, key point information obtained by key point detection processing of the image features is obtained, which can ensure that the image features are features corresponding to the target object, avoid the presence of information that does not belong to the target object in the output key point information, thereby improving the accuracy of the output key point information.

[0039] In some embodiments, the above step S11 may include the following steps: first, obtain an image to be detected. Then, perform target detection on the image to be detected to obtain a target detection result, which includes at least one detection frame. Subsequently, the image to be detected is cropped according to the position information of each detection frame to obtain a regional image corresponding to each detection frame. Then, for each regional image, feature extraction is performed on the regional image to obtain image features corresponding to the regional image.

[0040] The above step S11 may be to perform image preprocessing on the image to be detected to obtain image features corresponding to the image to be detected. The image features corresponding to the image to be detected include image features corresponding to several detection areas. Each detection area corresponds to each regional image one-to-one, and several detection areas correspond to at least one regional image. That is, the image features corresponding to the image to be detected include image features corresponding to at least one regional image. It can be understood that, when the image classification result is that several detection areas include the area where the target object is located, at least some of the image features corresponding to the several detection areas are image features corresponding to the target detection area.

[0041] Specifically, each regional image is obtained by cropping the image to be detected using the position information of at least one detection frame. For each detection frame, the boundary range of the detection frame is determined according to the position information of the detection frame. In some application scenarios, the regional image corresponding to the detection frame is obtained by cropping along the boundary range from the image to be detected. In other application scenarios, the boundary range is fine-tuned based on the regional image range requirement to obtain a new boundary range. The image cropped along the new boundary range from the image to be detected is used as the regional image corresponding to the detection frame. Specifically, the regional image range requirement may be that each regional image is an image of a preset size. The preset size can be dynamically set according to demand. Exemplarily, the preset size of each regional image may be an image of 256×256. The step of fine-tuning the boundary range based on the regional image range requirement to obtain a new boundary range may be: if the area of ​​the detection frame is not equal to the area corresponding to the preset size, the center point of the detection frame is determined as the center point of the preset size frame. Based on the center point of the detection frame, the size of the detection frame is adjusted to the size of the preset size frame to obtain the adjusted detection frame. The boundary range of the adjusted detection frame is determined as the new boundary range.

[0042] It can be considered that by cropping the image to be detected and only performing feature extraction on the cropped regional image, the interference of image feature information irrelevant to the target object and / or the current regional image in the image to be detected can be reduced, thereby improving the accuracy of the image features corresponding to each regional image in the feature extraction result.

[0043] In some embodiments, the key point detection method may further include the following steps: first, in response to the image classification result indicating that the target object is not included in the plurality of detection areas, an adjustment instruction is received, the adjustment instruction is used to instruct to adjust the position information of each detection frame. Subsequently, the position information of each detection frame is adjusted using the adjustment instruction to obtain the adjusted position information corresponding to each detection frame;

[0044] On the configuration interface, several detection frames are adjusted according to the adjustment instructions to obtain the adjusted position information corresponding to each detection frame. The adjustment instruction may be a preset instruction or may be obtained by custom editing on the configuration interface. In some application scenarios, the adjustment instruction may be a preset instruction. The preset instruction may be an instruction related to the preset size frame, and the size of the preset size frame may be dynamically set. The above-mentioned step of adjusting the position information of each detection frame using the adjustment instruction to obtain the adjusted position information corresponding to each detection frame may be: for each detection frame, the center point of the detection frame is determined as the center point of the preset size frame. Based on the center point of the detection frame, the size of the detection frame is adjusted to the size of the preset size frame to obtain the adjusted detection frame, and the boundary range of the adjusted detection frame is determined as the new boundary range. The new boundary range corresponding to each detection frame is used as the adjusted position information corresponding to each detection frame. In other application scenarios, the adjustment instruction may be to drag, stretch, and edit the custom size of any detection frame on the configuration interface to obtain a new size frame. The above step of using the adjustment instruction to adjust the position information of each detection frame to obtain the adjusted position information corresponding to each detection frame may be: for each detection frame, the center point of the detection frame is determined as the center point of the new size frame. Based on the center point of the detection frame, the size of the detection frame is adjusted to the size of the new size frame to obtain the adjusted detection frame, and the boundary range of the adjusted detection frame is determined as the new boundary range. The new boundary range corresponding to each detection frame is used as the adjusted position information corresponding to each detection frame.

[0045] In some embodiments, the above-mentioned step of cropping the image to be detected according to the position information of each detection frame to obtain the area image corresponding to each detection frame may include the following steps: cropping the image to be detected according to the adjusted position information of each detection frame to obtain the area image corresponding to each detection frame.

[0046] The adjusted position information of each detection frame includes the boundary range corresponding to each adjusted detection frame, that is, the new boundary range. For each detection frame, an image cut from the image to be detected along the new boundary range is used as the region image corresponding to the detection frame.

[0047] It can be considered that the dynamic adjustment of several detection frames included in the image to be detected is achieved through adjustment instructions, so that the adjusted position information corresponding to each detection frame is more in line with the cropping requirements, thereby improving the accuracy of the regional image corresponding to each detection frame obtained by cropping, and thus making the image features more accurate.

[0048] See also Figure 2 , Figure 2 This is a schematic diagram of the process of an embodiment of the key point detection method of the present application. Figure 2 .

[0049] In some embodiments, after the above step S12, the key point detection method may further include the following steps: Step S21: Input the image features into the key point detection module for key point detection to obtain an initial heat map corresponding to at least one key point. Step S22: For each initial heat map, determine the image point with the highest confidence in the initial heat map as the target point corresponding to the initial heat map. Step S23: Use the target points in each initial heat map as key points, and use the position information of each target point in the initial heat map as the position information of the corresponding key point in a number of detection areas.

[0050] The above-mentioned step S21 can be inputting the image features corresponding to several detection areas into the key point detection module for key point detection to obtain an initial heat map corresponding to at least one key point, or can be inputting the image features corresponding to several target detection areas into the key point detection module for key point detection to obtain an initial heat map corresponding to at least one key point. The present application takes the inputting of the image features corresponding to any target detection area into the key point detection module, and taking the image features corresponding to any target detection area as the target image features as an example. It can be understood that the present application does not limit the type of image features that need to be input into the key point detection module, and each image feature input into the key point detection module is the same processing operation, and an initial heat map corresponding to at least one key point corresponding to each image feature will be generated.

[0051] Specifically, the above step S21 may be that for each target image feature, the target image feature is input into the key point detection module for key point processing to obtain at least one initial heat map corresponding to the target image feature. Exemplarily, for each target image feature, the target image feature is upsampled to obtain an initial heat map corresponding to at least one key point. Each target detection area may include at least one initial heat map corresponding to a key point. In some application scenarios, the upsampling operation may be bilinear interpolation of the target image feature to obtain a processed image, and determine the initial heat map corresponding to at least one key point from the processed image. In other application scenarios, the target image feature is transposed convolution processed to obtain a processed image, and the initial heat map corresponding to at least one key point is determined from the processed image. Specifically, by performing transposed convolution processing on the target image feature, the resolution of the target image feature can be increased. For example, if the target image feature size is 8×6, performing a transposed convolution will change the feature size to 16×12.

[0052] The initial heat map corresponding to at least one key point may be an initial heat map corresponding to one key point or an initial heat map corresponding to multiple key points. The initial heat map corresponding to each key point is used to represent the probability map of the key point coordinates. Each image point in the initial heat map can represent the probability that the image point at the pixel position belongs to the corresponding category key point. Each image point corresponds to each pixel point in the initial heat map.

[0053] The image points with the largest confidence in each initial heat map are respectively used as the target points corresponding to each initial heat map. The target points in each initial heat map are respectively used as the key points corresponding to each initial heat map. The position information of each target point in the initial heat map may include the position information of each target point in each initial heat map, may include the position information of each target point in the detection area, and may also include the position information of each target point in the image to be detected. The position information of each target point in the initial heat map is directly used as the position information of the corresponding key point in several detection areas.

[0054] In some embodiments, the step of using the target points in each initial heat map as key points may include the following steps: first, filtering out the target points whose confidences meet a preset threshold range. Then, using the retained target points as key points.

[0055] The confidence of each target point may be different. Exemplarily, the initial heat map corresponding to at least one key point obtained in the above step S21 may be 3. The number of each target point is 3. Compared with other image points in each initial heat map, the confidence of each target point is the image point with the largest confidence in each initial heat map. The initial heat map includes a first initial heat map, a second initial heat map, and a third initial heat map. The preset threshold range is used to represent the confidence threshold range of the target point. The preset threshold range may be a preset confidence threshold range. For example, the preset threshold range may be a confidence value of 0 to 0.4. If the confidence of any target point is within the confidence threshold range, the target point is filtered out. If the confidence of any target point is not within the confidence threshold range, the target point is retained. Each retained target point is used as a key point. For example, the confidence of the target point corresponding to the first initial heat map is 0.8, the confidence of the target point corresponding to the second initial heat map is 0.35, and the confidence of the target point corresponding to the third initial heat map is 0.6. The target points corresponding to the first initial heat map and the target points corresponding to the third initial heat map are the retained target points. Then the target points corresponding to the first initial heat map and the target points corresponding to the third initial heat map are respectively used as the key points of the final output.

[0056] It can be considered that by filtering or retaining target points within a preset threshold range, key points with low confidence or inaccurate occlusion prediction can be filtered out, so that the key points of the final output are key points that meet the expected confidence values, thereby improving the accuracy of the key points of the final output.

[0057] See also Figure 3 , Figure 3 This is a schematic diagram of the process of an embodiment of the key point detection method of the present application. Figure 3 .

[0058] In some embodiments, the key point detection method may further include a training step for the key point detection module, the training step including: Step S31: Obtain sample image features and at least one label heat map about the sample image features. The label heat map is obtained from the sample image features, and the confidence of the occluded key point annotations in each label heat map is less than the confidence of the unoccluded key point annotations. Step S32: Input the sample image features into the key point detection module for key point detection to obtain at least one predicted heat map. Step S33: Based on the differences between each label heat map and each predicted heat map, adjust the parameters in the key point detection module to obtain a trained key point detection module.

[0059] Acquire several sample images. For each sample image, perform image preprocessing on the sample image to obtain sample image features corresponding to the sample image. It is understandable that each sample image may include several sample image features. Each sample image is processed by target detection to obtain several sample detection frames. Each sample image feature is an image feature of the sample image area corresponding to any sample detection frame after the sample image is processed by target detection. For each sample image feature, perform the following steps: perform labeling on the sample image feature to obtain at least one label heat map. Specifically, perform key point detection on each sample image feature to obtain a predicted heat map corresponding to at least one predicted key point. Each predicted heat map corresponds to a predicted key point. Label each predicted heat map to obtain at least one label heat map. The sample image point with the highest confidence in each label heat map is the sample target point.

[0060] Each label heat map corresponds to a sample target point. The labeling process includes: confidence labeling of the sample target point and / or attribute labeling of the sample target point. The attribute labeling of the sample target point includes labeling the sample target point as an occluded key point and labeling the sample target point as an unoccluded key point. The confidence labeling of the sample target point can be a labeling of the confidence of each sample target point. The confidence of the occluded key point in each label heat map is less than the confidence of the unoccluded key point.

[0061] The difference between each label heat map and each predicted heat map in the above step S33 may be a position difference and / or a confidence difference. In some application scenarios, the position difference in the difference between each label heat map and each predicted heat map in the above step S33 may be, for each label heat map and the predicted heat map corresponding to the label heat map, the difference between the position of the sample target point in the label heat map and the sample detection box where the predicted key point is located in the predicted heat map. Directly based on the position difference, the position loss is determined, and the position loss is used as the first loss.

[0062] In other application scenarios, the confidence difference in the difference between each label heat map and each prediction heat map in the above step S33 can be the difference between the sample confidence corresponding to the sample target point in the label heat map and the prediction confidence corresponding to the prediction key point in the prediction heat map. Based directly on the confidence difference, the confidence loss is determined, and the confidence loss is used as the first loss. In other application scenarios, the difference between each label heat map and each prediction heat map in the above step S33 can be the position difference and the confidence difference. Specifically, the position loss and the confidence loss are weighted and summed, and the weighted sum value is determined as the first loss. Using the first loss, the parameters in the key point detection module are adjusted, and the key point detection module is trained with the first loss reduced as the goal, until the first loss output by the trained key point detection module meets the requirements. In other application scenarios, the key point detection device includes a feature extraction module, a classification module and a key point detection module, and the feature extraction module is used to perform the above step S11. The classification module is used to perform the above step S12, and the key point detection module is used to perform the above step S13. The first loss is used to adjust the parameters of each module in the key point detection device, and the key point detection device is trained with the goal of reducing the first loss until the first loss output by the trained key point detection device meets the requirements.

[0063] In some embodiments, the above step S33 may include the following steps: First, based on the difference between each label heat map and each predicted heat map, a first loss is determined. Then, based on the confidence difference between each sample target point and each predicted key point in the predicted key point set, a second loss is determined, each sample target point is obtained by annotating each label heat map, and the confidence of each predicted key point is greater than a preset confidence threshold. Subsequently, the first loss and the second loss are used to adjust the parameters in the key point detection module to obtain a trained key point detection module.

[0064] The preset confidence threshold may be a preset value, or may be the maximum confidence value in the preset threshold range. For example, the preset confidence threshold may be 0.4.

[0065] For each sample image feature, the following steps are performed: the sample image feature is labeled to obtain at least one label heat map. Specifically, key point detection is performed on each sample image feature to obtain a prediction heat map corresponding to at least one predicted key point. Each predicted heat map corresponds to a predicted key point. Each predicted key point is filtered according to a preset confidence threshold to obtain a prediction set. In response to the confidence of any predicted key point being greater than the preset confidence threshold, the predicted key point is retained in the predicted key point set. Each sample target point is filtered according to the preset confidence threshold to obtain a sample key point set. In response to the confidence of any sample target point being greater than the preset confidence threshold, the sample target point is used as a sample key point and the sample target point is retained in the sample key point set. The confidence difference between each sample target point and each predicted key point in the predicted key point set can be the confidence difference between each sample target point in the sample key point set and each predicted key point in the predicted key point set. Based on the confidence difference, the second loss is determined. In some application scenarios, the confidence difference between each sample target point in the sample key point set and each predicted key point in the predicted key point set is directly determined as the second loss. In other application scenarios, the first candidate loss is determined based on the confidence difference between each sample target point in the sample key point set and each predicted key point in the predicted key point set. The second candidate loss is determined based on the quantity difference and / or position difference between each sample target point in the sample key point set and each predicted key point in the predicted key point set. The first candidate loss and the second candidate loss are weightedly summed, and the sum is used as the second loss. It can be understood that each sample target point includes each sample key point, and the sample target point in the sample key point set is a sample key point. That is, the sample target point in the sample key point set is a key point that meets the preset confidence requirements.

[0066] The second loss is used to adjust the parameters in the key point detection module, and the key point detection module is trained with the goal of reducing the second loss, until the second loss output by the trained key point detection module meets the requirements. In other application scenarios, the key point detection device includes a feature extraction module, a classification module, and a key point detection module, and the feature extraction module is used to execute the above step S11. The classification module is used to execute the above step S12, and the key point detection module is used to execute the above step S13. The second loss is used to adjust the parameters of each module in the key point detection device, and the key point detection device is trained with the goal of reducing the second loss, until the second loss output by the trained key point detection device meets the requirements.

[0067] The first loss and the second loss are weightedly summed, and the sum is used as the target loss. The target loss is used to adjust the parameters in the key point detection module, and the key point detection module is trained with the target loss reduced as the goal, until the target loss output by the trained key point detection module meets the requirements. In other application scenarios, the key point detection device includes a feature extraction module, a classification module and a key point detection module, and the feature extraction module is used to execute the above step S11. The classification module is used to execute the above step S12, and the key point detection module is used to execute the above step S13. The target loss is used to adjust the parameters of each module in the key point detection device, and the key point detection device is trained with the target loss reduced as the goal, until the target loss output by the trained key point detection device meets the requirements.

[0068] In some embodiments, the above-mentioned method of obtaining at least one label heat map of sample image features may include the following steps: First, obtain the sample target points in each label heat map, the annotation attributes corresponding to each label heat map, and the initial confidence of each image point in each label heat map. Each sample target point is an image point with the highest confidence in each label heat map, and the annotation attributes include an occlusion attribute or an unoccluded attribute. Next, lower the initial confidence of at least one point to be adjusted to obtain an updated confidence, and the point to be adjusted is the sample target point corresponding to the label heat map belonging to the occlusion attribute. Subsequently, use the updated confidence to update the initial confidence of each image point in each label heat map to obtain each updated label heat map.

[0069] Obtain an open source training data set. The training data set includes several sample detection images. Each sample detection image corresponds to at least one label heat map. Each label heat map corresponds to a sample target point. Among them, each sample target point is the image point with the highest confidence in the label heat map. The maximum value that the initial confidence of each image point in each of the label heat maps can take can be 1, or it can be customized or a preset value. This application takes the maximum value that the initial confidence of each image point in the label heat map can take as 1 as an example. In the case of distinguishing the annotation attributes of the sample target points corresponding to each label heat map, the maximum value of the confidence of the sample target points in the label heat map belonging to the occlusion attribute and the unoccluded attribute is 1. For example, for the label heat map belonging to the occlusion attribute, the maximum value of the confidence of the sample target point corresponding to the label heat map can be 1. Therefore, the maximum value of the initial confidence of each image point in each label heat map is 1. In some application scenarios, the confidence of the sample target point of the label heat map belonging to the unoccluded attribute is maintained. The maximum value of the confidence of the sample target point of the label heat map belonging to the occluded attribute is reduced to the preset confidence, and the updated sample target point of the label heat map belonging to the occluded attribute is obtained. For example, each image point in any label heat map includes the first image point, the second image point, and the third image point with the top three confidence rankings. Among them, the confidences of the first image point, the second image point, and the third image point are 0.9, 0.8, and 0.66, respectively. The maximum value corresponding to the initial confidence of each image point in any label heat map is 0.9, and the confidence of the sample target point in any label heat map is 0.9. At this time, the first image point is the sample target point corresponding to the label heat map. After the maximum value of the confidence of the sample target point of the label heat map of the occluded attribute is reduced to the preset confidence of 0.7, the third image point is used as the updated sample target point. And the updated sample target point is used as the sample target point in each updated label heat map. Then, the confidence and position information of the updated sample target point are used as the confidence and position information of the sample target point in each updated label heat map.

[0070] In other application scenarios, the point to be adjusted is a sample target point corresponding to the label heat map belonging to the occlusion attribute. The point to be adjusted is the image point with the maximum confidence in each label heat map, and the image point with the maximum confidence is the sample target point corresponding to the label heat map belonging to the occlusion attribute. In other application scenarios, the initial confidence of the point to be adjusted in each label heat map is reduced to obtain an updated confidence. Subsequently, the initial confidence of each image point in each label heat map is updated using the updated confidence to obtain each updated label heat map. Specifically, in some application scenarios, the step of reducing the initial confidence of the point to be adjusted in each label heat map may be to adjust the initial confidence to the target confidence in response to the initial confidence of the point to be adjusted being greater than the preset confidence, and to use the target confidence as the updated confidence. The target confidence may be less than or equal to the preset confidence. In other application scenarios, the step of reducing the initial confidence of the point to be adjusted in each label heat map may be to adjust the transformed value of the initial confidence to the target confidence, and to use the target confidence as the updated confidence. The transformed value of the initial confidence is used to represent the product between the initial confidence of the point to be adjusted and the preset confidence. The product between the initial confidence of the point to be adjusted and the preset confidence is determined as the target confidence. For example, if the initial confidence of the point to be adjusted in any label heat map is 0.9, the preset confidence is 0.7, and the target confidence is 0.63.

[0071] See also Figure 4 , Figure 4 It is a schematic diagram of key points in an embodiment of a key point detection method of the present application.

[0072] Perform image preprocessing on the image to be detected to obtain the image features corresponding to the image to be detected. Figure 4 The a shown in the figure can represent an image to be detected. b can represent any detection frame. The area where b is located is the detection area corresponding to the detection frame. The area image corresponding to the detection area includes the image features of the target object. For example, the target object is a cat. After the image features of the area image are classified, the classification results of the image features corresponding to the area image include that the detection area where the area image is located is the area where the target object is located. Execute step S13 to obtain the key point information obtained by key point detection processing of the image features corresponding to the area image. Figure 4 As shown, the key points in the detection area include at least key point P1, key point P2 and key point P3. The key point information includes the position information of the key points in the detection area. The key point information includes the coordinate information of the key point P1 in the image of the area, the coordinate information of the key point P2 in the image of the area, and the coordinate information of the key point P3 in the image of the area.

[0073] The above scheme obtains image features obtained by extracting features from several detection areas in the image to be detected collected from the target object, and then classifies the image features to obtain the image classification result of the image to be detected. In response to the image classification result being that the image classification result is an area where the target object is located in the several detection areas, key point information obtained by key point detection processing of the image features is obtained, which can ensure that the image features are features corresponding to the target object, avoid the presence of information that does not belong to the target object in the output key point information, thereby improving the accuracy of the output key point information.

[0074] See also Figure 5 , Figure 5 It is a structural schematic diagram of an embodiment of the key point detection device of the present application. The key point detection device 50 includes a first acquisition module 51, a classification module 52 and a second acquisition module 53; the first acquisition module 51 is used to acquire image features, which are obtained by extracting features from several detection areas in the image to be detected collected from the target object; the classification module 52 is used to classify the image features to obtain image classification results of the image to be detected, and the image classification results include several detection areas including the area where the target object is located or several detection areas not including the area where the target object is located; the second acquisition module 53 is used to respond to the image classification result that the image features are included in the area where the target object is located in the several detection areas, and to acquire key point information obtained by key point detection processing of the image features, and the key point information includes the position information of the key points in the several detection areas.

[0075] The above scheme obtains image features obtained by extracting features from several detection areas in the image to be detected collected from the target object, and then classifies the image features to obtain the image classification result of the image to be detected. In response to the image classification result being that the image classification result is an area where the target object is located in the several detection areas, key point information obtained by key point detection processing of the image features is obtained, which can ensure that the image features are features corresponding to the target object, avoid the presence of information that does not belong to the target object in the output key point information, thereby improving the accuracy of the output key point information.

[0076] Please refer to the key point detection method for the functions performed by each module, which will not be repeated here.

[0077] See also Figure 6 , Figure 66 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 60 includes a memory 61 and a processor 62, and the processor 62 is used to execute program instructions stored in the memory 61 to implement the steps in the above-mentioned key point detection method embodiment. In a specific implementation scenario, the electronic device 60 may include but is not limited to: a multi-camera device, a microcomputer, and a server. In addition, the electronic device 60 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0078] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps in the above-mentioned key point detection method embodiment. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0079] The above scheme obtains image features obtained by extracting features from several detection areas in the image to be detected collected from the target object, and then classifies the image features to obtain the image classification result of the image to be detected. In response to the image classification result being that the image classification result is an area where the target object is located in the several detection areas, key point information obtained by key point detection processing of the image features is obtained, which can ensure that the image features are features corresponding to the target object, avoid the presence of information that does not belong to the target object in the output key point information, thereby improving the accuracy of the output key point information.

[0080] See also Figure 7 , Figure 7 The computer-readable storage medium 70 stores program instructions 701, which implement the steps in any of the above key point detection method embodiments when executed by a processor.

[0081] The above scheme obtains image features obtained by extracting features from several detection areas in the image to be detected collected from the target object, and then classifies the image features to obtain the image classification result of the image to be detected. In response to the image classification result being that the image classification result is an area where the target object is located in the several detection areas, key point information obtained by key point detection processing of the image features is obtained, which can ensure that the image features are features corresponding to the target object, avoid the presence of information that does not belong to the target object in the output key point information, thereby improving the accuracy of the output key point information.

[0082] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0083] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0084] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0085] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0086] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A key point detection method, characterized in that: The method comprises: Acquire image features, where the image features are obtained by extracting features from a plurality of detection areas in the image to be detected collected from the target object; Classifying the image features to obtain an image classification result of the image to be detected, wherein the image classification result includes an area in which the target object is located among the plurality of detection areas or an area in which the target object is not located among the plurality of detection areas; In response to the image classification result being that the plurality of detection areas include an area where the target object is located, key point information obtained by key point detection processing of the image features is obtained, the key point information including position information of the key points in the plurality of detection areas.

2. The method according to claim 1, characterized in that After the image features are classified to obtain the image classification result of the image to be detected, the method further includes: Inputting the image features into a key point detection module to perform key point detection, and obtaining an initial heat map corresponding to at least one key point; For each of the initial heat maps, determining the image point with the largest confidence in the initial heat map as the target point corresponding to the initial heat map; The target points in each of the initial heat maps are respectively used as the key points, and the position information of each of the target points corresponding to the initial heat maps is respectively used as the position information of the corresponding key points in the several detection areas.

3. The method according to claim 2, characterized in that The method of taking the target points in each of the initial heat maps as the key points includes: Filtering out the target points whose confidences meet a preset threshold range among the target points; The retained target points are respectively used as the key points.

4. The method according to claim 2, characterized in that: The method further comprises a step of training the key point detection module, wherein the training step comprises: Acquire a sample image feature and at least one label heat map about the sample image feature, wherein the label heat map is obtained from the sample image feature, and the confidence of the annotation of the occluded key point in each of the label heat maps is less than the confidence of the annotation of the unoccluded key point; Inputting sample image features into the key point detection module to perform key point detection to obtain at least one predicted heat map; Based on the difference between each of the label heat maps and each of the predicted heat maps, the parameters in the key point detection module are adjusted to obtain a trained key point detection module.

5. The method according to claim 4, characterized in that The adjusting of the parameters in the key point detection module based on the difference between each of the label heat maps and each of the prediction heat maps to obtain a trained key point detection module includes: Determining a first loss based on a difference between each of the label heat maps and each of the prediction heat maps; Determining a second loss based on a confidence difference between each sample target point and each predicted key point in the predicted key point set, each of the sample target points being obtained by annotating each of the label heat maps, and each of the predicted key points having a confidence greater than a preset confidence threshold; The first loss and the second loss are used to adjust parameters in the key point detection module to obtain the trained key point detection module.

6. The method according to claim 4, characterized in that The method of obtaining at least one label heat map of the sample image feature includes: Obtaining sample target points in each of the label heat maps, the annotation attributes corresponding to each of the label heat maps, and the initial confidence of each image point in each of the label heat maps, wherein each of the sample target points is the image point with the largest confidence in each of the label heat maps, and the annotation attributes include an occlusion attribute or an unoccluded attribute; Lowering an initial confidence of at least one point to be adjusted to obtain an updated confidence, wherein the point to be adjusted is a sample target point corresponding to the label heat map belonging to the occlusion attribute; The updated confidence is used to update the initial confidence of each image point in each label heat map to obtain each updated label heat map.

7. The method according to any one of claims 1 to 6, characterized in that: The obtaining of image features comprises: Acquire the image to be detected; Performing target detection on the image to be detected to obtain a target detection result, wherein the target detection result includes at least one detection frame; Cropping the image to be detected according to the position information of each detection frame to obtain a regional image corresponding to each detection frame; For each of the region images, feature extraction is performed on the region image to obtain image features corresponding to the region image.

8. The method according to claim 7, characterized in that The method further comprises: In response to the image classification result being that the plurality of detection areas do not include an area where the target object is located, receiving an adjustment instruction, the adjustment instruction being used to instruct to adjust position information of each of the detection frames; Using the adjustment instruction to adjust the position information of each detection frame to obtain adjusted position information corresponding to each detection frame; The step of cropping the image to be detected according to the position information of each detection frame to obtain a regional image corresponding to each detection frame includes: The image to be detected is cropped according to the adjusted position information of each detection frame to obtain a regional image corresponding to each detection frame.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 8.

Images