Key point detection model training method, key point detection method and device

Abstract

The invention relates to a key point detection model training method and device, a key point detection method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps that an image sample containing a target object and a sample labeling result corresponding to the image sample are obtained, the sample labeling result comprises a labeling category of the target object, a labeling key point corresponding to the labeling category and a labeling detection frame, and the labeling category is used for representing the display form of the target object in the image sample; inputting the image sample into the initial key point detection model to obtain a sample prediction result of the image sample, the sample prediction result comprising a prediction category of the target object, and a prediction key point and a prediction detection frame corresponding to the prediction category; obtaining a target loss value according to the sample labeling result and the sample prediction result; and adjusting the initial key point detection model based on the target loss value to obtain a key point detection model. By adopting the method, the key point detection robustness can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a key point detection model training method, a key point detection method, an apparatus, a computer device, a storage medium and a computer program product. Background Art

[0002] In the field of intelligent driving, high-precision maps, as an important component, play an important role in vehicle positioning, perception, path planning, and decision-making. The production of high-precision maps requires the extraction of point cloud data of various road elements such as shoulders, signs, traffic lights, second counters, and guardrails.

[0003] In traditional technology, the key points of road elements can be extracted from road element images by using two-dimensional perception, and then the key points are used to reconstruct the road elements in three dimensions to obtain point cloud data of the road elements. However, when using the key point detection method in the above traditional technology, usually only rectangular road elements can be identified, which is difficult to meet the key point detection requirements of road elements of different shapes, and there is a problem of poor robustness. Summary of the invention

[0004] Based on this, it is necessary to provide a training method for a key point detection model with strong robustness, a key point detection method, an apparatus, a computer device, a computer-readable storage medium and a computer program product to address the above technical problems.

[0005] In a first aspect, the present application provides a method for training a key point detection model. The method comprises:

[0006] Acquire an image sample containing a target object, and a sample annotation result corresponding to the image sample, wherein the sample annotation result includes an annotation category of the target object, and an annotation key point and an annotation detection box corresponding to the annotation category, wherein the annotation category is used to characterize a display form of the target object in the image sample;

[0007] Inputting the image sample into an initial key point detection model to obtain a sample prediction result of the image sample, wherein the sample prediction result includes a predicted category of the target object, and predicted key points and a predicted detection box corresponding to the predicted category;

[0008] Obtaining a target loss value according to the sample labeling result and the sample prediction result;

[0009] The initial key point detection model is adjusted based on the target loss value to obtain a key point detection model.

[0010] In one embodiment, the number of predicted key points is a preset first number; and the method further includes:

[0011] In a case where a second number of actual key points corresponding to the predicted category is less than the first number, the predicted key points are obtained based on the actual key points and preset values;

[0012] When the second number is equal to the first number, the predicted key points are obtained according to the actual key points.

[0013] In one of the embodiments, when the second number of actual key points corresponding to the predicted category is less than the first number, the predicted key points are obtained according to the actual key points and preset values, including:

[0014] The detection values ​​of the actual key points are used to fill in the first second number of point information to be filled in the predicted key points, and the preset values ​​are used to fill in the remaining point information to be filled in the predicted key points to obtain the predicted key points.

[0015] In one embodiment, obtaining a target loss value according to the sample labeling result and the sample prediction result includes:

[0016] Obtaining a detection value corresponding to each of the predicted key points in the sample prediction result, and a true value of an annotated key point corresponding to each of the predicted key points in the sample annotated result;

[0017] According to the predicted category of the predicted key point, the detection value and the true value are masked to obtain the target loss value.

[0018] In one embodiment, the predicted category includes an occluded category and an unoccluded category; and the step of performing mask processing on the detection value and the true value according to the predicted category of the predicted key point to obtain the target loss value includes:

[0019] Using a first mask value to perform masking on a detection value and a corresponding true value of the predicted key point under the occlusion category to obtain a first loss value of the predicted key point under the occlusion category;

[0020] Using a second mask value to perform masking on the detection value and the true value corresponding to the predicted key point under the unoccluded category to obtain a second loss value of the predicted key point under the unoccluded category;

[0021] The target loss value is obtained according to the first loss value and the second loss value.

[0022] In one of the embodiments, when the annotation category is an occlusion category, the annotation key points include annotated occlusion key points and annotated unoccluded key points, and the display styles of the annotated occlusion key points and the annotated unoccluded key points are different;

[0023] Correspondingly, the predicted key points include predicted occluded key points and predicted unoccluded key points, and the predicted occluded key points and the predicted unoccluded key points have different display styles.

[0024] In one embodiment, the target object includes any one or more of a traffic sign, a traffic light or a second countdown device.

[0025] In a second aspect, the present application also provides a key point detection method. The method comprises:

[0026] Acquire the image to be tested;

[0027] The image to be tested is input into a key point detection model, and when a target object is detected in the image to be tested, a detection result of the image to be tested is obtained, wherein the detection result includes a category, key points and a detection frame of the target object, and the key point detection model is trained using the method described in any one of the first aspects above.

[0028] In a third aspect, the present application also provides a training device for a key point detection model. The device comprises:

[0029] A sample acquisition module, used to acquire an image sample containing a target object, and a sample annotation result corresponding to the image sample, wherein the sample annotation result includes an annotation category of the target object, and an annotation key point and an annotation detection box corresponding to the annotation category, wherein the annotation category is used to characterize the display form of the target object in the image sample;

[0030] A model prediction module, used to input the image sample into an initial key point detection model to obtain a sample prediction result of the image sample, wherein the sample prediction result includes a predicted category of the target object, and predicted key points and predicted detection boxes corresponding to the predicted category;

[0031] A loss calculation module, used to obtain a target loss value according to the sample labeling result and the sample prediction result;

[0032] A model training module is used to adjust the initial key point detection model based on the target loss value to obtain a key point detection model.

[0033] In a fourth aspect, the present application also provides a key point detection device. The device comprises:

[0034] An image acquisition module, used for acquiring an image to be tested;

[0035] A model detection module is used to input the image to be tested into a key point detection model, and obtain a detection result of the image to be tested when a target object is detected in the image to be tested, wherein the detection result includes the category, key points and detection frame of the target object, and the key point detection model is trained using the method described in any one of the first aspects above.

[0036] In a fifth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the key point detection method described in any one of the embodiments of the first aspect is implemented.

[0037] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the key point detection method described in any one of the embodiments of the first aspect is implemented.

[0038] In a seventh aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the key point detection method described in any one of the embodiments of the first aspect is implemented.

[0039] The training method, key point detection method, device, computer equipment, storage medium and computer program product of the above-mentioned key point detection model obtain an image sample containing a target object, and a sample annotation result corresponding to the image sample, the sample annotation result includes the annotation category of the target object, and the annotation key points and annotation detection box corresponding to the annotation category, and the annotation category is used to characterize the display form of the target object in the image sample; the image sample is input into the initial key point detection model to obtain a sample prediction result of the image sample, the sample prediction result includes the prediction category of the target object, and the prediction key points and prediction detection box corresponding to the prediction category; according to the sample annotation result and the sample prediction result, a target loss value is obtained; based on the target loss value, the initial key point detection model is adjusted to obtain a key point detection model, which can use the annotation category to achieve the key point detection scene segmentation of the target object in different display forms, thereby improving the robustness and accuracy of the key point detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of a flow chart of a method for training a key point detection model in one embodiment;

[0041] FIG2 is a schematic diagram of a key point marking sequence in one embodiment;

[0042] Figure 3 A schematic diagram of marking key points of a target object in another embodiment;

[0043] Figure 4 A schematic diagram of marking key points of a target object in another embodiment;

[0044] Figure 5 A schematic diagram of predicted key points under an occlusion category in one embodiment;

[0045] Figure 6 A schematic diagram of predicted key points under an occlusion category in another embodiment;

[0046] Figure 7 A schematic diagram of a process for generating a target loss value in one embodiment;

[0047] Figure 8 is a flow chart of a key point detection method in another embodiment;

[0048] Fig. 9 A schematic diagram of marking key points of a target object in another embodiment;

[0049] Fig.10 is a structural block diagram of a key point detection device in one embodiment;

[0050] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0053] In one embodiment, Figure 1As shown, a training method for a key point detection model is provided. This embodiment uses the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the server can be an independent server or a server cluster composed of multiple servers. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and Internet of Things devices. The Internet of Things devices can be smart TVs, smart car devices, etc.

[0054] In this embodiment, the method includes the following steps:

[0055] Step S102: obtaining an image sample containing a target object and a sample annotation result corresponding to the image sample.

[0056] The target object may be, but is not limited to, any one or more of a variety of objects such as vehicles, obstacles, traffic signs, traffic lights, and second counters.

[0057] The image sample may represent image data containing the target object that is acquired by an image acquisition device at a historical moment in time.

[0058] The sample annotation results may include but are not limited to the annotation category of the target object, and the annotation key points and the annotation detection box corresponding to the annotation category.

[0059] The annotation category can be used to characterize the display form of the target object in the image sample. The display form can include whether the target object is blocked in the image sample, the shape of the target object (rectangle, circle, regular triangle, inverted triangle, hexagon, octagon, etc.), whether the target object has image size restrictions, whether the target object is near or far, etc.

[0060] The labeled key points may represent the points where the target object is labeled in the image sample, such as the corner points, center points, boundary points, etc. of the target object. The number of the labeled key points may be one or more.

[0061] The annotated detection box may represent a box annotated for the area where the target object is located in the image sample.

[0062] Specifically, the server may obtain an image sample containing a target object, and a sample annotation result corresponding to the image sample consisting of an annotation category of the target object, an annotation key point corresponding to the target object and the annotation category, and an annotation detection frame of the target object.

[0063] Optionally, in some embodiments, the sample annotation result may be obtained through manual annotation. In other embodiments, the sample annotation result may also be generated by using a trained detection model with high detection accuracy.

[0064] Step S104: input the image sample into the initial key point detection model to obtain a sample prediction result of the image sample.

[0065] The initial key point detection model can be used to represent an untrained target detection model. In one example, the initial key point detection model can include but is not limited to any one or more of a region-based convolutional neural network (R-CNN), a single shot multi-box target detection model (SSD model), a lightweight target detection model (You Only Look Once, YOLO), and the like.

[0066] The sample prediction results may include but are not limited to the predicted category of the target object, and the predicted key points and predicted detection boxes corresponding to the predicted category.

[0067] The predicted category may represent the display form of the target object in the image sample predicted by the initial key point detection model. The predicted key point may represent the key point of the target object in the image sample predicted by the initial key point detection model. The predicted detection box may represent the detection box where the target object is located in the image sample predicted by the initial key point detection model. That is, the predicted category, the predicted key point, and the predicted detection box correspond to the labeled category, the labeled key point, and the labeled detection box, respectively.

[0068] Specifically, an initial key point detection model may be pre-deployed in the server. An image sample is input into the initial key point detection model, and features of the image sample are extracted by the initial key point detection model, and a predicted category of the target object, predicted key points corresponding to the target object and the predicted category, and a predicted detection box of the target object are generated based on the extracted image features. The sample annotation result consisting of the predicted category, predicted key points, and predicted detection box output by the initial key point detection model is read.

[0069] Step S106, obtaining a target loss value according to the sample labeling result and the sample prediction result.

[0070] Specifically, the server can generate a category loss value using the labeled category of the target object in the sample labeling result and the predicted category of the target object in the sample prediction result. Generate a key point loss value using the labeled key points in the sample labeling result and the predicted key points in the sample prediction result. Generate a detection box loss value using the labeled detection box in the sample labeling result and the predicted detection box in the sample prediction result. Add the category loss value, the key point loss value, and the detection box loss value to obtain the target loss value of the current initial key point detection model.

[0071] Step S108, adjusting the initial key point detection model based on the target loss value to obtain a key point detection model.

[0072] Specifically, the server can adjust the model parameters of the initial key point detection model in a back-propagation manner in the initial key point detection model based on the target loss value. Then the image sample is input into the adjusted initial key point detection model again for iterative training until the number of iterations meets the preset number threshold, or the target loss value after iteration is less than the preset loss threshold, to obtain a key point detection model that has completed training.

[0073] In the training method of the above-mentioned key point detection model, an image sample containing a target object and a sample annotation result corresponding to the image sample are obtained, the sample annotation result includes the annotation category of the target object, and the annotation key points and the annotation detection box corresponding to the annotation category, and the annotation category is used to characterize the display form of the target object in the image sample; the image sample is input into the initial key point detection model to obtain a sample prediction result of the image sample, the sample prediction result includes the prediction category of the target object, and the prediction key points and the prediction detection box corresponding to the prediction category; according to the sample annotation result and the sample prediction result, a target loss value is obtained; based on the target loss value, the initial key point detection model is adjusted to obtain a key point detection model, which can use the annotation category to realize the key point detection scene segmentation of the target object in different display forms, thereby improving the robustness and accuracy of the key point detection.

[0074] In one embodiment, the number of predicted key points is a preset first number. The training method of the key point detection model provided in the present application may also include: when the second number of actual key points corresponding to the predicted category is less than the first number, the predicted key points are obtained based on the actual key points and the preset value; when the second number is equal to the first number, the predicted key points are obtained based on the actual key points.

[0075] Specifically, the server may store a preset first quantity corresponding to the key point detection requirement. For example, when the key point detection requirement is a road element detection requirement, since the number of sides of a road element is at most eight (the traffic sign of the word "Stop" is an octagon), the first quantity may be set to eight or a value greater than eight to cover all detection situations in the road element detection scenario.

[0076] After the server inputs the image sample into the initial key point detection model, the initial key point detection model can be used to determine the detection frame where the target object is located from the image sample, obtain the grayscale value of each pixel point in the image sample that is located in the detection frame, and determine the grayscale value gradient change in multiple directions at the pixel point with each pixel point as the center of the sliding window. The pixel point whose grayscale value gradient change is greater than the preset gradient threshold is determined as the actual key point of the target object.

[0077] The target objects under different prediction categories have different numbers of actual key points that can be detected due to different display forms. For example, when the display form of the target object is a rectangle or a circle, the number of actual key points corresponding to the prediction category is four; when the display form of the target object is an equilateral triangle or an inverted triangle, the number of actual key points corresponding to the prediction category is three; when the display form of the target object is a hexagon, the number of actual key points corresponding to the prediction category is six; when the display form of the target object is an octagon, the number of actual key points corresponding to the prediction category is eight.

[0078] Therefore, in order to make the initial key point detection model finally output the first number of predicted key points, the server can generate predicted key points by using the actual key points currently detected and the preset values ​​through the initial key point detection model when the second number of actual key points corresponding to the predicted category is less than the first number. Alternatively, when the second number of actual key points corresponding to the predicted category is equal to the first number, the server can directly use the actual key points as predicted key points for output through the initial key point detection model.

[0079] Optionally, in some embodiments, the initial key point detection model can be improved based on an existing target detection model: the server can pass the first number of predicted key points that need to be detected by the model into the data reading network block and the loss calculation network block of the model. Thus, the improved target detection model can perform data reading operations according to the first number to meet the first number requirement for key point detection.

[0080] In this embodiment, the method for generating predicted key points is determined by determining the size relationship between the second number of actual key points corresponding to the predicted category determined based on the initial key point detection model and the preset first number, so that the initial key point detection model can output the same number of predicted key points for target objects in different display forms, thereby meeting the detection scenarios and detection requirements of any number of key points, and further improving the robustness of the initial key point detection model.

[0081] In one embodiment, when the second number of actual key points corresponding to the predicted category is less than the first number, the server can use the detection values ​​of the actual key points to fill in the first second number of point information to be filled in the predicted key points, and use preset values ​​to fill in the remaining point information to be filled in the predicted key points to obtain the predicted key points.

[0082] Specifically, since the initial key point detection model needs to output a first number of predicted key points, the predicted key points with the first number of point information to be filled in can be initialized in the data reading network of the initial key point detection model. In the case where the second number of actual key points corresponding to the predicted category detected by the initial key point detection model is less than the first number, the server can use the detection values ​​of the second number of actual key points through the initial key point detection model to fill the first second number of point information to be filled in the predicted key points, and use the preset values ​​to fill the remaining point information to be filled in the predicted key points, thereby obtaining the predicted key points of the image sample.

[0083] For example, when the first number is 8 and the second number of actual key points corresponding to the predicted category is 3, the server can use the detection values ​​of these 3 actual key points through the initial key point detection model to fill the first 3 points to be filled in the predicted key points, and use the preset values ​​to fill the remaining 5 points to be filled in the predicted key points, thereby obtaining the predicted key points.

[0084] Optionally, in some embodiments, the preset value may be 0 or 1. For example, when the detection values ​​of the actual key points are (x1, y1), (x2, y2), (x3, y3), (x4, y4), the first number is 8, and the preset value is 0, (x1, y1), (x2, y2), (x3, y3), (x4, y4) may be used to fill in the first 4 point information to be filled, and 0 may be used to fill in the last 4 point information, to obtain the predicted key points {(x1, y1), (x2, y2), (x3, y3), (x4, y4), (0, 0), (0, 0), (0, 0), (0, 0)}.

[0085] In this embodiment, when the second number of actual key points corresponding to the predicted category is less than the first number, the detection values ​​of the actual key points are used to fill the first second number of point information to be filled in the predicted key points, and the preset values ​​are used to fill the remaining point information to be filled in the predicted key points, so as to obtain predicted key points. This can be applicable to target object key point detection scenarios under a variety of different display forms, thereby improving the robustness of key point detection.

[0086] In one embodiment, step S106 obtains a target loss value based on the sample annotation results and the sample prediction results, including: obtaining the detection value corresponding to each predicted key point in the sample prediction result, and the true value of the annotated key point corresponding to each predicted key point in the sample annotation result, and masking the detection value and the true value according to the prediction category of the predicted key point to obtain the target loss value.

[0087] Among them, the detection value may include but is not limited to any one or more of the following values: the point coordinates of the predicted key point, the predicted category, the offset distance from the predicted detection box (generally the distance between the upper left corner of the predicted detection box and the predicted key point is the offset distance).

[0088] The true value may include but is not limited to any one or more values ​​such as the point coordinates of the marked key point, the marked category, and the offset distance from the marked detection box.

[0089] Specifically, the server can detect the initial key point model as shown in Figure 2. Figure 3 or Figure 4 The predicted key points of the image sample are generated in the preset order shown. For example, when the predicted category of the target object is a rectangle, a traffic light, or a second countdown device, the corner point of the upper left corner of the target object is used as the first actual key point of the target object, and the predicted key points corresponding to the target object are obtained in the order of the upper left corner, the upper right corner, the lower right corner, and the lower left corner. When the predicted category of the target object is a circle, the predicted key points are generated in the order of the left boundary point, the upper boundary point, the right boundary point, and the lower boundary point of the target object. When the predicted category of the target object is an equilateral triangle, the predicted key points are generated in the order of the left vertex, the upper vertex, and the right vertex of the target object. When the predicted category of the target object is an inverted triangle, the predicted key points are generated in the order of the left vertex, the right vertex, and the lower vertex. When the predicted category of the target object is a high-speed hexagon, the predicted key points are generated starting from the upper left corner of the target object in a clockwise direction. When the predicted category of the target object is an urban octagon, the predicted key points are generated in the order of the directions shown in the figure.

[0090] Using a preset order, determine the labeled key points corresponding to each predicted key point in the sample labeling results. Obtain the detection value corresponding to each predicted key point in the sample prediction results output by the initial key point detection model, as well as the true value of the labeled key point corresponding to each predicted key point in the sample labeling results. Determine the corresponding mask value according to the prediction category of the predicted key point. Use the corresponding mask value to mask the detection value of each predicted key point and the true value of the corresponding labeled key point, and use the masked detection value of the predicted key point and the true value of the corresponding labeled key point to calculate the target loss value of the initial key point detection model.

[0091] For example, when the predicted category score of the predicted key point exceeds the preset score threshold, the detection value of the predicted key point and the corresponding true value can be masked using the first threshold. When the predicted category score is less than the preset score threshold, the detection value and the true value can be masked using the second threshold.

[0092] Alternatively, when the prediction category of the predicted key point is a preset category, the detection value and the corresponding true value of the predicted key point under the prediction category may be masked using a first threshold, and the detection value and the true value of the predicted key points under the remaining prediction categories may be masked using a second threshold.

[0093] Optionally, in some embodiments, the target loss value may be obtained by referring to the following formula:

[0094]

[0095]

[0096] Wherein, x may represent the target loss value. weight may represent whether the currently processed image sample is a positive sample or a negative sample. When the image sample is a positive sample, weight is 1. When the image sample is a negative sample, weight is 0 to eliminate the influence of the negative sample on the model loss value. pred may represent the detection value of the predicted key point. mask may represent the mask value corresponding to the predicted category of the predicted key point. true may represent the true value of the labeled key point corresponding to the predicted key point.

[0097] Wing(x) can be expressed as the fitting function of the target loss value. w is the fitting coefficient, which is used to limit the nonlinear part to the interval [−w,w]. ϵ is the curvature of the constrained nonlinear region, and is a constant that smoothly connects the linear and nonlinear parts of the piece.

[0098] In this embodiment, by obtaining the detection value corresponding to each predicted key point in the sample prediction result and the true value of the labeled key point corresponding to each predicted key point in the sample labeling result, the detection value and the true value are masked using the prediction category of the predicted key point to obtain the target loss value of the initial key point detection model. The loss value weight of the predicted key points under different prediction categories for model training can be adjusted, which is beneficial to improve the accuracy of the target loss value.

[0099] In one embodiment, the prediction category may include an occlusion category and an unocclusion category. The occlusion category may indicate that the predicted key point is occluded in the image sample. The unocclusion category may indicate that the predicted key point is not occluded in the image sample. Figure 5As shown in , when a traffic sign is used as the target object, the two corner points on its left side can be used as the predicted key points in the unobstructed category, and the two corner points on its right side can be used as the predicted key points in the obstructed category because they are obstructed by obstacles. Figure 6 As shown in the figure, when a traffic sign is used as the target object, the target object is not fully displayed in the image sample due to image size limitations. In this case, the key points above the target object can be used as predicted key points under the occluded category, and the remaining key points of the target object can be used as predicted key points under the unoccluded category.

[0100] like Figure 7 As shown, a flowchart of the target loss value generation step is provided, including:

[0101] Step S702: Use a first mask value to perform mask processing on the detection value and the true value corresponding to the predicted key point under the occlusion category to obtain a first loss value of the predicted key point under the occlusion category.

[0102] Specifically, the server may obtain a pre-stored first mask value corresponding to the occlusion category. The first mask value is used to perform masking on the detection value and the corresponding true value of the predicted key point under the occlusion category to obtain the masked detection value and the corresponding true value of the predicted key point under the occlusion category. The difference between the masked detection value and the corresponding true value of the predicted key point under the occlusion category is used as the first loss value of the predicted key point under the occlusion category.

[0103] Step S704: Use the second mask value to perform masking on the detection value and the true value corresponding to the predicted key point in the unoccluded category to obtain a second loss value of the predicted key point in the unoccluded category.

[0104] Step S706, obtaining a target loss value according to the first loss value and the second loss value.

[0105] Specifically, the server can obtain a second mask value corresponding to the unobstructed category. Referring to the above-mentioned method of obtaining the first loss value, the second mask value can be used to mask the detection value and the true value corresponding to the predicted key point under the unobstructed category, thereby obtaining the second loss value of the predicted key point under the unobstructed category. The first loss value and the second loss value can be added to obtain the target loss value; or, the average of the first loss value and the second loss value can be used as the target loss value; or, the larger loss value between the first loss value and the second loss value can be used as the target loss value.

[0106] Optionally, in some embodiments, the first mask value may be 0. The second mask value may be 1.

[0107] Since the predicted key points under the occluded category are usually inferred by the initial key point detection model based on the key point features (geometric pattern) of the target object, for example, when the target object is a circle, the predicted key points of the circle can be inferred by using the characteristic pattern that the key points of the circle are distributed in a diamond shape within the detection frame. In addition, the predicted key points under the occluded category generally do not have obvious characteristics of pixel grayscale gradient changes, which is quite different from the predicted key points under the unoccluded category. This results in the predicted key point detection loss under the occluded category, which is likely to have a negative impact on the initial key point detection model learning the key point features under the unoccluded category.

[0108] Therefore, the detection loss of the predicted key points in the occluded category is processed with a mask value of 0, which can completely block the influence of the predicted key points in the occluded category on the loss value of the initial key point detection model training. The predicted key points in the unoccluded category are processed with a mask value of 1, which can retain the predicted key point features in the unoccluded category, so that the initial key point detection model can better learn the key point features of the target object, achieve effective fitting of the key points, and further improve the accuracy of the key point detection model obtained by subsequent training.

[0109] In this embodiment, by using different mask values ​​to calculate the loss value for the predicted key points under the occluded category and the predicted key points under the unoccluded category, and then using the first loss value of the predicted key points under the occluded category and the second loss value of the predicted key points under the unoccluded category to obtain the target loss value, the impact of the detection loss of the predicted key points under the occluded category on the model training can be adjusted, which is beneficial to improve the accuracy of the key point detection model training.

[0110] In one embodiment, when the annotation category is an occlusion category, the annotation key points include annotated occlusion key points and annotated unoccluded key points, and the display styles of the annotated occlusion key points and the annotated unoccluded key points are different. Correspondingly, the predicted key points include predicted occlusion key points and predicted unoccluded key points, and the display styles of the predicted occlusion key points and the predicted unoccluded key points are different.

[0111] For example, the display style of the marked occluded key points can be green. The display style of the marked unoccluded key points can be blue. Correspondingly, the display style of the predicted occluded key points generated by the initial key point detection model can also be green, and the display style of the predicted unoccluded key points can be blue. Alternatively, the display style of the marked occluded key points can be hollow points. The display style of the marked unoccluded key points can be solid points. Correspondingly, the predicted occluded key points are also hollow points. The predicted unoccluded key points are solid points.

[0112] In this embodiment, by adopting different display styles for key points in the occluded category and the unoccluded category, it is possible to intuitively display and distinguish key points in the occluded category and the unoccluded category.

[0113] In one embodiment, the target object may include any one or more of a traffic sign, a traffic light, or a second countdown device.

[0114] In one embodiment, Figure 8 As shown, a training method for a key point detection model is provided, including:

[0115] Step S802: Obtain an image sample containing a target object and a sample annotation result corresponding to the image sample.

[0116] Specifically, the server can obtain image samples containing target objects, such as traffic signs, traffic lights, or second countdown devices, and other road elements as target objects, which are derived from driving forward image data collected by a collection vehicle. The sample annotation results corresponding to the image samples are obtained. The sample annotation results may include but are not limited to the annotation categories of the target objects, as well as the annotation key points and annotation detection boxes corresponding to the annotation categories. The annotation categories are used to characterize the display form of the target objects in the image samples.

[0117] Optionally, in some embodiments, the annotation categories may include unobstructed categories in the case of conventional annotations corresponding to traffic signs (as shown in FIG. 2 ) or categories with complete shape expressions in the case of image size limitations (as shown in FIG. Fig. 9 ), and the occlusion category under the complex annotation category (such as Figure 5 as shown) or categories with incomplete shape expressions (e.g. Figure 6 As shown in the figure), as well as the near-field annotation category corresponding to the traffic light (when the traffic light is near, it is not a regular rectangle, so it is necessary to use specific positions as corner points for annotation) and the far-field annotation category (when the traffic light is collected at a distance, its shape is approximately rectangular, and rectangular corner points can be used for annotation, refer to Figure 3 As shown in the figure), as well as the near-field annotation category corresponding to the second counter (when the second counter is near, it is not a regular rectangle, so it is necessary to use specific positions as corner points for annotation) and the far-field annotation category (since only the part where the second counter counts needs to be framed, it is consistent with the traffic light annotation, which can be referred to as shown in the figure below). Figure 4 The situation shown is marked).

[0118] Step S804: input the image sample into the initial key point detection model.

[0119] Step S806, when the second number of actual key points corresponding to the predicted category is less than the first number of predicted key points, the detection values ​​of the actual key points are used to fill the first second number of point information to be filled, and the preset values ​​are used to fill the remaining point information to be filled to obtain the predicted key points.

[0120] Step S808: When the second number of actual key points corresponding to the predicted category is equal to the first number of predicted key points, the predicted key points are generated using the detection values ​​of the actual key points.

[0121] Specifically, the server can modify the data reading network and the loss calculation network in the YOLO7 detection model (a lightweight target detection model) to obtain an initial key point detection model that can detect a preset first number of predicted key points. The image sample is input into the initial key point detection model to obtain the actual key points corresponding to the target object and the predicted category in the image sample.

[0122] When the second number of actual key points is less than the first number, the detection values ​​of the actual key points are used to fill in the first second number of point information to be filled, and the preset values ​​are used to fill in the remaining point information to be filled, so as to obtain predicted key points. When the second number of actual key points is equal to the first number, the detection values ​​of the actual key points are used to fill in all the point information to be filled, so as to obtain predicted key points.

[0123] For example, when the first number is 8, for the predicted key points under the near-field markings of rectangle, circle, traffic light / second counter, 4 actual key points are used to fill the first 4 point information to be filled, and 0 is used to fill the last 4 point information to be filled. For the predicted key points under the prediction category of regular triangle / inverted triangle, 3 actual key points are used to fill the first 3 point information to be filled, and 0 is used to fill the last 5 point information to be filled. For the predicted key points whose prediction category is hexagon, 6 actual key points are used to fill the first 6 point information to be filled, and 0 is used to fill the last 2 point information to be filled. For the predicted key points whose prediction category is octagon, 8 actual key points are used to fill all the point information to be filled.

[0124] Step S810, using a first mask value to perform masking processing on the detection value and the corresponding true value of the predicted key point under the occlusion category, to obtain a first loss value under the occlusion category.

[0125] Step S812: Use the second mask value to perform mask processing on the detection value and the corresponding true value of the predicted key point in the unoccluded category to obtain a second loss value in the unoccluded category.

[0126] Specifically, the server can obtain the detection value corresponding to each predicted key point in the sample prediction result, and the true value of the annotated key point corresponding to each predicted key point in the sample annotation result. The detection value and the corresponding true value of the predicted key point under the occluded category are masked using the first mask value to obtain the first loss value under the occluded category. The detection value and the corresponding true value of the predicted key point under the unoccluded category are masked using the second mask value to obtain the second loss value under the unoccluded category.

[0127] Step S814, obtaining a target loss value according to the first loss value and the second loss value, and adjusting the initial key point detection model based on the target loss value to obtain a key point detection model.

[0128] Specifically, the server may calculate a target loss value based on the first loss value and the second loss value, perform back propagation in the initial key point detection model based on the target loss value to adjust the model parameters, and input the image sample into the initial key point detection model again for iterative training, thereby obtaining the key point detection model.

[0129] In this embodiment, the key point detection model is trained in the above manner, and the key point detection scenes of target objects in different display forms can be subdivided by labeling categories. The initial key point detection model is optimized and adjusted based on the occlusion of the target object and the number of key points. The robustness of the key point detection model can be improved, the key point detection requirements in different scenarios can be met, and higher detection accuracy can be achieved.

[0130] In one embodiment, a key point detection method is provided, including: obtaining an image to be tested, and inputting the image to be tested into a key point detection model trained in any of the above embodiments. When a target object is detected in the image to be tested, a detection result of the image to be tested is obtained. The detection result may include but is not limited to the category, key points and detection frame of the target object.

[0131] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0132] Based on the same inventive concept, the embodiment of the present application also provides a training device for a key point detection model and a key point detection device for the key point detection method for implementing the training method of the key point detection model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device / key point detection device for one or more key point detection models provided below can be found in the above limitations on the key point detection method, and will not be repeated here.

[0133] In one embodiment, Fig.10 As shown, a training device 1000 for a key point detection model is provided, comprising: a sample acquisition module 1002, a model prediction module 1004, a loss calculation module 1006 and a model training module 1008, wherein:

[0134] The sample acquisition module 1002 is used to obtain an image sample containing a target object and a sample annotation result corresponding to the image sample. The sample annotation result includes an annotation category of the target object, and annotation key points and an annotation detection box corresponding to the annotation category. The annotation category is used to characterize the display form of the target object in the image sample.

[0135] The model prediction module 1004 is used to input the image sample into the initial key point detection model to obtain the sample prediction result of the image sample, and the sample prediction result includes the predicted category of the target object, and the predicted key points and predicted detection boxes corresponding to the predicted category.

[0136] The loss calculation module 1006 is used to obtain a target loss value according to the sample labeling result and the sample prediction result.

[0137] The model training module 1008 is used to adjust the initial key point detection model based on the target loss value to obtain the key point detection model.

[0138] In one embodiment, the number of predicted key points is a preset first number. When the second number of actual key points corresponding to the predicted category is less than the first number, the predicted key points are obtained based on the actual key points and the preset value; when the second number is equal to the first number, the predicted key points are obtained based on the actual key points.

[0139] In one embodiment, the model prediction module 1004 is also used to fill the first second number of to-be-filled point information in the predicted key points with the detection values ​​of the actual key points, and to fill the remaining to-be-filled point information in the predicted key points with preset values ​​to obtain the predicted key points.

[0140] In one embodiment, the loss calculation module 1006 is also used to obtain the detection value corresponding to each predicted key point in the sample prediction result, and the true value of the labeled key point corresponding to each predicted key point in the sample labeling result; according to the prediction category of the predicted key point, the detection value and the true value are masked to obtain the target loss value.

[0141] In one embodiment, the prediction category includes an occluded category and an unoccluded category. The loss calculation module 1006 is further used to use a first mask value to perform mask processing on the detection value and the corresponding true value of the prediction key point under the occluded category to obtain a first loss value of the prediction key point under the occluded category; use a second mask value to perform mask processing on the detection value and the true value corresponding to the prediction key point under the unoccluded category to obtain a second loss value of the prediction key point under the unoccluded category; and obtain a target loss value based on the first loss value and the second loss value.

[0142] In one embodiment, when the annotation category is an occlusion category, the annotated key points include annotated occluded key points and annotated unoccluded key points, and the display styles of the annotated occluded key points and the annotated unoccluded key points are different; accordingly, the predicted key points include predicted occluded key points and predicted unoccluded key points, and the display styles of the predicted occluded key points and the predicted unoccluded key points are different.

[0143] In one embodiment, the target object includes any one or more of a traffic sign, a traffic light, or a second countdown device.

[0144] In one embodiment, a key point detection device is also provided, including: an image acquisition module for acquiring an image to be tested, and a model detection module for inputting the image to be tested into a key point detection model. When a target object is detected in the image to be tested, a detection result of the image to be tested is obtained, and the detection result includes the category, key points and detection frame of the target object. The key point detection model is trained using the training method of the key point detection model in any of the above embodiments.

[0145] Each module in the training device / key point detection device of the key point detection model can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0146] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.11As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as image samples, sample annotation results, sample prediction results, target loss values, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a key point detection method is implemented.

[0147] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0148] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0149] In one embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0150] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0151] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0152] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0153] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for a key point detection model, characterized in that: The method comprises: Acquire an image sample containing a target object, and a sample annotation result corresponding to the image sample, wherein the sample annotation result includes an annotation category of the target object, and an annotation key point and an annotation detection box corresponding to the annotation category, wherein the annotation category is used to characterize a display form of the target object in the image sample; Inputting the image sample into an initial key point detection model to obtain a sample prediction result of the image sample, wherein the sample prediction result includes a predicted category of the target object, and predicted key points and a predicted detection box corresponding to the predicted category; Obtaining a target loss value according to the sample labeling result and the sample prediction result; The initial key point detection model is adjusted based on the target loss value to obtain a key point detection model.

2. The method according to claim 1, characterized in that The number of predicted key points is a preset first number; the method further includes: In a case where a second number of actual key points corresponding to the predicted category is less than the first number, the predicted key points are obtained based on the actual key points and preset values; When the second number is equal to the first number, the predicted key points are obtained according to the actual key points.

3. The method according to claim 2, characterized in that In the case where the second number of actual key points corresponding to the predicted category is less than the first number, the predicted key points are obtained according to the actual key points and preset values, including: The detection values ​​of the actual key points are used to fill in the first second number of point information to be filled in the predicted key points, and the preset values ​​are used to fill in the remaining point information to be filled in the predicted key points to obtain the predicted key points.

4. The method according to claim 1, characterized in that: The obtaining a target loss value according to the sample labeling result and the sample prediction result includes: Obtaining a detection value corresponding to each of the predicted key points in the sample prediction result, and a true value of an annotated key point corresponding to each of the predicted key points in the sample annotated result; According to the predicted category of the predicted key point, the detection value and the true value are masked to obtain the target loss value.

5. The method according to claim 4, characterized in that The predicted categories include an occluded category and an unoccluded category; The step of performing mask processing on the detection value and the true value according to the prediction category of the prediction key point to obtain the target loss value includes: Using a first mask value to perform masking on a detection value and a corresponding true value of the predicted key point under the occlusion category to obtain a first loss value of the predicted key point under the occlusion category; Using a second mask value to perform masking on the detection value and the true value corresponding to the predicted key point under the unoccluded category to obtain a second loss value of the predicted key point under the unoccluded category; The target loss value is obtained according to the first loss value and the second loss value.

6. The method according to claim 1, characterized in that When the annotation category is an occlusion category, the annotation key points include annotated occlusion key points and annotated unoccluded key points, and the display styles of the annotated occlusion key points and the annotated unoccluded key points are different; Correspondingly, the predicted key points include predicted occluded key points and predicted unoccluded key points, and the predicted occluded key points and the predicted unoccluded key points have different display styles.

7. The method according to any one of claims 1 to 6, characterized in that The target object includes any one or more of a traffic sign, a traffic light or a second countdown device.

8. A key point detection method, characterized in that: The method comprises: Acquire the image to be tested; The image to be tested is input into a key point detection model, and when a target object is detected in the image to be tested, a detection result of the image to be tested is obtained, wherein the detection result includes a category, key points and a detection frame of the target object, and the key point detection model is trained using the method described in any one of claims 1 to 7.

9. A training device for a key point detection model, characterized in that: The device comprises: A sample acquisition module, used to acquire an image sample containing a target object, and a sample annotation result corresponding to the image sample, wherein the sample annotation result includes an annotation category of the target object, and an annotation key point and an annotation detection box corresponding to the annotation category, wherein the annotation category is used to characterize the display form of the target object in the image sample; A model prediction module, used to input the image sample into an initial key point detection model to obtain a sample prediction result of the image sample, wherein the sample prediction result includes a predicted category of the target object, and predicted key points and predicted detection boxes corresponding to the predicted category; A loss calculation module, used to obtain a target loss value according to the sample labeling result and the sample prediction result; A model training module is used to adjust the initial key point detection model based on the target loss value to obtain a key point detection model.

10. A key point detection device, characterized in that: The device comprises: An image acquisition module, used for acquiring an image to be tested; A model detection module is used to input the image to be tested into a key point detection model, and obtain a detection result of the image to be tested when a target object is detected in the image to be tested, wherein the detection result includes the category, key points and detection frame of the target object, and the key point detection model is trained using the method described in any one of claims 1 to 7.

Images