A method and device for model training and target detection

Abstract

This specification discloses a method and device for model training and target detection. It can be applied to an autonomous driving system. During the training process, a labeled heat map mapped based on an elliptical Gaussian kernel function is used to supervise the predicted heat map, so that the heat values ​​of each point on the target object in the predicted heat map output by the trained target detection model are distributed in an elliptical shape. The closer the point is to the center of the ellipse, the higher the heat value, and the farther the point is from the center of the ellipse, the lower the heat value. In the real world, the shape of a vehicle is more like a rectangle than a square or a circle. Therefore, the heat map output by the target detection model trained based on the above method can accurately show the distribution probability of the center point of the target object at each position.

Description

Technical Field

[0001] This specification relates to the field of machine learning technology, and in particular to a method and device for model training and target detection. Background Art

[0002] One of the tasks of the target detection algorithm is to locate the position of the target object in the data to be detected using a detection box with a relatively simple geometric shape.

[0003] In the ideas of some target detection algorithms, the prediction of the detection frame of the target object can be replaced by predicting the key points of the target object. For example, the position of the center point of the target object and the size of the detection frame of the target object can be predicted to obtain the predicted detection frame of the target object.

[0004] Among them, the target detection model can output a predicted heat map. In the predicted heat map, the heat value of each point on the target object represents the probability that the point is the center point of the target object. Generally, for each point on the target object in the predicted heat map, the greater the heat value of the point, the greater the probability that the point is the center point of the target object. Then, the point with the maximum heat value in the predicted heat map can be used as the key point of the target object where the point is located.

[0005] In the prior art, the annotation heat map is often obtained based on the annotation center point of the target object and the circular Gaussian kernel function projection (such as Figure 1 ), and uses the annotated heatmap to supervise the predicted heatmap output by the target detection model. In this case, in the area of ​​the annotated heatmap projected by each point on the target object, the heat values ​​of each point decrease along the radius from the annotated center point, indicating that the probability distribution of each point on the target object being the center point is that the greater the distance from the annotated center point, the smaller the probability of the point being the center point.

[0006] However, when the target object to be detected is a vehicle, especially a vehicle that is long and relatively narrow, such as a truck, the target detection model trained with the labeled heat map projected by the circular Gaussian kernel distribution as supervision cannot accurately represent the probability of the center point of the target object being distributed at each location. Summary of the Invention

[0007] This specification provides a method and device for model training and target detection to partially solve the above-mentioned problems existing in the prior art.

[0008] This manual adopts the following technical solutions:

[0009] This manual provides a model training method, including:

[0010] Determining a sample image to be detected;

[0011] Determining image features corresponding to the sample image to be detected by using a feature extraction subnet in the target detection model to be trained;

[0012] Using the image features output by the feature extraction subnet as input to a first prediction subnet of an object detection model, and outputting a predicted heat map of the sample image through the first prediction subnet;

[0013] Obtaining a pre-determined annotated heat map for the sample image, and adjusting parameters in the object detection model with the goal of minimizing the difference between the predicted heat map and the annotated heat map of the sample image;

[0014] The step of pre-determining a labeling heat map for the sample image specifically includes:

[0015] Obtain a labeling center point determined for the target object contained in the sample image, and based on the labeling center point and a specified elliptical Gaussian kernel function, map each point contained in the sample image to a labeling heat map to be labeled, thereby obtaining a labeling heat map of the labeled sample image.

[0016] Optionally, the method further includes:

[0017] Determining, based on the predicted heat map of the sample image, a plurality of predicted key points of the target object in the predicted heat map;

[0018] Inputting the image features output by the feature extraction subnet into the second prediction subnet of the object detection model, and outputting, through the second prediction subnet, for each predicted key point, the predicted distance between the predicted key point and the predicted detection box of the target object in each specified direction as each predicted distance corresponding to the predicted key point;

[0019] For each predicted key point, determining a predicted detection box predicted for the target object based on the predicted key point according to the predicted key point and the predicted distances corresponding to the predicted key point;

[0020] Obtaining a predetermined annotated detection frame of the target object in the annotated heat map, and adjusting parameters in the target detection model with the goal of minimizing the difference between each predicted detection frame and the annotated detection frame;

[0021] Predetermining the label detection box of the target object in the label heat map specifically includes:

[0022] Obtain a labeled detection box determined for the target object contained in the sample image, and a labeled center point determined for the target object contained in the sample image; based on the labeled center point and a specified elliptical Gaussian kernel function, map the labeled detection box contained in the sample image to the labeled heat map to be labeled, and obtain the labeled detection box of the target object in the labeled heat map.

[0023] Optionally, determining a plurality of predicted key points of the target object in the predicted heat map according to the predicted heat map of the sample image specifically includes:

[0024] Determining a heat value of each point on the target object in the predicted heat map, wherein, for each point on the target object in the predicted heat map, a greater heat value of the point indicates a greater probability that the point is the center point of the target object;

[0025] Points with heat values ​​greater than a preset heat threshold are selected from various points on the target object as predicted key points on the target object.

[0026] Optionally, adjusting parameters in the object detection model with the goal of minimizing the difference between each predicted detection box and the labeled detection box specifically includes:

[0027] Determine a detection frame set consisting of the predicted detection frames and the labeled detection frames;

[0028] For each detection box in the detection box set, determine each designated corner point of the detection box, and adjust parameters in the object detection model with the goal of minimizing the difference between the corresponding designated corner points of each predicted detection box and the labeled detection box, wherein the number of the designated corner points is at least two.

[0029] Optionally, the method further includes:

[0030] Using the image features output by the feature extraction subnet as input to a third prediction subnet of the target detection model, and predicting the deflection angle of the detection box of the target object relative to a specified direction through the third prediction subnet as the predicted deflection angle of the target object;

[0031] The determined annotated deflection angle of the target object is obtained, and parameters in the target detection model are adjusted with the goal of minimizing the difference between the predicted deflection angle and the annotated deflection angle of the target object.

[0032] Optionally, before determining the sample image to be detected, the method further includes:

[0033] Obtain the point cloud data to be detected;

[0034] Determine the sample image to be detected, including:

[0035] Projecting the point cloud data onto a specified plane in the space where the point cloud data is located, and obtaining a projection image on the specified plane as a sample image to be detected;

[0036] Determine the image features corresponding to the sample image to be detected by using the feature extraction subnet in the target detection model to be trained, specifically including:

[0037] Dividing the space where the point cloud data is located into a plurality of voxels, and performing feature extraction on the point cloud data in each voxel to obtain the extracted point cloud features in the voxel;

[0038] According to the extracted point cloud features of each voxel, a point cloud feature map on the specified plane is determined through the feature extraction subnet in the target detection model to be trained, as the determined image feature corresponding to the sample image to be detected.

[0039] Optionally, the method further includes:

[0040] Using the image features output by the feature extraction subnet as input to a fourth prediction subnet of the target detection model, and predicting the height of the detection box of the target object through the fourth prediction subnet as the predicted height of the target object;

[0041] The determined annotated height of the target object is obtained, and parameters in the target detection model are adjusted with the goal of minimizing the difference between the predicted height and the annotated height of the target object.

[0042] This specification provides a target detection method, including:

[0043] Determine the target image to be detected;

[0044] Determine the image features corresponding to the target image using a feature extraction subnet in the target detection model trained by any of the above methods;

[0045] The image features output by the feature extraction subnet are used as input to a first prediction subnet of a target detection model. The first prediction subnet outputs a predicted heat map of the target image, wherein, for each point on the target object in the predicted heat map, a greater heat value of the point indicates a greater probability that the point is the center point of the target object.

[0046] Based on the predicted heat map of the target image, the predicted detection box of the target object is output.

[0047] Optionally, outputting a predicted detection box of the target object according to the predicted heat map of the target image specifically includes:

[0048] Determine the heat value of each point in the prediction heat map, and use the point with the extreme heat value on the target object as the prediction key point of the target object;

[0049] Using the image features output by the feature extraction subnet as input to a second prediction subnet of the target detection model, and outputting, through the second prediction subnet, a predicted distance between the predicted key point of the target object and the predicted detection box of the target object in each specified direction;

[0050] The predicted detection frame of the target object is determined according to the predicted key points of the target object and the predicted distances of the predicted key points of the target object from the predicted detection frame of the target object in each specified direction.

[0051] Optionally, outputting a predicted detection box of the target object according to the predicted heat map of the target image specifically includes:

[0052] Determining a shape of a predicted detection box of the target object according to a predicted heat map of the target image;

[0053] Using the image features output by the feature extraction subnet as input to a third prediction subnet of the target detection model, and outputting a deflection angle of the detection frame of the target object relative to a specified direction as a predicted deflection angle of the target object through the third prediction subnet;

[0054] Outputting the predicted detection box of the target object according to the shape of the predicted detection box of the target object and the predicted deflection angle of the target object.

[0055] Optionally, extracting image features corresponding to the target image specifically includes:

[0056] Obtain the point cloud data to be detected;

[0057] Dividing the space where the point cloud data is located into a plurality of voxels, and performing feature extraction on the point cloud data in each voxel to obtain the extracted point cloud features in the voxel;

[0058] According to the extracted point cloud features of each voxel, a point cloud feature map on the specified plane is determined by a feature extraction subnet in the target detection model as the extracted image features corresponding to the target image;

[0059] Outputting a predicted detection box of the target object based on the predicted heat map of the target image specifically includes:

[0060] According to the predicted heat map of the target image, a three-dimensional detection box of the target object contained in the target image in the three-dimensional space is output.

[0061] Optionally, outputting a three-dimensional detection box of a target object contained in the target image in three-dimensional space according to the predicted heat map of the target image specifically includes:

[0062] Determine a two-dimensional detection frame of the target object in the target image according to the predicted heat map of the target image, and use the two-dimensional detection frame as the predicted detection frame of the target object;

[0063] Using the image features output by the feature extraction subnet as input to a fourth prediction subnet of the target detection model, and outputting the height of the target object from the specified plane as the predicted height of the target object through the fourth prediction subnet;

[0064] A three-dimensional detection frame of the target object in three-dimensional space is determined based on the two-dimensional detection frame of the target object in the target image and the predicted height of the target object, and the determined three-dimensional detection frame is updated as the predicted detection frame of the target object.

[0065] This specification provides a model training device, including:

[0066] An image determination module, used to determine a sample image to be detected;

[0067] A feature extraction module is used to determine the image features corresponding to the sample image to be detected through the feature extraction subnet in the target detection model to be trained;

[0068] a heat map prediction module, configured to use the image features output by the feature extraction subnet as input to a first prediction subnet of an object detection model, and output a predicted heat map of the sample image through the first prediction subnet;

[0069] a parameter adjustment module, configured to obtain a pre-determined annotated heat map for the sample image, and adjust parameters in the object detection model with the goal of minimizing the difference between the predicted heat map and the annotated heat map of the sample image;

[0070] The labeling module is used to pre-determine a labeling heat map for the sample image, specifically comprising: obtaining a labeling center point determined for a target object contained in the sample image; based on the labeling center point and a specified elliptical Gaussian kernel function, mapping each point contained in the sample image to the labeling heat map to be labeled, thereby obtaining the labeled labeling heat map of the sample image.

[0071] This specification provides a target detection device, including:

[0072] An image determination module, used to determine a target image to be detected;

[0073] A feature extraction module is used to determine the image features corresponding to the target image using the feature extraction subnet in the target detection model trained by the above method;

[0074] a heat map prediction module, configured to use the image features output by the feature extraction subnet as input to a first prediction subnet of a target detection model, and output a predicted heat map of the target image through the first prediction subnet, wherein, for each point on the target object in the predicted heat map, the greater the heat value of the point, the greater the probability that the point is the center point of the target object;

[0075] The target detection module is used to output the predicted detection box of the target object based on the predicted heat map of the target image.

[0076] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned model training and target detection methods.

[0077] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned model training method when executing the program.

[0078] This specification provides an autonomous driving device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned target detection method when executing the program.

[0079] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0080] In the model training and target detection methods provided in this specification, during the training process, a labeled heat map mapped based on an elliptical Gaussian kernel function is used to supervise the predicted heat map, so that the heat values ​​of each point on the target object in the predicted heat map output by the trained target detection model are distributed in an elliptical shape. The closer the point is to the center of the ellipse, the higher the heat value, and the farther the point is from the center of the ellipse, the lower the heat value. The shape of a vehicle in the real world is more like a rectangle than a square or a circle. Therefore, the heat map output by the target detection model trained based on the above method can accurately show the probability of the center point of the target object being distributed at each position. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0082] Figure 1This is a schematic diagram of a heat map mapped based on a circular Gaussian kernel function in this specification;

[0083] Figure 2 A schematic diagram of the structure of a target detection model provided in this manual;

[0084] Figure 3 A flowchart of a model training method provided in this manual;

[0085] Figure 4 A schematic diagram of a model training device provided in this specification;

[0086] Figure 5 A schematic diagram of a target detection device provided in this specification;

[0087] Figure 6 This is a schematic diagram of the structure of the autonomous driving equipment provided in this manual. DETAILED DESCRIPTION

[0088] The present invention provides a target detection model, a model training method for training the target detection model, and a target detection method using the target detection model.

[0089] The purpose of target detection in the embodiments of this specification is to predict a detection frame for the target object, that is, to locate the position of the target object in the data to be detected using a detection frame with a relatively simple geometric shape.

[0090] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0091] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0092] First, the model structure diagram of the target detection model is shown in the embodiment of this specification. Figure 2 As shown, the target detection model includes a feature extraction layer, a first prediction subnet, a second prediction subnet, a third prediction subnet, and a fourth prediction subnet.

[0093] In practical applications, the feature extraction layer extracts the image features corresponding to the target image from the input data to be extracted, and then inputs the image features into the first prediction subnet, the second prediction subnet, the third prediction subnet, and the fourth prediction subnet respectively to obtain the output results of each prediction subnet, such as Figure 2 As shown, in one embodiment of the present specification, the outputs of each prediction sub-network are a predicted heat map, a predicted distance, a predicted deflection angle, and a predicted height. Then, based on the output of each prediction sub-network, a detection box predicted for the target object, i.e., a predicted detection box, can be determined.

[0094] The target object is the object to be located using the detection frame in the target detection method provided in this specification. Specifically, according to the application scenario of the target detection model, when the target detection model is used for face recognition, the target object can be the face of a person in the data to be extracted collected by the sensor. When the target detection model is used in the field of autonomous driving to detect obstacles around the autonomous driving equipment, the target object can be each obstacle in the data to be extracted collected by the sensor. The obstacles can be pedestrians, motor vehicles, non-motor vehicles, etc. This specification below takes the target object as an example of an obstacle to illustrate the target detection model, target detection method, and model training method provided in the embodiments of this specification.

[0095] The execution subject of the target detection method provided in this specification and the execution subject of the model training method provided in this specification can be the same or different. Taking the example that the execution subject of the target detection method provided in this specification and the execution subject of the model training method provided in this specification are different, the execution subject of the target detection method and the execution subject of the model training method can both be any existing server or electronic device. Furthermore, when the execution subject executing the target detection method is not the autonomous driving device itself, there is a communication connection between the execution subject executing the target detection method and the autonomous driving device. On this basis, for each of the execution subject of the target detection method and the execution subject of the model training method, when the execution subject is an electronic device, the execution subject can be any existing electronic device, such as a mobile phone, a laptop computer, a tablet computer, etc. When the execution subject is a server, the execution subject can be a cluster server, a distributed server, etc.

[0096] In one embodiment of this specification, the execution subject of the model training method is a server, and the execution subject of the target detection method is an autonomous driving device as an example for description.

[0097] Furthermore, the autonomous driving equipment described in this specification may include autonomous vehicles and vehicles with assisted driving functions. The autonomous driving equipment may be a delivery vehicle used in the delivery field.

[0098] The data to be extracted includes a target object. The data to be extracted may be data collected by a sensor. Depending on the type of sensor, the data type of the data to be extracted may also vary. For example, when the sensor is a detection device such as a radar, the data to be extracted may be point cloud data, while when the sensor is an image acquisition device, the data to be extracted may be image data. In one embodiment of this specification, the sensor that collects the data to be extracted may be a lidar. In another embodiment of this specification, the sensor that collects the data to be extracted may be a time of flight (TOF) camera. In one embodiment of this specification, the data to be extracted may be a target image collected by the image acquisition device.

[0099] In one embodiment of this specification, a two-dimensional detection box of a target object on a specified plane can be predicted based on the predicted heat map output by the first prediction subnet, the predicted distance output by the second prediction subnet, and the predicted deflection angle output by the third prediction subnet. Based on the two-dimensional detection box and the predicted height output by the fourth prediction subnet, a three-dimensional detection box of the target object predicted by the target detection model can be output. In one embodiment of this specification, the plane where the image coordinate system of the target image resides can be the specified plane.

[0100] Generally speaking, when the target detection model is used in the field of autonomous driving, the designated plane can be the road surface where the autonomous driving device is located. Furthermore, the target image can be a top-down view of the road surface where the autonomous driving device is located. Of course, any plane within the space where the autonomous driving device is located can also be used. In this case, the fourth prediction subnet needs to output a first height of the near end of the target object from the designated plane and a second height of the target object from the far end of the designated plane. Then, based on the difference between the second height and the first height and the predicted detection frame of the target object, a three-dimensional detection frame of the target object is determined. For the sake of simplicity, the following description will be based on the example of the road surface where the autonomous driving device is located as the designated plane.

[0101] In the embodiment of this specification, the two-dimensional detection box may be referred to as the predicted detection box of the target object.

[0102] Specifically, the first prediction subnetwork outputs a prediction heat map. In the prediction heat map, the heat value of each point on the target object represents the probability that the point is the center point of the target object. For each point on the target object in the prediction heat map, the greater the heat value of the point, the greater the probability that the point is the center point of the target object. Therefore, the point with a maximum heat value can be used as the key point of the target object. Of course, when for each point on the target object in the prediction heat map, the smaller the heat value of the point, the greater the probability that the point is the center point of the target object, the point with a minimum heat value can be used as the key point of the target object.

[0103] In one embodiment of the present specification, the key point of the target object may be the center point of the target object. In this case, a standard detection frame with a pre-set standard width and standard height may be used, and then the center point output by the first prediction subnet may be used as the center point of the standard detection frame to obtain the predicted detection frame of the target object.

[0104] In another embodiment of the present specification, the standard detection frame of the target object can also be determined based on the outputs of the first prediction subnet and the second prediction subnet. Specifically, the second prediction subnet outputs the predicted distance of the key point from the predicted detection frame of the target object in each specified direction. In one embodiment of the present specification, for each specified direction, the specified direction can correspond to an edge of the predicted detection frame of the target object and be perpendicular to the edge corresponding to the specified direction. Furthermore, the edge of the predicted detection frame corresponding to the specified direction is different from the edge of the predicted detection frame corresponding to each other specified direction.

[0105] In practical applications, a rectangular frame is usually used as the two-dimensional detection frame of the target object. In this case, the specified direction can be a direction perpendicular to the four sides of the rectangular frame, and the predicted distances can be the distances between the key points of the target object and the four sides of the rectangular frame. For the sake of brevity, the following part of this specification takes the two-dimensional detection frame of the target object as a rectangular frame as an example.

[0106] In one embodiment of this specification, when the key point of the target object is the center point of the target object, the predicted distances from opposite sides of the predicted detection box output by the second prediction subnet are the same. Therefore, in this case, the second prediction subnet may only output the width and height of the predicted detection box. Of course, in this embodiment of this specification, there is no restriction on whether the key point of the target object is the center point of the target object.

[0107] In addition, the target detection model also includes a third prediction subnet, which outputs the deflection angle of the target object detection frame relative to the specified direction as the predicted deflection angle of the target object. Then, the shape of the predicted detection frame of the target object can be determined according to any of the above methods to determine the predicted detection frame of the target object. The embodiment of this specification does not limit which direction the specified direction is. For example, the specified direction can be the direction of the centerline of the road where the autonomous driving device is located.

[0108] Furthermore, the predicted detection box of the target object may also be a three-dimensional detection box.

[0109] Specifically, the image features output by the feature extraction subnet can be used as the input of the fourth prediction subnet of the target detection model. Through the fourth prediction subnet, the height of the target object from the specified plane is output as the predicted height of the target object. Then, based on the two-dimensional detection frame of the target object determined by any of the above methods and the predicted height of the target object, the three-dimensional detection frame of the target object in the three-dimensional space is determined, and the determined three-dimensional detection frame is updated to the predicted detection frame of the target object.

[0110] Generally speaking, when the data to be detected only includes two-dimensional features, it is difficult to predict the three-dimensional detection frame of the target object. Therefore, in one embodiment of the present specification, the data to be detected can be three-dimensional data, for example, point cloud data, or image data containing depth data. Below, the embodiment of the present specification takes the data to be detected as point cloud data to be detected as an example to illustrate the target detection method provided in the embodiment of the present specification.

[0111] Specifically, point cloud data to be detected can be obtained, and then the space containing the point cloud data can be divided into a number of voxels. For each voxel, feature extraction is performed on the point cloud data in the voxel to obtain the extracted point cloud features of the voxel. Based on the extracted point cloud features of each voxel, the feature extraction subnet in the target detection model is used to determine the point cloud feature map on the specified plane as the image feature corresponding to the extracted target image. The voxel is a volume element, which represents a unit of segmentation in three-dimensional space.

[0112] The feature extraction subnet can be any existing machine learning model, for example, a deep residual network (ResNet), a multilayer perceptron (MLP), etc., and the embodiments of this specification do not limit this.

[0113] It can be seen that in this case, the image features are extracted from the point cloud type of data to be detected. Therefore, in one embodiment of the present specification, the two-dimensional detection frame can be directly determined in the specified plane of the space where the point cloud data is located using any of the above methods, and then the three-dimensional detection frame of the target object in the space where the point cloud data is located can be determined based on the prediction of the target object output by the fourth prediction subnet. That is, there is no need to project the point cloud data to obtain the target image.

[0114] Instead of projecting the data to be detected onto an image obtained by projecting it onto a certain plane, the data to be detected is directly annotated to obtain the annotation center point and annotation detection frame of the target object, etc. Then, each annotation is projected onto a specified plane using any of the above methods to obtain the annotation on the specified plane. At this time, the target object contained in the target image can refer to the area within the annotation detection frame in the target image.

[0115] Before the above-mentioned target detection model is actually applied, in order to ensure the accuracy of the detection results output by the target detection model, the target detection model needs to be trained. Of course, after the target detection model has been applied for a period of time, the accuracy of the detection results output by the target detection model may be poor due to the differences between the newly added data and the samples used during training. At this time, the target detection model can also be trained.

[0116] It can be seen that any target detection model that needs to be trained can be the target detection model in the embodiments of this specification. The following embodiments of this specification provide a model training method for training the target detection model to be trained.

[0117] First, you can mark the sample graphics.

[0118] Similar to the application phase, the sample image can also be an image containing the target object pre-captured by the sensor. During the training phase, the sample image can be pre-annotated with the target object's annotation center point. Then, the annotation center point determined for the target object contained in the sample image can be obtained. Based on the annotation center point and a specified elliptical Gaussian kernel function, each point contained in the sample image can be mapped to the annotation heat map to be annotated, thereby obtaining the annotated annotation heat map of the sample image.

[0119] In the embodiments of this specification, the specific parameters contained in the elliptical Gaussian kernel function are not limited, but it should be noted that, in the annotated heat map obtained by projecting the elliptical Gaussian kernel function, the heat value of the point projected from the annotated center point of the target object in the sample image to the annotated heat map is greater than the heat value of the point projected from other points on the target object in the sample image to the annotated heat map. In addition, the heat values ​​of other points around the point projected from the annotated center point of the target object in the sample image in the annotated heat map decrease as the distance between the point and the point projected from the annotated center point increases, and along the circumferential direction from the point projected from the annotated center point to the ellipse, the smaller the angle with the short axis of the ellipse, the greater the change in the heat value of each point.

[0120] Then, this specification embodiment provides a Figure 3 The model training method flow chart shown in the figure specifically includes the following steps:

[0121] S300: Determine a sample image to be detected.

[0122] S302: Determine image features corresponding to the sample image to be detected through a feature extraction subnet in the target detection model to be trained.

[0123] Generally speaking, in order to improve the accuracy of the model output, it is ensured that the data contained in the sample set in the model training phase and the test set in the actual application phase are similar. Therefore, in the embodiment of this specification, the sample image is similar to the target image in the application phase. Specifically, the image coordinate system of the target image can also be a specified plane. Furthermore, the specified plane can also be a road surface, and the target image is a top view of the road surface, and so on.

[0124] The sample images with corresponding annotations can be used as sample images to be detected. The feature extraction subnet in the object detection model to be trained can then be used to determine the image features corresponding to the sample images to be detected. Similar to the application phase, the image features can also be extracted from the data to be detected. The following explanation assumes that the data to be detected is the sample images.

[0125] Furthermore, during the training phase, when the data to be detected is point cloud data, this specification provides an exemplary annotation method:

[0126] The target objects are labeled with their center points and 3D annotation boxes within the 3D space of the point cloud data. The labeled center points and 3D annotation boxes are then projected onto a specified plane in the 3D space to obtain the target objects' center points and 2D annotation boxes on the specified plane. In other words, the target image on the specified plane only includes the center points and 2D annotation boxes, excluding the target objects themselves. At this point, points within the 2D annotation boxes are considered to be located on the target objects.

[0127] Similar to the application stage, the feature extraction subnet can be any existing machine learning model, for example, a deep residual network (ResNet), a multilayer perceptron (MLP), etc., and the embodiments of this specification do not limit this.

[0128] S304: Using the image features output by the feature extraction subnet as input to a first prediction subnet of a target detection model, and outputting a predicted heat map of the sample image through the first prediction subnet.

[0129] The image features output by the feature extraction subnet are used as the input of the first prediction subnet of the target detection model. Through the first prediction subnet, a predicted heat map of the sample image is output. In the predicted heat map, the heat value of each point on the target object represents the probability that the point is the center point of the target object. For each point on the target object in the predicted heat map, the greater the heat value of the point, the greater the probability that the point is the center point of the target object. Therefore, the point with the maximum heat value can be used as the key point of the target object. Of course, when for each point on the target object in the predicted heat map, the smaller the heat value of the point, the greater the probability that the point is the center point of the target object, the point with the minimum heat value can be used as the key point of the target object.

[0130] S306: Obtain a pre-determined annotated heat map for the sample image, and adjust parameters in the target detection model with the goal of minimizing the difference between the predicted heat map and the annotated heat map of the sample image.

[0131] Then, the parameters in the object detection model are adjusted with the goal of minimizing the difference between the predicted heat map and the annotated heat map of the sample image.

[0132] In one embodiment of the present specification, the sample image, the annotated heat map, and the predicted heat map may all be set to have the same image size and the same number of pixels.

[0133] Specifically, the difference between each pair of points at corresponding locations in the predicted heatmap and the annotated heatmap can be determined, and then the parameters in the object detection model can be adjusted with the goal of minimizing the difference between each pair of points. Furthermore, parameters can be set for the difference between each pair of points, such that the closer the point pair is to the point projected from the annotated center point in the annotated heatmap, the higher the weight of the point pair.

[0134] It can be understood that during the training process, since the predicted heat map is supervised by the labeled heat map mapped based on the elliptical Gaussian kernel function, the heat values ​​of each point on the target object in the output predicted heat map are distributed in an elliptical shape, and the closer the point is to the center of the ellipse, the higher the heat value, and the farther the point is from the center of the ellipse, the lower the heat value.

[0135] Based on the above Figure 3 According to the method described, after the target detection model is trained, the heat values ​​of each point on the target object in the predicted heat map output by the target detection model are distributed in an elliptical shape. The closer the point is to the center of the ellipse, the higher the heat value, and the farther the point is from the center of the ellipse, the lower the heat value. In the real world, the shape of a vehicle is more similar to a rectangle than a square or a circle. Therefore, the heat map output by the target detection model trained based on the above method can accurately show the distribution probability of the center point of the target object at each position.

[0136] Then, based on the predicted heat map of the sample image, several predicted key points of the target object in the predicted heat map can be determined. The image features output by the feature extraction subnet are input into the second prediction subnet of the target detection model. The second prediction subnet outputs, for each predicted key point, the predicted distance between the predicted key point and the predicted detection box of the target object in each specified direction as the predicted distances corresponding to the predicted key point. Then, for each predicted key point, based on the predicted key point and the predicted distances corresponding to the predicted key point, the predicted detection box predicted for the target object based on the predicted key point is determined.

[0137] In the embodiment of this specification, during pre-annotation, a label detection frame is annotated in the sample image. Then, after projection based on the elliptical Gaussian kernel function in the above manner, the obtained label heat map includes the label detection frame.

[0138] It can be understood that the parameters in the object detection model are adjusted with the goal of minimizing the difference between each predicted detection box and the labeled detection box.

[0139] For example, a detection frame set consisting of the predicted detection frames and the annotated detection frames can be determined. Then, for each detection frame in the detection frame set, the designated corner points of the detection frame are determined. Parameters in the object detection model are adjusted with the goal of minimizing the difference between the corresponding designated corner points of each predicted detection frame and the annotated detection frame. The number of designated corner points is at least two. In this embodiment of the present specification, the corner points are the connection points between the line segments that constitute the annotated detection frame.

[0140] In one embodiment of the present specification, the heat value of each point on the target object in the predicted heat map can be determined, and points with heat values ​​greater than a preset heat threshold are selected from the points on the target object as predicted key points on the target object.

[0141] Similarly, the sample image may also be annotated with a deflection angle of the target object, where the deflection angle may indicate a deflection angle of the detection frame of the target object relative to a specified direction. In one embodiment of this specification, the specified direction may be the centerline direction of the road on which the autonomous driving device is located.

[0142] Similar to the application process, the image features output by the feature extraction subnet can be used as the input of the third prediction subnet of the target detection model. Through the third prediction subnet, the deflection angle of the detection box of the target object relative to the specified direction is predicted as the predicted deflection angle of the target object. Then, the parameters in the target detection model can be adjusted with the goal of minimizing the difference between the predicted deflection angle and the labeled deflection angle of the target object.

[0143] It should be noted that the above description only applies to the data to be detected being a sample image. In another embodiment of this specification, the data to be detected may be other data collected by a sensor, for example, point cloud data. In this case, the sample image may be an image obtained by projecting the data to be detected onto a plane. Alternatively, the data to be detected may be directly annotated to obtain the target object's annotated center point and annotated detection frame, etc. Each annotation is then projected onto a specified plane using any of the above methods to obtain annotations on the specified plane. In this case, the target object contained in the target image may refer to the area within the annotated detection frame in the target image.

[0144] In this case, similar to the application process, any of the above methods can be used to extract the image features corresponding to the sample object from the point cloud data to be detected, and the embodiments of this specification will not be repeated.

[0145] Of course, when the data to be detected is point cloud data, the height of the target object can also be marked, that is, the distance of the target object relative to the specified plane. In the embodiment of this specification, the marked height of the target object can be used as the marked height of the target object. Then, any of the above methods can be adopted to implement it, and the image features output by the feature extraction subnet are used as the input of the fourth prediction subnet of the target detection model. Through the fourth prediction subnet, the detection box height of the target object is predicted as the predicted height of the target object, and the parameters in the target detection model are adjusted with the goal of minimizing the difference between the predicted height and the marked height of the target object.

[0146] The above is a model training method provided in one or more embodiments of this specification. Based on the same idea, this specification also provides corresponding model training devices and target detection devices.

[0147] Figure 4 This is a schematic diagram of a model training device provided in this specification, which includes:

[0148] An image determination module 400 is used to determine a sample image to be detected;

[0149] A feature extraction module 402 is configured to determine image features corresponding to the sample image to be detected by using a feature extraction subnet in the object detection model to be trained;

[0150] a heat map prediction module 404 configured to use the image features output by the feature extraction subnet as input to a first prediction subnet of an object detection model, and output a predicted heat map of the sample image through the first prediction subnet;

[0151] a parameter adjustment module 406 for obtaining a pre-determined annotated heat map for the sample image and adjusting parameters in the object detection model with the goal of minimizing the difference between the predicted heat map and the annotated heat map of the sample image;

[0152] The labeling module 408 is used to pre-determine a labeling heat map for the sample image, specifically including: obtaining a labeling center point determined for the target object contained in the sample image, and mapping each point contained in the sample image to the labeling heat map to be labeled based on the labeling center point and a specified elliptical Gaussian kernel function, to obtain the labeled labeling heat map of the labeled sample image.

[0153] Optionally, the parameter adjustment module 406 is specifically used to determine a number of predicted key points of the target object in the predicted heat map according to the predicted heat map of the sample image; input the image features output by the feature extraction subnet into the second prediction subnet of the target detection model, and output the predicted distance of the predicted key point from the predicted detection box of the target object in each specified direction through the second prediction subnet for each predicted key point as each predicted distance corresponding to the predicted key point; for each predicted key point, determine the predicted distance of the target object based on the predicted key point according to the predicted key point and the predicted distances corresponding to the predicted key point. A predicted detection frame of a key point; obtaining a predetermined labeled detection frame of the target object in the labeled heat map, and adjusting the parameters in the target detection model with the goal of minimizing the difference between each predicted detection frame and the labeled detection frame; the labeling module 408 is specifically used to obtain a labeled detection frame determined for the target object contained in the sample image, and a labeled center point determined for the target object contained in the sample image, and based on the labeled center point and a specified elliptical Gaussian kernel function, map the labeled detection frame contained in the sample image to the labeled heat map to be labeled, so as to obtain the labeled detection frame of the target object in the labeled heat map.

[0154] Optionally, the parameter adjustment module 406 is specifically used to determine the heat value of each point on the target object in the predicted heat map, wherein, for each point on the target object in the predicted heat map, the greater the heat value of the point, the greater the probability that the point is the center point of the target object; and select points with heat values ​​greater than a pre-set heat threshold from each point on the target object as predicted key points on the target object.

[0155] Optionally, the parameter adjustment module 406 is specifically used to determine a detection frame set consisting of each predicted detection frame and the labeled detection frame; for each detection frame in the detection frame set, determine each designated corner point of the detection frame, and adjust the parameters in the target detection model with the goal of minimizing the difference between the corresponding designated corner points of each predicted detection frame and the labeled detection frame, wherein the number of the designated corner points is at least two.

[0156] Optionally, the parameter adjustment module 406 is specifically used to use the image features output by the feature extraction subnet as the input of the third prediction subnet of the target detection model, and predict the deflection angle of the detection box of the target object relative to the specified direction through the third prediction subnet as the predicted deflection angle of the target object; obtain the determined annotated deflection angle of the target object, and adjust the parameters in the target detection model with the goal of minimizing the difference between the predicted deflection angle and the annotated deflection angle of the target object.

[0157] Optionally, before determining the sample image to be detected, the image determination module 400 is specifically used to obtain the point cloud data to be detected; project the point cloud data to a specified plane in the space where the point cloud data is located, and obtain the projection image on the specified plane as the sample image to be detected; the feature extraction module 402 is specifically used to divide the space where the point cloud data is located into a number of voxels, and for each voxel, perform feature extraction on the point cloud data in the voxel to obtain the extracted point cloud features in the voxel; based on the extracted point cloud features of each voxel, the feature extraction subnet in the target detection model to be trained is used to determine the point cloud feature map on the specified plane as the image feature corresponding to the determined sample image to be detected.

[0158] Optionally, the parameter adjustment module 406 is specifically used to use the image features output by the feature extraction subnet as the input of the fourth prediction subnet of the target detection model, and predict the detection box height of the target object through the fourth prediction subnet as the predicted height of the target object; obtain the determined labeled height of the target object, and adjust the parameters in the target detection model with the goal of minimizing the difference between the predicted height and the labeled height of the target object.

[0159] Figure 5 This is a schematic diagram of a target detection device provided in this specification, which includes:

[0160] An image determination module 500 is used to determine a target image to be detected;

[0161] A feature extraction module 502 is configured to determine image features corresponding to the target image using a feature extraction subnet in the target detection model trained using any of the above methods;

[0162] a heat map prediction module 504 configured to use the image features output by the feature extraction subnet as input to a first prediction subnet of a target detection model, and output a predicted heat map of the target image through the first prediction subnet, wherein, for each point on the target object in the predicted heat map, the greater the heat value of the point, the greater the probability that the point is the center point of the target object;

[0163] The target detection module 506 is configured to output a predicted detection frame of the target object based on the predicted heat map of the target image.

[0164] Optionally, the target detection module 506 is specifically used to determine the heat value of each point in the predicted heat map, and use the point with the extreme heat value on the target object as the predicted key point of the target object; use the image features output by the feature extraction subnet as the input of the second prediction subnet of the target detection model, and output the predicted distance of the predicted key point of the target object from the predicted detection box of the target object in each specified direction through the second prediction subnet; determine the predicted detection box of the target object based on the predicted key point of the target object and the predicted distance of the predicted key point of the target object from the predicted detection box of the target object in each specified direction.

[0165] Optionally, the target detection module 506 is specifically used to determine the shape of the predicted detection box of the target object based on the predicted heat map of the target image; use the image features output by the feature extraction subnet as the input of the third prediction subnet of the target detection model, and output the deflection angle of the detection box of the target object relative to the specified direction through the third prediction subnet as the predicted deflection angle of the target object; output the predicted detection box of the target object based on the shape of the predicted detection box of the target object and the predicted deflection angle of the target object.

[0166] Optionally, the feature extraction module 502 is specifically used to obtain point cloud data to be detected; divide the space where the point cloud data is located into a number of voxels, and for each voxel, perform feature extraction on the point cloud data in the voxel to obtain the extracted point cloud features of the voxel; based on the extracted point cloud features of each voxel, determine the point cloud feature map on the specified plane through the feature extraction subnet in the target detection model as the image feature corresponding to the extracted target image; the target detection module 506 is specifically used to output a three-dimensional detection box of the target object contained in the target image in the three-dimensional space according to the predicted heat map of the target image.

[0167] Optionally, the target detection module 506 is specifically used to determine the two-dimensional detection box of the target object in the target image according to the predicted heat map of the target image, and use it as the predicted detection box of the target object; use the image features output by the feature extraction subnet as the input of the fourth prediction subnet of the target detection model, and output the height of the target object from the specified plane through the fourth prediction subnet as the predicted height of the target object; determine the three-dimensional detection box of the target object in the three-dimensional space according to the two-dimensional detection box of the target object in the target image and the predicted height of the target object, and update the determined three-dimensional detection box to the predicted detection box of the target object.

[0168] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above-mentioned model training and target detection methods.

[0169] This manual also provides Figure 6 The structural diagram of the electronic device shown in FIG. Figure 6 As shown in the figure, at the hardware level, the autonomous driving device includes a processor, an internal bus, memory, and non-volatile storage, and may also include hardware required for other operations. The processor reads the corresponding computer program from the non-volatile storage into the memory and then runs it to implement the above-mentioned model training and target detection methods.

[0170] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0171] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0172] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0173] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0174] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0175] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0176] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0177] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0178] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0179] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0180] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0181] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0182] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0183] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0184] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0185] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0186] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A model training method, characterized in that: include: Determining a sample image to be detected; Determining image features corresponding to the sample image to be detected by using a feature extraction subnet in the target detection model to be trained; Using the image features output by the feature extraction subnet as input to a first prediction subnet of an object detection model, and outputting a predicted heat map of the sample image through the first prediction subnet; Obtaining a pre-determined annotated heat map for the sample image, and adjusting parameters in the object detection model with the goal of minimizing the difference between the predicted heat map and the annotated heat map of the sample image; The step of pre-determining a labeling heat map for the sample image specifically includes: A labeling center point determined for a target object contained in the sample image is obtained, and based on the labeling center point and a specified elliptical Gaussian kernel function, each point contained in the sample image is mapped to a labeling heat map to be labeled, so as to obtain a labeling heat map of the sample image, wherein the target object includes a vehicle, and the heat values ​​of other points around the point to which the labeling center point of the target object in the sample image is projected decrease as the distance between the point and the point to which the labeling center point is projected increases, and along the circumferential direction from the point to which the labeling center point is projected to the ellipse, the smaller the angle with the minor axis of the ellipse is, the greater the change amplitude of the heat value of each point is.

2. The training method according to claim 1, wherein: The method further comprises: Determining, based on the predicted heat map of the sample image, a plurality of predicted key points of the target object in the predicted heat map; Inputting the image features output by the feature extraction subnet into the second prediction subnet of the object detection model, and outputting, through the second prediction subnet, for each predicted key point, the predicted distance between the predicted key point and the predicted detection box of the target object in each specified direction as each predicted distance corresponding to the predicted key point; For each predicted key point, determining a predicted detection box predicted for the target object based on the predicted key point according to the predicted key point and the predicted distances corresponding to the predicted key point; Obtaining a predetermined annotated detection frame of the target object in the annotated heat map, and adjusting parameters in the target detection model with the goal of minimizing the difference between each predicted detection frame and the annotated detection frame; Predetermining the label detection box of the target object in the label heat map specifically includes: Obtain a labeled detection box determined for the target object contained in the sample image, and a labeled center point determined for the target object contained in the sample image; based on the labeled center point and a specified elliptical Gaussian kernel function, map the labeled detection box contained in the sample image to the labeled heat map to be labeled, and obtain the labeled detection box of the target object in the labeled heat map.

3. The method according to claim 2, wherein Determining, based on the predicted heat map of the sample image, a number of predicted key points of the target object in the predicted heat map, specifically comprising: Determining a heat value of each point on the target object in the predicted heat map, wherein, for each point on the target object in the predicted heat map, a greater heat value of the point indicates a greater probability that the point is the center point of the target object; Points with heat values ​​greater than a preset heat threshold are selected from various points on the target object as predicted key points on the target object.

4. The method according to claim 2, wherein With the goal of minimizing the difference between each predicted detection box and the labeled detection box, adjusting the parameters in the object detection model includes: Determine a detection frame set consisting of the predicted detection frames and the labeled detection frames; For each detection box in the detection box set, determine each designated corner point of the detection box, and adjust parameters in the object detection model with the goal of minimizing the difference between the corresponding designated corner points of each predicted detection box and the labeled detection box, wherein the number of the designated corner points is at least two.

5. The method according to claim 1, wherein The method further comprises: Using the image features output by the feature extraction subnet as input to a third prediction subnet of the target detection model, and predicting the deflection angle of the detection box of the target object relative to a specified direction through the third prediction subnet as the predicted deflection angle of the target object; The determined annotated deflection angle of the target object is obtained, and parameters in the target detection model are adjusted with the goal of minimizing the difference between the predicted deflection angle and the annotated deflection angle of the target object.

6. The method according to claim 1, wherein Before determining the sample image to be detected, the method further includes: Obtain the point cloud data to be detected; Determine the sample image to be detected, including: Projecting the point cloud data onto a specified plane in the space where the point cloud data is located, and obtaining a projection image on the specified plane as a sample image to be detected; Determine the image features corresponding to the sample image to be detected by using the feature extraction subnet in the target detection model to be trained, specifically including: Dividing the space where the point cloud data is located into a plurality of voxels, and performing feature extraction on the point cloud data in each voxel to obtain the extracted point cloud features in the voxel; According to the extracted point cloud features of each voxel, a point cloud feature map on the specified plane is determined through the feature extraction subnet in the target detection model to be trained, as the determined image feature corresponding to the sample image to be detected.

7. The method according to claim 6, wherein The method further comprises: Using the image features output by the feature extraction subnet as input to a fourth prediction subnet of the target detection model, and predicting the height of the detection box of the target object through the fourth prediction subnet as the predicted height of the target object; The determined annotated height of the target object is obtained, and parameters in the target detection model are adjusted with the goal of minimizing the difference between the predicted height and the annotated height of the target object.

8. A target detection method, characterized in that: include: Determine the target image to be detected; Determining image features corresponding to the target image using a feature extraction subnet in a target detection model trained using the method according to any one of claims 1 to 7; The image features output by the feature extraction subnet are used as input to a first prediction subnet of a target detection model. The first prediction subnet outputs a predicted heat map of the target image, wherein, for each point on the target object in the predicted heat map, a greater heat value of the point indicates a greater probability that the point is the center point of the target object. Based on the predicted heat map of the target image, the predicted detection box of the target object is output.

9. The method according to claim 8, wherein Outputting a predicted detection box of the target object based on the predicted heat map of the target image specifically includes: Determine the heat value of each point in the prediction heat map, and use the point with the extreme heat value on the target object as the prediction key point of the target object; Using the image features output by the feature extraction subnet as input to a second prediction subnet of the target detection model, and outputting, through the second prediction subnet, a predicted distance between the predicted key point of the target object and the predicted detection box of the target object in each specified direction; The predicted detection frame of the target object is determined according to the predicted key points of the target object and the predicted distances of the predicted key points of the target object from the predicted detection frame of the target object in each specified direction.

10. The method according to claim 8, wherein Outputting a predicted detection box of the target object based on the predicted heat map of the target image specifically includes: Determining a shape of a predicted detection box of the target object according to a predicted heat map of the target image; Using the image features output by the feature extraction subnet as input to a third prediction subnet of the target detection model, and outputting a deflection angle of the detection frame of the target object relative to a specified direction as a predicted deflection angle of the target object through the third prediction subnet; Outputting the predicted detection box of the target object according to the shape of the predicted detection box of the target object and the predicted deflection angle of the target object.

11. The method according to claim 8, wherein Extracting image features corresponding to the target image, specifically including: Obtain the point cloud data to be detected; Dividing the space where the point cloud data is located into a plurality of voxels, and performing feature extraction on the point cloud data in each voxel to obtain the extracted point cloud features in the voxel; According to the extracted point cloud features of each voxel, a point cloud feature map on the specified plane is determined by a feature extraction subnet in the target detection model as the extracted image features corresponding to the target image; Outputting a predicted detection box of the target object based on the predicted heat map of the target image specifically includes: According to the predicted heat map of the target image, a three-dimensional detection box of the target object contained in the target image in the three-dimensional space is output.

12. The method according to claim 11, wherein Outputting a three-dimensional detection box of a target object contained in the target image in three-dimensional space according to the predicted heat map of the target image specifically includes: Determine a two-dimensional detection frame of the target object in the target image according to the predicted heat map of the target image, and use the two-dimensional detection frame as the predicted detection frame of the target object; Using the image features output by the feature extraction subnet as input to a fourth prediction subnet of the target detection model, and outputting the height of the target object from the specified plane as the predicted height of the target object through the fourth prediction subnet; A three-dimensional detection frame of the target object in three-dimensional space is determined based on the two-dimensional detection frame of the target object in the target image and the predicted height of the target object, and the determined three-dimensional detection frame is updated as the predicted detection frame of the target object.

13. A model training device, characterized in that: The device specifically includes: An image determination module, used to determine a sample image to be detected; A feature extraction module is used to determine the image features corresponding to the sample image to be detected through the feature extraction subnet in the target detection model to be trained; a heat map prediction module, configured to use the image features output by the feature extraction subnet as input to a first prediction subnet of an object detection model, and output a predicted heat map of the sample image through the first prediction subnet; a parameter adjustment module, configured to obtain a pre-determined annotated heat map for the sample image, and adjust parameters in the object detection model with the goal of minimizing the difference between the predicted heat map and the annotated heat map of the sample image; The labeling module is used to pre-determine a labeling heat map for the sample image, specifically including: obtaining a labeling center point determined for the target object contained in the sample image, and mapping each point contained in the sample image to the labeling heat map to be labeled based on the labeling center point and a specified elliptical Gaussian kernel function, to obtain the labeled labeling heat map of the labeled sample image, wherein the target object includes a vehicle, and the heat values ​​of other points around the point to which the labeling center point of the target object in the sample image is projected decrease as the distance between the point and the point to which the labeling center point is projected increases, and along the circumferential direction from the point to which the labeling center point is projected to the ellipse, the smaller the angle with the short axis of the ellipse, the greater the change in the heat value of each point.

14. A target detection device, characterized in that: The device specifically includes: An image determination module, used to determine a target image to be detected; A feature extraction module, configured to determine image features corresponding to the target image using a feature extraction subnet in a target detection model trained by the method according to any one of claims 1 to 7; a heat map prediction module, configured to use the image features output by the feature extraction subnet as input to a first prediction subnet of a target detection model, and output a predicted heat map of the target image through the first prediction subnet, wherein, for each point on the target object in the predicted heat map, the greater the heat value of the point, the greater the probability that the point is the center point of the target object; The target detection module is used to output the predicted detection box of the target object based on the predicted heat map of the target image.

15. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

17. An autonomous driving device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 8 to 12 is implemented.

Images