Three-dimensional target detection model training method, three-dimensional target detection method and equipment

Abstract

The invention provides a three-dimensional target detection model training method, a three-dimensional target detection method and equipment, and the method comprises the steps: obtaining a sample image shot by a fisheye camera, and marking the three-dimensional attribute marking information of a to-be-detected target, comprising a first direction angle, a first division area to which the first direction angle belongs and a first deviation angle in the first division area. And inputting the sample image into the initial detection model, and obtaining three-dimensional attribute prediction information of the to-be-detected target, including a second direction angle, a second division area to which the second direction angle belongs and a second deviation angle in the second division area. And determining a loss function value based on the three-dimensional attribute prediction information and the three-dimensional attribute labeling information. And training the initial detection model based on the loss function value to obtain a three-dimensional target detection model. By marking the first division area to which the first direction angle belongs, the error of the predicted second direction angle is reduced, and the accuracy of the three-dimensional target detection model is improved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and specifically to a three-dimensional target detection model training method, a three-dimensional target detection method and a device. Background Art

[0002] During the automatic parking process, it is necessary to be able to accurately sense and identify obstacles around the vehicle and determine the distance and direction of these obstacles relative to the vehicle, so as to control the vehicle to avoid obstacles and achieve automatic parking.

[0003] The target detection currently used in the field of automatic parking is usually a two-dimensional (2D) target detection algorithm, which uses a camera to obtain images around the vehicle and identifies obstacles in the image by processing the image. However, the 2D target detection algorithm can only identify objects within the two-dimensional image plane and obtain a 2D detection frame for identifying the object. However, when the object is deflected at an angle, the 2D detection frame cannot fit the object tightly. For example, when the vehicle has a horizontal deflection angle, the 2D detection frame cannot fit the edge of the vehicle tightly, resulting in large errors in distance, direction, etc. obtained based on the post-processing of the 2D detection frame, affecting the accuracy of target detection, thereby affecting the accuracy and safety of the subsequent automatic parking process. Summary of the invention

[0004] In view of this, the present application is committed to providing a three-dimensional target detection model training method, a three-dimensional target detection method and a device to improve the accuracy of target detection.

[0005] In a first aspect, the present application provides a method for training a three-dimensional object detection model, the method comprising:

[0006] Acquire a sample image taken by a fisheye camera, wherein the sample image is annotated with three-dimensional attribute annotation information of a target to be detected, wherein the three-dimensional attribute annotation information includes a first direction angle, a first divided area to which the first direction angle belongs, and a first deviation angle within the first divided area;

[0007] Inputting the sample image into an initial detection model to obtain three-dimensional attribute prediction information of the target to be detected, wherein the three-dimensional attribute prediction information includes a second direction angle, a second divided area to which the second direction angle belongs, and a second deviation angle within the second divided area;

[0008] Determining a loss function value based on the three-dimensional attribute prediction information and the three-dimensional attribute annotation information;

[0009] The initial detection model is trained based on the loss function value until a cutoff condition is met and the training process is stopped to obtain a trained three-dimensional object detection model.

[0010] In a possible implementation manner, determining the first divided area to which the first direction angle belongs includes:

[0011] Divide the range interval to which the direction angle belongs to obtain multiple divided areas;

[0012] Based on the direction angle range corresponding to each divided area, the first divided area to which the first direction angle belongs is determined.

[0013] In a possible implementation, the three-dimensional attribute annotation information further includes: a first center point, a first three-dimensional size, and a depth value corresponding to the target to be detected, and the three-dimensional attribute prediction information further includes: a second center point and a second three-dimensional size;

[0014] The determining of the loss function value based on the three-dimensional attribute prediction information and the three-dimensional attribute labeling information includes:

[0015] Determining a position weight corresponding to the depth value based on a position weight threshold and a relationship between the depth value and the depth threshold, the position weight threshold corresponding to the depth threshold;

[0016] Determine a first loss function value based on the first center point, the second center point, and the position weight;

[0017] Determining a second loss function value based on the first three-dimensional size, the second three-dimensional size, and the position weight;

[0018] Determine a third loss function value based on the first direction angle, the second direction angle, and the position weight;

[0019] Determine a fourth loss function value based on the first divided area and the second divided area;

[0020] A fifth loss function value is determined based on the first deviation angle and the second deviation angle.

[0021] In a second aspect, the present application provides a three-dimensional target detection method, which is applied to detecting obstacles around a vehicle in a parking scenario, and the method comprises:

[0022] Acquire image data captured by a fisheye camera on a vehicle;

[0023] Inputting the image data into the three-dimensional object detection model described in any one of the implementations of the first aspect above to obtain three-dimensional attribute information of the object to be detected in the image data;

[0024] Based on the three-dimensional attribute information, a three-dimensional detection result of the target to be detected is obtained.

[0025] In a possible implementation, the three-dimensional object detection model includes a backbone network, a feature fusion network, and a detection head. The image data is input into the three-dimensional object detection model to obtain three-dimensional attribute information of the object to be detected, including:

[0026] Inputting the image data into the backbone network to obtain image features of multiple different scales;

[0027] Based on the feature fusion network, the image features of the multiple different scales are fused to obtain fused image features;

[0028] Inputting the fused image features into the detection head to obtain a central point heat map of the target to be detected;

[0029] The three-dimensional attribute information is obtained based on the center point heat map regression.

[0030] In a possible implementation, the image data includes image data respectively captured by four fisheye cameras at the same time;

[0031] Inputting the image data into a three-dimensional object detection model to obtain three-dimensional attribute information of the object to be detected includes:

[0032] The image data captured by each fisheye camera is input into the three-dimensional object detection model to obtain the three-dimensional attribute information of the target to be detected corresponding to each camera.

[0033] In a possible implementation manner, the three-dimensional attribute information includes category confidence, and obtaining the three-dimensional detection result of the target to be detected based on the three-dimensional attribute information includes:

[0034] Based on the three-dimensional attribute information of the target to be detected corresponding to each camera, an initial three-dimensional detection result is obtained;

[0035] Calculate the overlap between the initial 3D detection results corresponding to two adjacent cameras;

[0036] When the overlap is greater than a preset overlap, initial three-dimensional detection results with lower category confidence are deleted to obtain remaining three-dimensional detection results.

[0037] In a possible implementation, the three-dimensional attribute information includes a center point, a three-dimensional size, a divided area, and a deviation angle;

[0038] The obtaining of a three-dimensional detection result of the target to be detected based on the three-dimensional attribute information includes:

[0039] Determining the direction angle of the target to be detected based on the direction angle range corresponding to the divided area and the deviation angle;

[0040] Determine a three-dimensional bounding box of the target to be detected based on the center point, the three-dimensional size, and the direction angle;

[0041] The three-dimensional bounding box is mapped to a preset type of image coordinate system through coordinate system conversion to obtain the three-dimensional detection result.

[0042] In a possible implementation manner, the three-dimensional attribute information includes category confidence, and obtaining the three-dimensional detection result of the target to be detected based on the three-dimensional attribute information includes:

[0043] Comparing the category confidence in the three-dimensional attribute information with a confidence threshold, and filtering to obtain the target three-dimensional attribute information, wherein the category confidence represents the accuracy rate of the target to be detected being a predicted category, and the category confidence included in the target three-dimensional attribute information is greater than the confidence threshold;

[0044] Based on the target three-dimensional attribute information, the three-dimensional detection result is obtained.

[0045] In a third aspect, the present application provides a training device for a three-dimensional object detection model, the device comprising:

[0046] A sample acquisition module, used to obtain a sample image taken by a fisheye camera, wherein the sample image is annotated with three-dimensional attribute annotation information of a target to be detected, wherein the three-dimensional attribute annotation information includes a first direction angle, a first divided area to which the first direction angle belongs, and a first deviation angle within the first divided area;

[0047] a detection module, configured to input the sample image into an initial detection model to obtain three-dimensional attribute prediction information of the target to be detected, wherein the three-dimensional attribute prediction information includes a second direction angle, a second divided area to which the second direction angle belongs, and a second deviation angle within the second divided area;

[0048] A loss function determination module, used to determine a loss function value based on the three-dimensional attribute prediction information and the three-dimensional attribute annotation information;

[0049] The training module is used to train the initial detection model based on the loss function value until the training process is stopped when a cutoff condition is met, thereby obtaining a trained three-dimensional object detection model.

[0050] In a fourth aspect, the present application provides a three-dimensional object detection device, the device comprising:

[0051] A data acquisition module, used to acquire image data captured by a fisheye camera;

[0052] A model detection module, used to input the image data into a three-dimensional object detection model to obtain three-dimensional attribute information of the object to be detected, wherein the three-dimensional object detection model includes a backbone network, a feature fusion network and a detection head, and the detection head realizes three-dimensional object detection based on a center point;

[0053] The three-dimensional detection module is used to obtain a three-dimensional detection result of the target to be detected based on the three-dimensional attribute information.

[0054] In a fifth aspect, the present application provides an electronic device, the device comprising: a memory and a processor;

[0055] The memory is used to store relevant program codes;

[0056] The processor is used to call the program code to execute the training method of the three-dimensional target detection model described in any implementation of the first aspect or the three-dimensional target detection method described in any implementation of the second aspect.

[0057] In a sixth aspect, the present application provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the training method of the three-dimensional target detection model described in any one of the implementations of the first aspect or the three-dimensional target detection method described in any one of the implementations of the second aspect.

[0058] In the seventh aspect, the present application provides a computer program product, which includes a computer program / instructions, and when the computer program / instructions are executed by a processor, it implements the training method of the three-dimensional target detection model described in any one of the implementations of the first aspect or the three-dimensional target detection method described in any one of the implementations of the second aspect.

[0059] In order to detect obstacles around the vehicle in the automatic parking scene, a three-dimensional target detection model is first pre-trained to enable it to detect three-dimensional targets. Specifically, a sample image taken by a fisheye camera is obtained, wherein the sample image is annotated with three-dimensional attribute annotation information of the target to be detected, and the three-dimensional attribute annotation information includes a first direction angle, a first divided area to which the first direction angle belongs, and a first deviation angle in the first divided area. The sample image is input into the initial detection model to obtain three-dimensional attribute prediction information of the target to be detected, and the three-dimensional attribute prediction information includes a second direction angle, a second divided area to which the second direction angle belongs, and a second deviation angle in the second divided area. Based on the three-dimensional attribute prediction information and the three-dimensional attribute annotation information, a loss function value is determined. The initial detection model is trained based on the loss function value until the training process is stopped when the cutoff condition is met, and a trained three-dimensional target detection model is obtained. The initial detection model is trained using the three-dimensional attribute annotation information of the sample image to enable it to detect three-dimensional targets. And by annotating the first divided area to which the first direction angle belongs, the error of the predicted second direction angle is reduced, thereby improving the accuracy of the three-dimensional target detection model.

[0060] In the above implementation of the present application, it can be applied to obstacle detection in automatic parking scenes. During the parking process, the image captured by the fisheye camera installed on the vehicle is obtained, and the image is subjected to three-dimensional target detection to detect obstacles around the vehicle, and the vehicle is controlled to avoid obstacles to complete automatic parking to prevent collisions. Specifically, the image data captured by the fisheye camera is obtained, and then the image data is input into the three-dimensional target detection model to obtain the three-dimensional attribute information of the target to be detected. Among them, the three-dimensional target detection model includes a backbone network, a feature fusion network and a detection head, and the detection head is based on the center point to achieve three-dimensional target detection. After obtaining the three-dimensional attribute information of the target to be detected, the three-dimensional detection result of the target to be detected is obtained based on the three-dimensional attribute information. That is, the positioning of the target to be detected is achieved. Through the method provided in the embodiment of the present application, the three-dimensional target detection model can be used to achieve three-dimensional detection of the target, and the accuracy of target detection can be improved. And the three-dimensional target detection model provided in the present application is based on the center point to achieve three-dimensional target detection, which can reduce the time of model prediction, reduce the computing resources occupied by the model, and is more suitable for low computing power chips. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. It is obvious that the drawings described below are only some embodiments provided in the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0062] Figure 1A flowchart of a method for training a three-dimensional object detection model provided in an embodiment of the present application.

[0063] Figure 2 A flowchart of a three-dimensional target detection method provided in an embodiment of the present application.

[0064] Figure 3a A schematic diagram of three-dimensional target detection in an image coordinate system provided in an embodiment of the present application.

[0065] Figure 3b A schematic diagram of three-dimensional target detection in a BEV coordinate system provided in an embodiment of the present application.

[0066] Figure 4 A flowchart of another three-dimensional target detection method provided in an embodiment of the present application.

[0067] Figure 5 A schematic diagram of a training device for a three-dimensional target detection model provided in an embodiment of the present application.

[0068] Figure 6 A schematic diagram of a three-dimensional target detection device provided in an embodiment of the present application.

[0069] Figure 7 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. The described embodiments are only exemplary implementation methods of the present application, not all implementation methods. Those skilled in the art can combine the embodiments of the present application to obtain other embodiments without creative work, and these embodiments are also within the scope of protection of the present application.

[0071] During the automatic parking process, it is necessary to be able to accurately sense and identify obstacles around the vehicle and determine the distance and direction of these obstacles relative to the vehicle, so as to control the vehicle to avoid obstacles and achieve automatic parking.

[0072] At present, the target detection used in the field of automatic parking is usually a two-dimensional (2D) target detection algorithm, which obtains images around the vehicle through a camera and identifies obstacles in the image through image processing. However, the 2D target detection algorithm can only identify objects in the two-dimensional image plane and obtain a 2D detection frame for identifying the object. However, when the object has an angle deflection, for example, when an object in a rectangular area (such as other vehicles) has a horizontal deflection angle, the 2D detection frame used to identify the object will be elongated, making it impossible for the 2D detection frame to fit the contour of the object tightly, resulting in large errors in the distance and direction relative to the vehicle obtained based on the post-processing of the 2D detection frame, affecting the accuracy of target detection, and thus affecting the accuracy and safety of the subsequent automatic parking process.

[0073] Based on this, an embodiment of the present application provides a training method for a three-dimensional target detection model to improve the accuracy of target detection. Specifically, a sample image taken by a fisheye camera is obtained, wherein the sample image is annotated with three-dimensional attribute annotation information of the target to be detected, and the three-dimensional attribute annotation information includes a first direction angle, a first divided area to which the first direction angle belongs, and a first deviation angle in the first divided area. The sample image is input into an initial detection model to obtain three-dimensional attribute prediction information of the target to be detected, and the three-dimensional attribute prediction information includes a second direction angle, a second divided area to which the second direction angle belongs, and a second deviation angle in the second divided area. Based on the three-dimensional attribute prediction information and the three-dimensional attribute annotation information, a loss function value is determined. The initial detection model is trained based on the loss function value until the training process is stopped when the cutoff condition is met, and a trained three-dimensional target detection model is obtained. The initial detection model is trained using the three-dimensional attribute annotation information of the sample image so that it has the ability to detect three-dimensional targets. And by annotating the first divided area to which the first direction angle belongs, the error of the predicted second direction angle is reduced, thereby improving the accuracy of the three-dimensional target detection model.

[0074] In order to facilitate the understanding of the technical solution provided by the embodiments of the present application, a detailed introduction will be given below with reference to the accompanying drawings.

[0075] See also Figure 1 As shown, Figure 1 A flowchart of a method for training a three-dimensional object detection model provided in an embodiment of the present application.

[0076] Optionally, the method may be performed by a training device, wherein the training device may be an electronic device or other device.

[0077] The method may include the following steps:

[0078] S101: Acquire a sample image taken by a fisheye camera.

[0079] Among them, the fisheye camera is a lens with a short focal length and a viewing angle of nearly 180 degrees, which is a type of ultra-wide-angle lens. Due to its extremely large viewing angle, the fisheye camera can provide a wider field of view than ordinary cameras, so the fisheye camera is also suitable for automatic parking scenarios. During the automatic parking process, the fisheye camera can cover a larger perception range, making up for the shortcomings of ordinary cameras in near-field perception, and helping to more accurately identify the vehicle's surroundings and obstacles. Therefore, sample images taken by fisheye cameras can be used to train the model.

[0080] In order to train the model to detect the target in the sample image and output the three-dimensional detection result, the three-dimensional attribute annotation information of the target to be detected can be pre-marked in the sample image. Among them, the target to be detected can represent obstacles around the vehicle, such as pedestrians, other vehicles, etc. The three-dimensional attribute annotation information may include the first direction angle of the target to be detected, the first divided area to which the first direction angle belongs, and the first deviation angle in the first divided area. The first direction angle can represent the deflection angle of the target to be detected from the horizontal direction.

[0081] In a possible implementation, in order to enable the initial detection model to more accurately predict the direction angle of the target to be detected, the range interval to which the direction angle belongs can be divided in advance, and the divided area to which the first direction angle belongs can be determined, so that the initial detection model can predict the divided area to which the direction angle belongs. In this way, the deviation range of the direction angle predicted by the initial detection model can be limited to a divided area, so as to improve the accuracy of the initial detection model.

[0082] In a specific implementation, the range interval to which the direction angle belongs is divided to obtain a plurality of divided areas. The range interval to which the direction angle belongs can be calibrated according to actual needs. Based on the direction angle range corresponding to each divided area, the first divided area to which the first direction angle belongs is determined.

[0083] Optionally, the range interval of the direction angle can be calibrated as [-180°, 180°], and then the range interval is equally divided into 8 divided areas, that is, the direction angle range corresponding to each divided area is 45°. Then 8 divided areas can be obtained, and the range intervals corresponding to the 8 divided areas are [-180°, -135°], [-135°, -90°], [-90°, -45°], [-45°, 0°], [0°, 45°], [45°, 90°], [90°, 135°], [135°, 180°]. After determining the first direction angle of the target to be detected, the first divided area to which the first direction angle belongs can be determined based on the direction angle range corresponding to each divided area.

[0084] Since the first divided area corresponds to a direction angle range, in order to accurately determine the relationship between the first direction angle and the first divided area, the first deviation angle between the first direction angle and the starting angle of the first divided area can also be determined. For example, when the first direction angle is 20° and the first divided area is [0°, 45°], the first deviation angle is the difference between 20° and 0°, 20°.

[0085] Optionally, the three-dimensional attribute annotation information also includes: a first center point, a first three-dimensional dimension, and a depth value corresponding to the target to be detected. Among them, the first center point can represent the center of the target to be detected, including the coordinate values ​​of the x-axis, y-axis, and z-axis. The first three-dimensional dimension includes the length, width, and height of the target to be detected. The depth value corresponding to the target to be detected can indicate the distance from the fisheye camera (the vehicle where it is located). The smaller the depth value, the closer it is to the fisheye camera. Among them, the depth value can be represented by the coordinate value of the z-axis.

[0086] In one possible implementation, the image information captured by the fisheye camera only includes information in the two-dimensional space, while the sample image needs to be annotated with three-dimensional attribute annotation information of the target to be detected. Therefore, the point cloud data captured by the radar can be obtained first, and the point cloud data includes information in the three-dimensional space. Then, based on the conversion relationship between the radar coordinate system and the camera coordinate system, the point cloud data is converted into image data in the camera coordinate system, and the three-dimensional space information is retained. That is, the three-dimensional attribute annotation information in the sample image can be obtained.

[0087] S102: Input the sample image into the initial detection model to obtain the three-dimensional attribute prediction information of the target to be detected.

[0088] The three-dimensional attribute prediction information includes the second direction angle, the second divided area to which the second direction angle belongs, and the second deviation angle within the second divided area.

[0089] Optionally, when the three-dimensional attribute annotation information also includes a first center point and a first three-dimensional size, the three-dimensional attribute prediction information output by the initial detection model may also include a second center point and a second three-dimensional size, indicating the center point and three-dimensional size of the target to be detected predicted by the initial detection model.

[0090] Among them, the structure of the initial detection model may include a backbone network, a feature fusion network, and a detection head. The backbone network is usually used for feature extraction. For example, the backbone network can usually extract high-level features of the image through operations such as convolution and pooling, and provide rich information for subsequent downstream tasks (such as classification, detection, etc.). Optionally, the backbone network can usually use a pre-trained network, that is, a network trained on a large dataset, in order to achieve good performance in various downstream tasks.

[0091] The feature fusion network can fuse multiple image features output by the backbone network to obtain fused image features, which contain information of multiple image features and facilitate subsequent image processing.

[0092] In the target detection task, the detection head mainly performs target detection based on the input image features. The functions of the detection head are mainly divided into two types, one is classification, that is, predicting the category of the object to be detected. When determining the category of the target to be detected, the category confidence can also be output to indicate the accuracy of the target to be detected as the predicted category. The other is regression, that is, regression prediction of the three-dimensional size, direction angle and other information of the target to be detected. In an embodiment of the present application, the detection head realizes three-dimensional target detection by detecting the center point of the target to be detected, which can reduce the computational complexity of the detection head, reduce the time for the detection head to perform target detection, and is more suitable for low-computing power chips. For example, the detection head can be a detection head in a centerpoint model, and the centerpoint heat map of the target to be detected is obtained by using the detection head of the centerpoint model, and the three-dimensional attribute prediction information of the target to be detected is obtained by regression.

[0093] After the sample image is input into the initial detection model, the three-dimensional attribute prediction information of the target to be detected can be obtained, including the second direction angle of the target to be detected, the second divided area to which the second direction angle belongs, and the second deviation angle in the second divided area.

[0094] S103: Determine a loss function value based on the three-dimensional attribute prediction information and the three-dimensional attribute annotation information.

[0095] The three-dimensional attribute annotation information pre-annotated in the sample image can be used as the true value for training, so as to determine the loss function value based on the three-dimensional attribute prediction information and the three-dimensional attribute annotation information. The loss function value is used to represent the error between the three-dimensional attribute prediction information and the three-dimensional attribute annotation information. When the loss function value is larger, it indicates that the error between the two is larger, and the accuracy of the initial detection model is lower. Then, the initial detection model needs to be trained based on the loss function value.

[0096] S104: Train the initial detection model based on the loss function value until the training process is stopped when a cutoff condition is met, thereby obtaining a trained three-dimensional object detection model.

[0097] Specifically, when the loss function value is greater than or equal to the preset value, it indicates that the accuracy of the initial detection model does not meet the requirements, and it is necessary to adjust the parameters of the initial detection model, and re-execute the sample image data into the initial detection model after adjusting the parameters and the subsequent training process. Repeat the above process until the training cutoff condition is met, and then stop the training process to obtain a trained three-dimensional target detection model. Optionally, the cutoff condition can be that the re-determined loss function value is less than the preset value. Or the training process can be stopped when the number of training iterations reaches a preset number.

[0098] According to the above embodiment, the three-dimensional attribute annotation information may include a first center point, a first three-dimensional size, a depth value corresponding to the target to be detected, a first direction angle, a first divided area to which the first direction angle belongs, and a first deviation angle within the first divided area. The three-dimensional attribute prediction information may include a second center point, a second three-dimensional size, a second direction angle, a second divided area to which the second direction angle belongs, and a second deviation angle within the second divided area. In a possible implementation method, the loss function value may be determined by the following steps:

[0099] A1: Determine the position weight corresponding to the depth value based on the relationship between the position weight threshold, the depth value and the depth threshold, wherein the position weight threshold corresponds to the depth threshold.

[0100] When detecting obstacles around a vehicle, different distances from the vehicle (fisheye camera) have different effects on vehicle parking. Therefore, the initial detection model can be trained by assigning different position weights to obstacles at different distances, thereby improving the ability of the initial detection model to detect targets.

[0101] Specifically, obstacles closer to the vehicle have a greater impact on parking, and the position weight corresponding to such obstacles can be set to be larger, while the position weight corresponding to obstacles farther from the vehicle can be set to be smaller. Optionally, the farthest distance detected by the fisheye camera can be pre-set, and the position weight corresponding to the obstacle at the farthest distance can be set to the minimum weight. The position weight corresponding to the obstacle closest to the vehicle is set to the maximum weight, thereby determining the range of the position weight threshold.

[0102] When shooting obstacles at different distances from the vehicle, the corresponding depth values ​​of the obstacles in the image are different. The closer the obstacle is to the vehicle, the smaller the depth value corresponding to the obstacle. Since the farthest distance that the fisheye camera can detect can be set, the depth value corresponding to the obstacle shot at the farthest distance is the maximum depth value, which can be expressed as a depth threshold. In other words, the obstacle farthest from the vehicle can be represented by setting a depth threshold. At this time, the position weight corresponding to the obstacle at the farthest distance is the minimum weight, that is, the depth threshold corresponds to the minimum weight of the position weight threshold. Based on this, the position weight corresponding to the depth value of the target to be detected can be determined based on the relationship between the depth value of the target to be detected and the depth threshold, and the relationship between the depth threshold and the position weight threshold.

[0103] In one possible implementation, the position weight threshold can be set to a range of [min_weight, max_weight], the depth threshold is max_depth, the depth value of the target to be detected is represented as gt_depth, and the position weight weight corresponding to the target to be detected can be calculated in the following way: weight = max_weight-(gt_depth / max_depth)×(max_weight-min_weight).

[0104] It should be noted that the embodiments of the present application do not limit the specific ranges of the depth threshold and the position weight threshold, and can be set in combination with actual application scenarios, which does not affect the implementation of the embodiments of the present application. For example, the depth threshold can be set to 50, and the position weight threshold range can be [0.5, 2], then the position weight weight of the target to be detected is weight = 2-1.5 (gt_depth / 50).

[0105] A2: Determine a first loss function value based on the first center point, the second center point and the position weight.

[0106] The error between the first center point and the second center point may be determined first, and the error may be multiplied by the position weight to obtain a first loss function value. Alternatively, the error between the first center point and the second center point may be calculated using an L1 loss function.

[0107] A3: Determine a second loss function value based on the first three-dimensional size, the second three-dimensional size, and the position weight.

[0108] The error between the first three-dimensional size and the second three-dimensional size may be determined first, and the error may be multiplied by the position weight to obtain the second loss function value.

[0109] A4: Determine a third loss function value based on the first direction angle, the second direction angle and the position weight.

[0110] The error between the first direction angle and the second direction angle may be determined first, and the error may be multiplied by the position weight to obtain the third loss function value.

[0111] A5: Determine a fourth loss function value based on the first divided area and the second divided area.

[0112] For example, the fourth loss function values ​​of the first divided area and the second divided area may be calculated using the L1 loss function.

[0113] A6: Determine a fifth loss function value based on the first deviation angle and the second deviation angle.

[0114] For example, the fifth loss function values ​​of the first deviation angle and the second deviation angle may be calculated using the L1 loss function.

[0115] After determining the five loss function values, the five loss function values ​​are used to train the initial detection model. For example, the cutoff condition of the training process is that the five loss function values ​​are respectively less than the corresponding preset values, or the number of training iterations reaches a preset number, thereby obtaining a trained three-dimensional object detection model, which can be used to detect obstacles around the vehicle to achieve three-dimensional object detection.

[0116] See also Figure 2 As shown, Figure 2 A flowchart of a three-dimensional target detection method provided in an embodiment of the present application.

[0117] The method can be applied to an image processing device. Optionally, the image processing device can be a terminal device or a server. The terminal device includes but is not limited to a desktop computer, a laptop computer, a tablet computer, and a smart phone. After acquiring the image data taken by the fisheye camera, the image processing device can process the image data using a three-dimensional object detection model to realize three-dimensional object detection in the image.

[0118] Optionally, the image processing device and the training device in the above embodiment can be the same device or different devices. When the image processing device and the training device are different devices, the image processing device can obtain the three-dimensional object detection model in the training device and store it, so as to facilitate the subsequent direct call of the stored three-dimensional object detection model for image processing.

[0119] In one possible implementation, the method can be applied to an automatic parking scenario to detect obstacles around the vehicle during the automatic parking process. For example, a fisheye camera installed on the vehicle can be used to capture images around the vehicle. The captured image data is then sent to an image processing device. The image processing device can be a device with an image processing function at the vehicle control end, or it can be a cloud server. The image processing device processes the image data, detects the target to be detected in the image, such as vehicles, pedestrians and other obstacles, and obtains a three-dimensional detection result. The three-dimensional detection result can then be sent to a device that controls the vehicle to park, and the parking path is planned to avoid obstacles, thereby controlling the vehicle to complete automatic parking. Since three-dimensional target detection can improve the accuracy of the detected target, it can also improve the accuracy of the parking path planning, and improve the accuracy and safety of automatic parking.

[0120] In specific implementation, the method may include the following steps:

[0121] S201: Acquire image data captured by a fisheye camera.

[0122] Among them, the fisheye camera is a lens with a short focal length and a viewing angle of nearly 180 degrees, which is a type of ultra-wide-angle lens. Due to its extremely large viewing angle, the fisheye camera can provide a wider field of view than ordinary cameras, so the fisheye camera is also suitable for automatic parking scenarios. During the automatic parking process, the fisheye camera can cover a larger perception range, making up for the shortcomings of ordinary cameras in near-field perception, and helping to more accurately identify the vehicle's surrounding environment and obstacles.

[0123] In order to facilitate the 3D object detection model to process the image data and simplify the processing process, in a possible implementation, the image data can be adjusted to a uniform size to facilitate the 3D object detection model to process the image data. Specifically, the initial image data taken by the fisheye camera can be first obtained, and then the initial image data can be affine transformed to obtain the processed image data.

[0124] The original affine transformation usually involves operations such as rotation, scaling, translation, and shearing, but the amount of calculation of these processing processes is relatively large, and it is difficult to meet the needs of real-time detection in automatic parking scenarios. Based on this, in a possible implementation method, the affine transformation of the initial image data in the embodiment of the present application mainly includes scaling and translation processes to adjust the size and position of the image data. For example, the initial image data can be scaled to the required image size, and the processing process is relatively simple, which reduces the calculation time. Through the translation process, the center point position (or key point position) of the image can be kept correct. The translation process only needs to calculate the offset of the pixel coordinates, and the calculation process is also relatively simple. The affine transformation calculation process provided in the embodiment of the present application can reduce the complexity of the calculation, and can be applied to chips with low computing power to realize automatic parking control, which can better meet the needs of real-time detection.

[0125] S202: Input the image data into a three-dimensional object detection model to obtain three-dimensional attribute information of the object to be detected.

[0126] After acquiring the image data of the fisheye camera, the image data can be input into the three-dimensional target detection model, and the three-dimensional target detection model is used to detect the target to be detected in the image, and the three-dimensional attribute information of the target to be detected is output. The positioning of the target to be detected can be achieved through the three-dimensional attribute information, and the target detection is also achieved. Among them, the three-dimensional target detection model is trained based on the training method of the three-dimensional target detection model provided in the above embodiment. The three-dimensional attribute information may include the center point, three-dimensional size, divided area, deviation angle and category confidence of the target to be detected. The target to be detected may represent obstacles such as pedestrians and other vehicles around the vehicle. Among them, the divided area represents the divided area to which the direction angle of the target to be detected belongs, and the deviation angle represents the deviation angle of the direction angle of the target to be detected relative to the starting angle of the divided area to which it belongs. The direction angle can be determined based on the divided area and the deviation angle. The category confidence indicates the accuracy rate of the target to be detected as the predicted category.

[0127] According to the above embodiments, the three-dimensional target detection model provided by the embodiments of the present application may include a backbone network, a feature fusion network, and a detection head. In specific implementation, the image data is input into the backbone network to obtain image features of multiple different scales. Based on the feature fusion network, feature fusion is performed on image features of multiple different scales to obtain fused image features. The fused image features are input into the detection head to obtain a center point heat map of the target to be detected. Three-dimensional attribute information is obtained based on the center point heat map regression.

[0128] In the process of extracting image features using the backbone network, the scale of the image features will be changed, usually by downsampling the original image data to extract higher-level image features. Although the final image features include higher-level image features in the original image data, some feature information in the image data may also be lost during the downsampling process. In a possible implementation, in order to obtain more feature information in the image data to improve the accuracy of target detection, after the image data is input into the backbone network, multiple image features of different scales output by the backbone network can be obtained, thereby retaining more feature information of the image data.

[0129] Although obtaining image features of multiple different scales can retain more feature information of the image data, processing more image features will increase the complexity of the calculation, which is difficult to apply to low-computing power chips. Therefore, the embodiment of the present application can determine an appropriate number of image features. For example, the last three image features of different scales output by the backbone network can be obtained. It can retain more feature information in the image data and improve the accuracy of target detection, and it can also reduce the complexity of the calculation, which is suitable for low-computing power chips.

[0130] Based on this, when the feature fusion network performs feature fusion on image features of three different scales, it is necessary to upsample and downsample the three image features of different scales to obtain three image features of the same scale, thereby performing feature fusion on the three image features of the same scale. For example, two image features of smaller scales can be upsampled to obtain three image features of the same size as the image feature of the largest scale. Alternatively, two image features of larger scales can be downsampled to obtain three image features of the same size as the image feature of the smallest scale, so that the fused image features can include information of image features of different scales, thereby improving the accuracy of target detection by the detection head based on the fused image features.

[0131] In a possible implementation, the embodiment of the present application can use the backbone network part and feature fusion part of the DLA34 network to extract multiple image features of the image data and perform feature fusion. Among them, the basic principle of the DLA34 network can be found in existing materials and will not be introduced in detail in the embodiment of the present application.

[0132] After the fused image features are input to the detection head, the detection head can directly predict the three-dimensional properties of the target to be detected on the feature map and output the center heat map of the target to be detected in the image. The center point heat map is used to predict the position of the center point of the target to be detected. Each pixel in the feature map corresponds to a predicted value in the center point heat map, and the predicted value indicates the probability that the pixel is the center point.

[0133] Optionally, when the detection head generates a center point heat map based on the fused image features, k center point heat maps may be generated, where k represents the number of categories of the target to be detected included in the image, and each center point heat map corresponds to a category of the target to be detected. For example, when the image includes multiple pedestrians and electric vehicles, the detection head may generate two center point heat maps based on the fused image features, corresponding to the center point heat maps of pedestrians and the center point heat maps of electric vehicles, respectively.

[0134] Then, based on the center point heat map regression, the three-dimensional attribute information is obtained, which includes the center point, three-dimensional size, divided area, deviation angle and category confidence of the target to be detected. The center point represents the center of the target to be detected predicted in the center point heat map.

[0135] In a possible implementation, in the application scenario of automatic parking, in order to obtain more comprehensive information about obstacles around the vehicle and prevent collisions during parking, four fisheye cameras can be installed in four directions of the vehicle to more comprehensively capture the vehicle's surroundings. For example, four fisheye cameras can be installed in four locations: the front, left, right, and rear of the vehicle.

[0136] When obstacles around the vehicle need to be detected, the image data taken by four fisheye cameras at the same time can be obtained, and then the image data taken by each fisheye camera can be input into the 3D object detection model to obtain the 3D attribute information of the target to be detected corresponding to each camera. In this way, obstacles around the vehicle can be detected more comprehensively and accurately, improving the safety and accuracy of automatic parking.

[0137] In a possible implementation, the image data captured by four fisheye cameras at the same time can be simultaneously input into the three-dimensional object detection model to improve the efficiency of object detection. Specifically, the backbone network in the three-dimensional object detection model can include four identical feature extraction modules, and the image data captured by the four fisheye cameras at the same time are respectively input into the four feature extraction modules, and each feature extraction module outputs multiple image features of different scales of the corresponding image data.

[0138] The feature fusion network may include four identical feature fusion modules, and the multiple image features of different scales output by each feature extraction module are respectively input into a feature fusion module for feature fusion to obtain corresponding fused image features. That is, the image data captured by the four fisheye cameras correspond to four fused image features respectively.

[0139] The detection head may include four detection modules, and each fused image feature is input into a detection module to obtain the three-dimensional attribute information of the target to be detected in the corresponding fused image feature.

[0140] S203: Obtaining a three-dimensional detection result of the target to be detected based on the three-dimensional attribute information.

[0141] After obtaining the three-dimensional attribute information of the target to be detected, such as the center point, three-dimensional size, divided area, deviation angle, etc., the specific position of the target to be detected can be determined and the three-dimensional detection result of the target to be detected can be obtained.

[0142] According to the above embodiments, in the automatic parking scenario, the four-way fisheye camera installed on the vehicle can be used to capture image data to detect obstacles around the vehicle. In a possible implementation, the initial three-dimensional detection result can be obtained based on the three-dimensional attribute information of the target to be detected corresponding to each camera. Then the initial three-dimensional detection results corresponding to the two adjacent cameras are compared, and the repeated three-dimensional detection results are deleted to obtain the remaining three-dimensional detection results. That is, the image data captured by the four-way fisheye camera may contain image data of repeated areas. Therefore, after the initial three-dimensional detection results of all the targets to be detected are obtained based on the four-way fisheye camera, the repeated three-dimensional detection results can be deleted to prevent the subsequent repeated calculation of the same target to be detected during the automatic parking path planning, thereby reducing the waste of resources.

[0143] Specifically, after obtaining the initial three-dimensional detection result based on the three-dimensional attribute information of the target to be detected corresponding to each camera, the overlap between the initial three-dimensional detection results corresponding to two adjacent cameras is calculated. When the overlap is greater than the preset overlap, the initial three-dimensional detection result with a lower category confidence is deleted to obtain the remaining three-dimensional detection results. For example, the area where the target to be detected is located can be determined as the initial three-dimensional detection result through the three-dimensional attribute information, and the overlap between the areas where the target to be detected is located respectively determined by the two adjacent cameras is determined. Among them, the overlap between the two areas can be determined by calculating the intersection and union ratio. When the overlap between the initial three-dimensional detection results corresponding to two adjacent cameras is high, it indicates that the target to be detected determined by the two adjacent cameras is likely to be the same target, so one initial three-dimensional detection result can be deleted. The category confidence indicates the accuracy of the target to be detected as the predicted category, so the initial three-dimensional detection result of the target to be detected with a lower category confidence can be deleted.

[0144] Since the output three-dimensional attribute information includes the category confidence of the target to be detected, which is used to indicate the accuracy of the target to be detected being the predicted category, in a possible implementation, the three-dimensional attribute information can be screened based on the confidence threshold to obtain the three-dimensional attribute information that meets the requirements. The confidence threshold indicates the lower limit value of the accuracy of the predicted category of the target to be detected that meets the requirements. When the category confidence is less than or equal to the confidence threshold, it indicates that the predicted category to be detected is inaccurate and does not meet the requirements.

[0145] Specifically, the category confidence in the three-dimensional attribute information is compared with the confidence threshold, and the target three-dimensional attribute information is screened, wherein the category confidence included in the target three-dimensional attribute information is greater than the confidence threshold. That is, the three-dimensional attribute information corresponding to the category confidence less than or equal to the confidence threshold is removed, thereby obtaining the target three-dimensional attribute information. Then, based on the target three-dimensional attribute information, the three-dimensional detection result of the target to be detected is obtained.

[0146] In a possible implementation, when multiple fisheye cameras are installed on a vehicle, the multiple fisheye cameras may capture the same target to be detected, and the three-dimensional attribute information of the same target to be detected output by the three-dimensional target detection model may include multiple three-dimensional attribute information. Since a three-dimensional space region can be determined based on the three-dimensional attribute information, there may be an intersecting region between multiple three-dimensional attribute information. In order to accurately obtain the three-dimensional attribute information of the target to be detected, the three-dimensional attribute information can be screened using a non-maximum suppression algorithm to remove redundant three-dimensional attribute information. It can be mainly implemented by the following steps: for any one of the multiple three-dimensional attribute information, determine the region corresponding to the current three-dimensional attribute information. Then calculate the intersection over union (IOU) of the region corresponding to the current three-dimensional attribute information and the region corresponding to other three-dimensional attribute information. IOU can represent the degree of overlap between two regions, and its value is between 0 and 1. The higher the IOU value, the higher the degree of overlap between the two regions. If the IOU between the region corresponding to a certain three-dimensional attribute information and the region corresponding to the current three-dimensional attribute information is higher than the threshold, it can be considered that they detect the same region, and the three-dimensional attribute information is removed. After removing the current 3D attribute information, select the next 3D attribute information from the remaining 3D attribute information and repeat the above process until all the 3D attribute information is processed, and the 3D attribute information finally retained is the optimal result. Therefore, the 3D detection result of the target to be detected can be obtained based on the final 3D attribute information.

[0147] In a possible implementation, the three-dimensional detection result can be determined in the following manner: based on the three-dimensional attribute information of the target to be detected, a three-dimensional bounding box is determined. That is, the area determined by the three-dimensional bounding box is the detected target to be detected. Since the three-dimensional attribute information includes the center point, three-dimensional size, divided area, and deviation angle of the target to be detected, the three-dimensional bounding box can be drawn according to the three-dimensional attribute information.

[0148] In specific implementation, the direction angle of the target to be detected is determined based on the direction angle range and deviation angle corresponding to the divided area. The direction angle range corresponding to each divided area can be determined when training the three-dimensional target detection model. The deviation angle represents the deviation angle from the starting angle of the divided area. Therefore, the direction angle of the target to be detected can be determined based on the sum of the starting angle and the deviation angle in the direction angle range. Then, the three-dimensional bounding box of the target to be detected is determined based on the center point, three-dimensional size and direction angle.

[0149] Since the image data is in the camera coordinate system, the obtained 3D bounding box is also in the camera coordinate system. In order to more conveniently determine the target to be detected in the image, the 3D bounding box can be mapped to a preset type of image coordinate system through coordinate system conversion to obtain a specific type of 3D detection result.

[0150] In a possible implementation, the 3D bounding box in the camera coordinate system can be mapped to the 3D bounding box in the image coordinate system through the conversion relationship between the camera coordinate system and the image coordinate system, so that the target to be detected can be directly determined in the image taken by the fisheye camera, making the 3D target detection result visual. Figure 3a FIG. 1 is a schematic diagram of three-dimensional target detection in an image coordinate system provided in an embodiment of the present application. Figure 3a The includes multiple bounding box areas, each of which is a corresponding area converted from a three-dimensional bounding box to an image coordinate system, and the multiple bounding box areas represent multiple targets to be detected in the image taken by the fisheye camera.

[0151] In a possible implementation, the 3D bounding box in the camera coordinate system can also be mapped to a bird's eye view (BEV) coordinate system. The image in the BEV coordinate system represents a 2D plane. In the BEV coordinate system, obstacles (targets to be detected) and road information around the vehicle can be uniformly represented on a 2D plane. Therefore, the automatic parking system can more easily perform path planning and control the vehicle to complete automatic parking. For details, please refer to Figure 3b , which is a schematic diagram of three-dimensional target detection in a BEV coordinate system provided in an embodiment of the present application, Figure 3b The 3D bounding box region is converted from the 3D bounding box to the corresponding region in the BEV coordinate system. The 3D bounding box regions represent the detection of multiple targets to be detected in the image captured by the fisheye camera. In order to facilitate the automatic parking system to plan the automatic parking path for the vehicle based on the targets to be detected, the 3D position of the vehicle itself can also be converted to the BEV coordinate system, such as Figure 3b White filled area shown.

[0152] Through the method provided in the embodiment of the present application, the three-dimensional target detection model can be used to realize three-dimensional detection of the target, thereby improving the accuracy of target detection. In addition, the three-dimensional target detection model provided in the present application realizes three-dimensional target detection based on the center point, which can reduce the time of model prediction, reduce the computing resources occupied by the model, and is more suitable for low-computing power chips.

[0153] Based on the above method embodiment, the present application embodiment also provides a three-dimensional target detection method. Figure 4 As shown, Figure 4 A flowchart of another three-dimensional target detection method provided in an embodiment of the present application.

[0154] The method mainly includes the following steps:

[0155] B1: Obtain image data captured by four fisheye cameras on the vehicle at the same time.

[0156] B2: For each fisheye camera, the image data captured by the fisheye camera is input into the feature extraction module to obtain image features of multiple different scales.

[0157] B3: Input multiple image features of different scales into the feature fusion module for feature fusion to obtain the corresponding fused image features.

[0158] B4: Input each fused image feature into a detection module, obtain the three-dimensional attribute information of the target to be detected in the corresponding fused image feature, and determine the three-dimensional detection result.

[0159] B5: Map the 3D detection results corresponding to each fisheye camera to the BEV coordinate system.

[0160] B6: In the BEV coordinate system, the repeated 3D detection results in two adjacent cameras are deleted.

[0161] B7: Send the retained 3D detection results to the downstream task module.

[0162] For example, the downstream task module may be an automatic parking control system, which, after obtaining the obstacle detection results around the vehicle, performs parking path planning and controls the vehicle to complete automatic parking.

[0163] Through the method provided in the embodiment of the present application, the three-dimensional target detection model can be used to realize three-dimensional detection of the target, thereby improving the accuracy of target detection. After the target detection result is sent to the automatic parking control system, the accuracy of automatic parking based on the target detection result can be improved. In addition, the embodiment of the present application completes automatic parking based on the image data captured by the four-way fisheye camera, which can further improve the accuracy and safety of automatic parking. Since the three-dimensional target detection model (detection module) realizes three-dimensional target detection based on the center point, it can reduce the time of model prediction, reduce the computing resources occupied by the model, and is more suitable for low-computing power chips.

[0164] Based on the above method embodiment, the present application embodiment provides a training device for a three-dimensional object detection model. Figure 5 As shown, Figure 5 A schematic diagram of a training device for a three-dimensional target detection model provided in an embodiment of the present application.

[0165] The sample acquisition module 501 is used to obtain a sample image taken by a fisheye camera, wherein the sample image is annotated with three-dimensional attribute annotation information of a target to be detected, wherein the three-dimensional attribute annotation information includes a first direction angle, a first divided area to which the first direction angle belongs, and a first deviation angle within the first divided area;

[0166] A detection module 502 is used to input the sample image into an initial detection model to obtain three-dimensional attribute prediction information of the target to be detected, wherein the three-dimensional attribute prediction information includes a second direction angle, a second divided area to which the second direction angle belongs, and a second deviation angle in the second divided area;

[0167] A loss function determination module 503, configured to determine a loss function value based on the three-dimensional attribute prediction information and the three-dimensional attribute annotation information;

[0168] The training module 504 is used to train the initial detection model based on the loss function value until the training process is stopped when a cutoff condition is met, thereby obtaining a trained three-dimensional object detection model.

[0169] In a possible implementation, the sample collection module 501 is specifically configured to divide the range interval to which the direction angle belongs to obtain a plurality of divided areas; and determine the first divided area to which the first direction angle belongs based on the direction angle range corresponding to each divided area.

[0170] In a possible implementation, the three-dimensional attribute annotation information further includes: a first center point, a first three-dimensional size, and a depth value corresponding to the target to be detected, and the three-dimensional attribute prediction information further includes: a second center point and a second three-dimensional size;

[0171] The loss function determination module 503 is specifically used to determine the position weight corresponding to the depth value based on a position weight threshold and the relationship between the depth value and the depth threshold, wherein the position weight threshold corresponds to the depth threshold; determine a first loss function value based on the first center point, the second center point and the position weight; determine a second loss function value based on the first three-dimensional size, the second three-dimensional size and the position weight; determine a third loss function value based on the first direction angle, the second direction angle and the position weight; determine a fourth loss function value based on the first divided area and the second divided area; determine a fifth loss function value based on the first deviation angle and the second deviation angle.

[0172] Based on the above method embodiment, the present application embodiment also provides a three-dimensional object detection device. Figure 6 , which is a schematic diagram of a three-dimensional target detection device provided in an embodiment of the present application.

[0173] The device 600 comprises:

[0174] The data acquisition module 601 is used to acquire image data captured by a fisheye camera on the vehicle;

[0175] A model detection module 602 is used to input the image data into the three-dimensional object detection model described in the above method embodiment to obtain three-dimensional attribute information of the object to be detected in the image data;

[0176] The three-dimensional detection module 603 is used to obtain a three-dimensional detection result of the target to be detected based on the three-dimensional attribute information.

[0177] In one possible implementation, the three-dimensional target detection model includes a backbone network, a feature fusion network and a detection head, and the model detection module 602 is specifically used to input the image data into the backbone network to obtain image features of multiple scales; perform feature fusion on the image features of multiple scales based on the feature fusion network to obtain fused image features; input the fused image features into the detection head to obtain a center point heat map of the target to be detected; and regress the three-dimensional attribute information based on the center point heat map.

[0178] In a possible implementation, the image data includes image data respectively captured by four fisheye cameras at the same time;

[0179] The model detection module 602 is specifically used to input the image data taken by each fisheye camera into the three-dimensional object detection model to obtain the three-dimensional attribute information of the target to be detected corresponding to each camera.

[0180] In one possible implementation, the three-dimensional attribute information includes category confidence, and the three-dimensional detection module 603 is specifically used to obtain an initial three-dimensional detection result based on the three-dimensional attribute information of the target to be detected corresponding to each camera; calculate the overlap between the initial three-dimensional detection results corresponding to two adjacent cameras; when the overlap is greater than a preset overlap, delete the initial three-dimensional detection result with a lower category confidence to obtain the remaining three-dimensional detection results.

[0181] In a possible implementation, the three-dimensional attribute information includes a center point, a three-dimensional size, a divided area, and a deviation angle;

[0182] The three-dimensional detection module 603 is specifically used to determine the direction angle of the target to be detected based on the direction angle range corresponding to the divided area and the deviation angle; determine the three-dimensional bounding box of the target to be detected based on the center point, the three-dimensional size and the direction angle; and map the three-dimensional bounding box to a preset type of image coordinate system through a coordinate system conversion to obtain the three-dimensional detection result.

[0183] In a possible implementation, the three-dimensional attribute information includes category confidence, and the three-dimensional detection module 603 is specifically used to compare the category confidence in the three-dimensional attribute information with a confidence threshold, and filter out the target three-dimensional attribute information, the category confidence indicates the accuracy of the target to be detected being a predicted category, and the category confidence included in the target three-dimensional attribute information is greater than the confidence threshold; based on the target three-dimensional attribute information, the three-dimensional detection result is obtained.

[0184] Based on the above method embodiment and device embodiment, the present application embodiment further provides an electronic device, which will be described below in conjunction with the accompanying drawings.

[0185] See also Figure 7 , Figure 7 A schematic diagram of an electronic device provided in an embodiment of the present application.

[0186] The device 700 includes: a memory 701 and a processor 702;

[0187] The memory 701 is used to store relevant program codes;

[0188] The processor 702 is used to call the program code to execute the three-dimensional target detection model training method or the three-dimensional target detection method described in the above method embodiment.

[0189] In addition, an embodiment of the present application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the three-dimensional target detection model training method or the three-dimensional target detection method described in the above method embodiment.

[0190] An embodiment of the present application also provides a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the three-dimensional target detection method described in the above method embodiment is implemented.

[0191] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0192] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0193] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can refer to each other. In particular, for the system or device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only exemplary, in which the units or modules described as separate components may or may not be physically separated, and the components displayed as units or modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units, and some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0194] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the method, device and equipment etc. of various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0195] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0196] It should also be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0197] The steps of the method or algorithm described in conjunction with the embodiments disclosed in this application can be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0198] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined in this application can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown in the present application, but will conform to the widest range consistent with the principles and novel features disclosed in the present application.

Claims

1. A training method for a three-dimensional object detection model, characterized in that: The method comprises: Acquire a sample image taken by a fisheye camera, wherein the sample image is annotated with three-dimensional attribute annotation information of a target to be detected, wherein the three-dimensional attribute annotation information includes a first direction angle, a first divided area to which the first direction angle belongs, and a first deviation angle within the first divided area; Inputting the sample image into an initial detection model to obtain three-dimensional attribute prediction information of the target to be detected, wherein the three-dimensional attribute prediction information includes a second direction angle, a second divided area to which the second direction angle belongs, and a second deviation angle within the second divided area; Determining a loss function value based on the three-dimensional attribute prediction information and the three-dimensional attribute annotation information; The initial detection model is trained based on the loss function value until a cutoff condition is met and the training process is stopped to obtain a trained three-dimensional object detection model.

2. The method according to claim 1, characterized in that: Determining a first divided area to which the first direction angle belongs includes: Divide the range interval to which the direction angle belongs to obtain multiple divided areas; Based on the direction angle range corresponding to each divided area, the first divided area to which the first direction angle belongs is determined.

3. The method according to claim 1, characterized in that The three-dimensional attribute annotation information further includes: a first center point, a first three-dimensional size, and a depth value corresponding to the target to be detected, and the three-dimensional attribute prediction information further includes: a second center point and a second three-dimensional size; The determining of the loss function value based on the three-dimensional attribute prediction information and the three-dimensional attribute labeling information includes: Determining a position weight corresponding to the depth value based on a position weight threshold and a relationship between the depth value and the depth threshold, the position weight threshold corresponding to the depth threshold; Determine a first loss function value based on the first center point, the second center point, and the position weight; Determining a second loss function value based on the first three-dimensional size, the second three-dimensional size, and the position weight; Determine a third loss function value based on the first direction angle, the second direction angle, and the position weight; Determine a fourth loss function value based on the first divided area and the second divided area; A fifth loss function value is determined based on the first deviation angle and the second deviation angle.

4. A three-dimensional target detection method, characterized in that: The method is applied to detecting obstacles around a vehicle in a parking scenario, and the method comprises: Acquire image data captured by a fisheye camera on a vehicle; Inputting the image data into the three-dimensional object detection model according to any one of claims 1 to 3 to obtain three-dimensional attribute information of the object to be detected in the image data; Based on the three-dimensional attribute information, a three-dimensional detection result of the target to be detected is obtained.

5. The method according to claim 4, characterized in that The three-dimensional object detection model includes a backbone network, a feature fusion network and a detection head. The image data is input into the three-dimensional object detection model to obtain three-dimensional attribute information of the object to be detected, including: Inputting the image data into the backbone network to obtain image features of multiple different scales; Based on the feature fusion network, the image features of the multiple different scales are fused to obtain fused image features; Inputting the fused image features into the detection head to obtain a central point heat map of the target to be detected; The three-dimensional attribute information is obtained based on the center point heat map regression.

6. The method according to claim 4, characterized in that The image data includes image data taken by four fisheye cameras at the same time; Inputting the image data into a three-dimensional object detection model to obtain three-dimensional attribute information of the object to be detected includes: The image data captured by each fisheye camera is input into the three-dimensional object detection model to obtain the three-dimensional attribute information of the target to be detected corresponding to each camera.

7. The method according to claim 6, characterized in that The three-dimensional attribute information includes category confidence, and obtaining the three-dimensional detection result of the target to be detected based on the three-dimensional attribute information includes: Based on the three-dimensional attribute information of the target to be detected corresponding to each camera, an initial three-dimensional detection result is obtained; Calculate the overlap between the initial 3D detection results corresponding to two adjacent cameras; When the overlap is greater than a preset overlap, initial three-dimensional detection results with lower category confidence are deleted to obtain remaining three-dimensional detection results.

8. The method according to claim 4, characterized in that The three-dimensional attribute information includes a center point, a three-dimensional size, a divided area, and a deviation angle; The obtaining of a three-dimensional detection result of the target to be detected based on the three-dimensional attribute information includes: Determining the direction angle of the target to be detected based on the direction angle range corresponding to the divided area and the deviation angle; Determine a three-dimensional bounding box of the target to be detected based on the center point, the three-dimensional size, and the direction angle; The three-dimensional bounding box is mapped to a preset type of image coordinate system through coordinate system conversion to obtain the three-dimensional detection result.

9. The method according to claim 4, characterized in that The three-dimensional attribute information includes category confidence, and obtaining the three-dimensional detection result of the target to be detected based on the three-dimensional attribute information includes: Comparing the category confidence in the three-dimensional attribute information with a confidence threshold, and filtering to obtain the target three-dimensional attribute information, wherein the category confidence represents the accuracy rate of the target to be detected being a predicted category, and the category confidence included in the target three-dimensional attribute information is greater than the confidence threshold; Based on the target three-dimensional attribute information, the three-dimensional detection result is obtained.

10. An electronic device, characterized in that: The device comprises: a memory and a processor; The memory is used to store relevant program codes; The processor is used to call the program code to execute the training method of the three-dimensional target detection model described in any one of claims 1 to 3 or the three-dimensional target detection method described in any one of claims 4 to 9.

Images