2D target and 3D target matching method, electronic equipment and storage medium

Abstract

The invention provides a 2D target and 3D target matching method, electronic equipment and a storage medium, and the method comprises the steps: determining the detection frame position of each 2D target in a camera image based on a 2D detection model, determining the spatial position of each 3D target in point cloud data according to a 3D detection model, and carrying out the detection of each 3D target for each 3D target, the matching degree is determined according to the spatial position of the 3D target and the detection frame position of each 2D target, the matching threshold corresponding to the 3D target is determined according to the spatial position of the 3D target, and finally the 2D target matched with the 3D target is determined according to the matching threshold and the matching degree between the 2D targets, so that automatic matching between the 2D targets is realized. According to the method, self-adaptive adjustment of the threshold value based on the spatial position can be achieved, the matching threshold value can be dynamically adjusted for targets of different sizes presented at different distances, mismatching and missed matching can be reduced, and then the quality of automatic labeling can be improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and specifically to a 2D target and 3D target matching method, electronic device and storage medium. Background Art

[0002] With the continuous development of autonomous driving technology, the requirements for the quantity and quality of data annotation are also increasing. Real-life driving scenarios are complex and diverse, and the problems that need to be solved in different scenarios are not the same. Autonomous driving models must adaptively iterate and optimize various scenarios. Therefore, a large amount of new scene annotation data is needed.

[0003] Among the existing annotation methods, manual annotation is costly and inefficient, and it is difficult to meet the standards for large amounts of data. Therefore, achieving a closed loop of data for automated annotation is the key to large-scale mass production of autonomous driving. Existing methods usually use a single threshold to determine whether 2D and 3D targets match. They do not consider that targets in the camera's view will appear larger when closer and smaller when farther away, and do not consider occlusion, truncation, and different target sizes and distances, which may lead to mismatches, thus affecting the quality of automated annotation and having a negative impact on model training. They are difficult to apply to actual data annotation production. Summary of the invention

[0004] In view of the above-mentioned defects or deficiencies in the prior art, the present application aims to provide a 2D target and 3D target matching method, electronic device and storage medium to solve the problems of mismatching and missed matching, thereby improving the quality of automatic labeling.

[0005] The embodiment of the present application provides a method for matching a 2D object with a 3D object, the method comprising:

[0006] Determine the detection box position of each 2D target in the camera image based on the trained 2D detection model, and determine the spatial position of each 3D target in the point cloud data based on the trained 3D detection model;

[0007] For each of the 3D objects, determining a matching degree between the 3D object and each 2D object according to a spatial position of the 3D object and a detection frame position of each 2D object, and determining a matching threshold corresponding to the 3D object based on the spatial position of the 3D object;

[0008] Based on the matching threshold corresponding to the 3D object and the matching degree, a 2D object matching the 3D object is determined.

[0009] Optionally, for each of the 3D objects, determining a matching degree between the 3D object and each 2D object according to a spatial position of the 3D object and a detection frame position of each 2D object includes:

[0010] Determine the predicted distance between the center of each 3D object and the center of the vehicle based on the spatial position of each 3D object, and sort all the 3D objects in order of predicted distance from small to large;

[0011] For each 3D object in the sorting results, according to the sorting order, based on the spatial position of the 3D object and the position of the detection frame of the unmatched 2D object, determining a matching degree between the 3D object and the unmatched 2D object;

[0012] After determining a 2D object matching the 3D object based on the matching threshold corresponding to the 3D object and the matching degree, the method further includes:

[0013] The matching state of the 2D object matched with the 3D object is updated from unmatched to matched.

[0014] Optionally, determining the predicted distance between the center of each of the 3D objects and the center of the vehicle based on the spatial position of each of the 3D objects includes:

[0015] For each of the 3D objects, based on the spatial position of the 3D object, the 3D object is converted from the 3D space to the camera coordinate system to obtain the 3D frame coordinate point of the 3D object in the camera coordinate system;

[0016] The predicted distance between the center of the 3D object and the center of the vehicle is determined based on the 3D box coordinate points in the camera coordinate system.

[0017] Optionally, determining a degree of matching between the 3D object and the unmatched 2D object based on the spatial position of the 3D object and the detection frame position of the unmatched 2D object includes:

[0018] Based on the 3D frame coordinate points of the 3D target in the camera coordinate system, projecting the 3D target from the camera coordinate system onto the camera image to obtain a projection frame circumscribed rectangle of the 3D target;

[0019] Based on the projection frame circumscribed rectangle of the 3D object and the detection frame position of the unmatched 2D object, a matching degree between the 3D object and the unmatched 2D object is determined.

[0020] Optionally, before determining the degree of matching between the 3D object and the unmatched 2D object based on the projection frame circumscribed rectangle of the 3D object and the detection frame position of the unmatched 2D object, the method further includes:

[0021] Determine an occlusion rate of the 3D object in the camera image based on a circumscribed rectangle of a projection frame of the 3D object;

[0022] If the occlusion rate is less than a preset occlusion threshold, an operation of determining a degree of matching between the 3D object and the unmatched 2D object based on a projection frame circumscribed rectangle of the 3D object and a detection frame position of the unmatched 2D object is performed.

[0023] Optionally, determining the occlusion rate of the 3D object in the camera image based on the circumscribed rectangle of the projection frame of the 3D object includes:

[0024] obtaining an occupancy mask image, wherein the occupancy mask image is constructed based on the camera image;

[0025] Using the circumscribed rectangle of the projection frame of the 3D target as prompt information of an image segmentation model, segmenting a 2D target corresponding to the 3D target in the camera image based on the image segmentation model to obtain a segmentation mask image corresponding to the 3D target;

[0026] An occlusion rate of the 3D object in the camera image is determined based on the occupancy mask image and the segmentation mask image, and the occupancy mask image is updated according to the segmentation mask image.

[0027] Optionally, determining a matching threshold corresponding to the 3D object based on the spatial position of the 3D object includes:

[0028] Get the preset farthest sensing distance, minimum matching threshold and maximum matching threshold;

[0029] A matching threshold corresponding to the 3D target is determined based on the predicted distance between the center of the 3D target and the center of the vehicle, the farthest perceived distance, the minimum matching threshold, and the maximum matching threshold.

[0030] Optionally, while determining the detection frame position of each 2D target in the camera image based on the trained 2D detection model and determining the spatial position of each 3D target in the point cloud data based on the trained 3D detection model, it also includes:

[0031] Determining a detection category of each 2D object in the camera image based on the 2D detection model, and determining a detection category of each 3D object in the point cloud data based on the 3D detection model;

[0032] After determining the 2D object matching the 3D object, the method further includes:

[0033] updating the detection category of the 3D object based on the detection category of the 2D object matching the 3D object; and / or,

[0034] For 3D targets that are not matched with 2D targets, the point cloud data and the spatial position of the 3D target are associated with the detection category and stored in a visual database, wherein the visual database is used for training a visual detection model.

[0035] The present application also provides an electronic device, the electronic device comprising:

[0036] Processor and memory;

[0037] The processor is used to execute the steps of the method for matching a 2D object with a 3D object provided in any embodiment of the present application by calling the program or instruction stored in the memory.

[0038] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program or instruction, wherein the program or instruction enables a computer to execute the steps of the method for matching a 2D object with a 3D object provided in any embodiment of the present application.

[0039] In summary, the present application proposes a method for matching 2D targets and 3D targets. The method determines the detection frame position of each 2D target in the camera image based on a trained 2D detection model, and determines the spatial position of each 3D target in the point cloud data according to the trained 3D detection model. Then, for each 3D target, the matching degree is determined by the spatial position of the 3D target and the detection frame position of each 2D target, and the matching threshold corresponding to the 3D target is determined by the spatial position of the 3D target. Finally, the 2D target matching the 3D target is determined by the matching threshold corresponding to the 3D target and the matching degree with each 2D target, thereby realizing automatic matching between the 2D target and the 3D target. The method can determine the corresponding matching threshold based on the spatial position of the 3D target, realizes adaptive adjustment of the threshold based on the spatial position, and can dynamically adjust the matching threshold for targets of different sizes presented at different distances, which is beneficial to improving the accuracy of matching of long-distance targets, ensuring the reliability of matching of close-range targets, reducing the occurrence of false matching and missed matching, and thus helping to improve the quality of automatic labeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 A flowchart of a method for matching a 2D target and a 3D target provided in an embodiment of the present application;

[0042] Figure 2 is a schematic diagram of updating a projection frame circumscribed rectangle provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of a matching process between a 2D target and a 3D target provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of the structure of a 2D target and a 3D target matching device provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0047] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0048] Before introducing in detail the method for matching a 2D target and a 3D target provided in an embodiment of the present application, the technical problem solved by the method is first described.

[0049] Autonomous driving data collection vehicles rely on various sensors throughout the vehicle body to collect a variety of data such as point cloud (Lidar), millimeter wave radar (Radar), camera images, inertial measurement unit (IMU), differential positioning (Real-Time Kinematic, RTK), etc. However, the unprocessed data collected cannot be directly used for model training. It can only be used after annotation. The primary problem between data and model training is how to efficiently annotate massive data. High-quality and large-scale annotated data has become the key to realizing autonomous driving.

[0050] Matching 3D and 2D objects is an important step in automatic labeling. In existing methods, the matching of 2D and 3D objects is usually based on IOU or center distance, and a single threshold is set. There is no effective method for matching 2D and 3D objects in occlusion, truncation, multi-size and different distances, which leads to mismatching or missing matching results, resulting in the quality of automatic labeling having a negative impact on model training and difficult to apply to actual data labeling production.

[0051] As mentioned in the background technology, in order to solve the problems in the prior art, the present application proposes a matching method for a 2D object and a 3D object. Figure 1 A flowchart of a method for matching a 2D object with a 3D object provided in an embodiment of the present application. Figure 1 , the matching method of the 2D target and the 3D target specifically includes:

[0052] S110 , determining a detection frame position of each 2D target in the camera image based on the trained 2D detection model, and determining a spatial position of each 3D target in the point cloud data based on the trained 3D detection model.

[0053] Among them, the 2D detection model can be a target detection model pre-trained with image data, such as DETR (Detection Transformer, a model that introduces the Transformer structure into the field of target detection), DINO (DETR with Improved DeNoising Anchor Boxes, based on the DETR model with improved denoising anchor boxes), etc. The 3D detection model can be a target detection model pre-trained with lidar point cloud data (it can also be combined with image data, millimeter wave radar point cloud data, etc.), such as BEVFusion (Bird's Eye View Fusion, multi-sensor fusion), CMT (Convolutional Neural Networks Meet Vision Transformers, convolutional neural networks and visual transformers).

[0054] Exemplarily, multi-sensor data such as point cloud data, millimeter-wave radar data, and camera image data can be collected by a data acquisition vehicle. After pre-processing such as scaling, the image data enters the 2D detection model for training. In addition, the image data enters the multimodal 3D detection model for training after pre-processing such as scaling and the lidar point cloud enters the multimodal 3D detection model for training after pre-processing such as voxelization.

[0055] In an embodiment of the present application, various sensors mounted on the vehicle can be used to collect information about the vehicle's surrounding environment while the vehicle is driving, to obtain camera images and point cloud data from different camera perspectives, and then model reasoning can be performed based on the trained 2D detection model and 3D detection model.

[0056] Specifically, the camera image can be input into the trained 2D detection model to obtain the detection frame position of each 2D target in the camera image; at the same time, the detection category of each 2D target in the camera image can also be obtained based on the 2D detection model.

[0057] The position of the detection box can be represented by the position of the vertex in the target detection box of the 2D target in the image. For example, the position of the detection box is represented by (x min ,y min ,x max ,y max ) means, x min ,y min Represents the coordinate of the upper left corner of the target detection box, x max ,y max Represents the coordinates of the lower right corner of the target detection box.

[0058] In addition, the point cloud data (lidar point cloud data, or lidar point cloud data + millimeter wave radar data) can be input into the trained 3D detection model to obtain the spatial position of each 3D target in the point cloud data; at the same time, the detection category of each 3D target in the point cloud data can also be obtained based on the 3D detection model. It should be noted that in order to improve the detection accuracy of 3D targets, the lidar point cloud data, millimeter wave radar data and camera images can also be input into the 3D detection model together to improve the detection accuracy of the model.

[0059] The spatial position can be formed by the coordinates of the center point of the 3D object in the 3D space, and the physical size and yaw angle of the object detection box of the 3D object. x ,c y ,c z ,l,w,h,r) indicates that c x 、c y 、c z Represents the center point coordinates of the target detection box of the 3D target, l, w, h represent the length, width, and height of the target detection box of the 3D target, respectively, and r represents the yaw angle of the target detection box of the 3D target.

[0060] S120 , for each 3D object, determine the degree of match between the 3D object and each 2D object according to the spatial position of the 3D object and the detection frame position of each 2D object, and determine a matching threshold corresponding to the 3D object based on the spatial position of the 3D object.

[0061] In the embodiment of the present application, for each 3D object in the point cloud data, the 3D object can be matched with each 2D object according to the spatial position of the 3D object and the detection frame position of each 2D object to obtain the matching degree between the 3D object and each 2D object. The matching degree can reflect the degree of overlap between the 3D object and the 2D object on the same plane.

[0062] In an embodiment of the present application, for each 3D target, in the process of matching a 2D target for it, the matching degree between the 3D target and each 2D target can be calculated. In order to improve the matching efficiency, the successfully matched 2D targets can also be marked, and subsequently only the matching degree between the 3D target and each unmatched 2D target is calculated, so as to avoid calculating the matching degree between all 2D targets every time the 3D target is matched with a 2D target, thereby reducing the number of executions of the matching degree calculation, thereby improving the matching efficiency and reducing the computational overhead.

[0063] In addition, considering that for 3D targets that are closer to the vehicle, their corresponding 2D targets have richer information in the camera image and the detection frame position is more accurate, therefore, 3D targets that are closer to the vehicle may be matched to 2D targets more efficiently. Combined with this feature, 2D targets can be matched to each 3D target in order from near to far. During each matching process, 2D targets that have been matched to other 3D targets will no longer participate in the matching, so as to further improve the matching efficiency.

[0064] In a specific implementation, for each 3D object, according to the spatial position of the 3D object and the detection frame position of each 2D object, determining the matching degree between the 3D object and each 2D object includes the following steps:

[0065] Step 11: Determine the predicted distance between the center of each 3D object and the center of the vehicle based on the spatial position of each 3D object, and sort all 3D objects in order of predicted distance from small to large;

[0066] Step 12: for each 3D target in the sorting result, according to the sorting order, based on the spatial position of the 3D target and the position of the detection box of the unmatched 2D target, determine the matching degree between the 3D target and the unmatched 2D target;

[0067] Correspondingly, after determining the 2D object matching the 3D object based on the matching threshold and the matching degree corresponding to the 3D object, the method further includes: updating the matching state of the 2D object matching the 3D object from unmatched to matched.

[0068] Among them, in step 11, the predicted distance between the center of the 3D target and the center of the vehicle can be determined by the center point coordinates and the yaw angle in the spatial position of the 3D target, or can be determined based on the position of the 3D target in the camera coordinate system.

[0069] With respect to the above step 11, in one example, determining the predicted distance between the center of each 3D object and the center of the vehicle based on the spatial position of each 3D object includes the following steps:

[0070] Step 111: for each 3D object, based on the spatial position of the 3D object, convert the 3D object from the 3D space to the camera coordinate system to obtain the 3D frame coordinate point of the 3D object in the camera coordinate system;

[0071] Step 112: Determine the predicted distance between the center of the 3D object and the center of the vehicle based on the 3D box coordinate points in the camera coordinate system.

[0072] Among them, in step 111, for each 3D coordinate, the spatial position can be converted into a set of coordinate points based on the spatial position of the 3D target. The set of coordinate points includes the coordinates of the corner points of the target detection frame and the coordinates of the center point of the target detection frame. The spatial position of each point is represented by x, y, and z.

[0073] Exemplarily, the spatial position of a 3D target can be converted into a set of coordinate points by the following formula:

[0074]

[0075] Where cor3d represents the coordinates of each corner point or the center point in the coordinate point set, l, w, h are the length, width, and height of the target detection box of the 3D target, respectively, and c x ,c y ,c z is the center point coordinate of the target detection box of the 3D target, and Ro is the transformation matrix, as shown below:

[0076]

[0077] Where r is the yaw angle of the target detection box of the 3D target.

[0078] Furthermore, based on the set of coordinate points of the 3D target, the 3D target can be converted from the 3D space to the camera coordinate system, thereby obtaining the 3D frame coordinate points of the 3D target in the camera coordinate system. The 3D frame coordinate points may include the coordinates of each corner point and the center point coordinates of the target detection frame of the 3D target in the camera coordinate system. Exemplarily, the 3D frame coordinate points of the 3D target in the camera coordinate system can be determined by the following formula:

[0079]

[0080] Where cam3d is the 3D frame coordinate point of the 3D target in the camera coordinate system, and RT is the rotation and translation matrix, which can be expressed by the following formula:

[0081]

[0082] Where R and T represent the rotation matrix and translation matrix of the external parameters respectively.

[0083] After obtaining the 3D frame coordinate points of the 3D target in the camera coordinate system, the predicted distance between the center of the 3D target and the center of the vehicle can be further determined by the 3D frame coordinate points in the camera coordinate system, as shown in the following formula:

[0084] distance = sqrt(cam3d cx 2 +cam3d cy 2 );

[0085] In the formula, cam3d cx cam3d is the horizontal distance of the center point in the 3D frame coordinate point in the camera coordinate system. cy It is the longitudinal distance of the center point in the 3D frame coordinate point in the camera coordinate system.

[0086] Through the above steps 111-112, the 3D target can be converted from the 3D space to the camera coordinate system, and the predicted distance between the center of the 3D target and the center of the vehicle can be determined in the camera coordinate system. Compared with directly determining the distance between the 3D target and the vehicle in the 3D space, the accuracy of the distance prediction can be guaranteed.

[0087] After determining the predicted distance between the center of each 3D object and the center of the vehicle, further, all 3D objects may be sorted in order of predicted distance from small to large, that is, in order of 3D objects from near to far.

[0088] Furthermore, in step 12, the 3D target may be matched with each unmatched 2D target in a sorted order, starting from the first 3D target to the last 3D target, based on the spatial position of the 3D target and the detection frame position of the unmatched 2D target, to determine the degree of matching.

[0089] For example, the 2D object may be mapped to the 3D space to determine the matching degree between the 3D object and the 2D object, or the 3D object may be mapped to the camera image to determine the matching degree between the 3D object and the 2D object.

[0090] With respect to the above step 12, in one example, determining the degree of match between the 3D object and the unmatched 2D object based on the spatial position of the 3D object and the detection frame position of the unmatched 2D object includes the following steps:

[0091] Step 121: Based on the 3D frame coordinate points of the 3D target in the camera coordinate system, project the 3D target from the camera coordinate system onto the camera image to obtain a projection frame circumscribed rectangle of the 3D target;

[0092] Step 122: Determine the degree of match between the 3D object and the unmatched 2D object based on the projection frame circumscribed rectangle of the 3D object and the detection frame position of the unmatched 2D object.

[0093] Among them, in step 121, the 3D target can be further projected from the camera coordinate system to the camera image according to the 3D frame coordinate points of the 3D target in the camera coordinate system to obtain the circumscribed rectangle of the projection frame; the circumscribed rectangle of the projection frame can be understood as the circumscribed rectangle of the projection frame of the 3D target projected onto the camera image.

[0094] For example, the projection box circumscribed rectangle of the 3D target can be obtained by the following formula:

[0095]

[0096] In the formula, pobj is the external rectangle of the projection frame, K represents the internal parameter matrix, u0, v0 represent the center offset, f x 、f y Indicates the focal length of the camera.

[0097] After obtaining the projection frame circumscribed rectangle of the 3D target according to the sorting order of the predicted distances of the 3D targets, in step 122, the matching degree between the 3D target and the unmatched 2D target can be determined according to the projection frame circumscribed rectangle of the 3D target and the detection frame position of the unmatched 2D target.

[0098] For example, the intersection-and-union ratio between the circumscribed rectangle of the projection frame of the 3D target and the target detection frame of the unmatched 2D target may be calculated, and the intersection-and-union ratio may be used as the matching degree, as shown in the following formula:

[0099]

[0100] Where IOU is the union-intersection ratio between the circumscribed rectangle of the projection box of the 3D target and the target detection box of the 2D target, that is, the matching degree, and box2d is the target detection box of the 2D target.

[0101] Through the above steps 121-122, the 3D target can be first projected onto the camera image, and then the matching degree between the target detection frame of the unmatched 2D target can be determined through the circumscribed rectangle of the projection frame obtained by projection, so that the overlap between the 3D target and the 2D target can be determined on the same plane, thereby ensuring the accuracy of the matching process.

[0102] It should be noted that, in the embodiment of the present application, taking into account the situation that some 2D targets in the camera image are occluded, in order to ensure that targets that are occluded too much participate in the matching and affect the matching accuracy and efficiency, it is also possible to determine whether the current 3D target is occluded in the camera image before calculating the matching degree between the 3D target and the unmatched 2D target, that is, after obtaining the circumscribed rectangle of the projection frame of the current 3D target, and if so, not match it.

[0103] Optionally, before determining the degree of matching between the 3D object and the unmatched 2D object based on the projection frame circumscribed rectangle of the 3D object and the detection frame position of the unmatched 2D object, the following steps are further included:

[0104] Step 21, determining the occlusion rate of the 3D target in the camera image based on the circumscribed rectangle of the projection frame of the 3D target;

[0105] Step 22: If the occlusion rate is less than a preset occlusion threshold, an operation of determining a degree of matching between the 3D target and the unmatched 2D target based on the projection frame circumscribed rectangle of the 3D target and the detection frame position of the unmatched 2D target is performed.

[0106] In step 21, the occlusion rate of the 3D target in the camera image can be determined by combining the circumscribed rectangle of the projection frame of the 3D target with the target detection frame of the 2D target in the camera image.

[0107] With respect to the above step 21, in some implementations, determining the occlusion rate of the 3D object in the camera image based on the circumscribed rectangle of the projection frame of the 3D object includes the following steps:

[0108] Step 211: Obtain an occupancy mask image, wherein the occupancy mask image is constructed based on the camera image;

[0109] Step 212: Using the circumscribed rectangle of the projection frame of the 3D target as prompt information of the image segmentation model, segmenting the 2D target corresponding to the 3D target in the camera image based on the image segmentation model to obtain a segmentation mask image corresponding to the 3D target;

[0110] Step 213: determine the occlusion rate of the 3D object in the camera image based on the occupancy mask image and the segmentation mask image, and update the occupancy mask image according to the segmentation mask image.

[0111] The occupancy mask image can be used to maintain the current invisible area under the viewing angle corresponding to the camera image. Specifically, an image with the same size as the camera image can be pre-established, and the pixel values ​​of all pixels in the established image are set to 0 (i.e., the image appears black), to obtain an initialized occupancy mask image.

[0112] Specifically, the circumscribed rectangle of the projection box of the 3D target can be used as prompt information and input into the image segmentation model together with the camera image, wherein the image segmentation model can be a SAM (Segment Anything Model) model. Through the image segmentation model, the 2D target corresponding to the 3D target in the camera image can be segmented and the corresponding segmentation mask image can be output.

[0113] The segmentation mask image may describe the segmentation region of the 2D object corresponding to the 3D object. If the 2D object is blocked, the pixel value of each pixel in the blocked part of the 2D object in the segmentation region is 0.

[0114] Furthermore, the overlap between the occupied mask image and the segmented mask image can be calculated, and the overlap can be the area where the 2D target corresponding to the 3D target is blocked in the camera image; then the proportion of the overlap in the segmented mask image is calculated, that is, the ratio between the local area where the 2D target is blocked and the segmented area of ​​the 2D target, and the occlusion rate of the 3D target in the camera image is obtained. As shown in the following formula:

[0115]

[0116] In the formula, I occ is the occlusion rate, mask is the segmentation mask image, and occmask is the occupancy mask image.

[0117] It should be noted that when calculating the occlusion rate, the occupancy mask image also needs to be updated according to the segmentation mask image. For example, the union between the segmentation mask image and the occupancy mask image is obtained, and the union is used as the new occupancy mask image:

[0118] occmask = mask ∪ occmask;

[0119] In the formula, the union between the segmentation mask image and the occupation mask image is obtained, which can be understood as replacing the pixel values ​​in the segmentation area of ​​the 2D target with the pixel values ​​in the area corresponding to the segmentation area of ​​the 2D target in the occupation mask image.

[0120] The purpose of updating the occupancy mask image is that the occupancy mask image maintains the current invisible area. Therefore, after calculating the occupancy rate of a 3D target, the occupancy mask image needs to be updated to remove the 2D target corresponding to the 3D target from the invisible area, so as to avoid the occlusion of other 3D targets in the subsequent calculation of the occlusion rate of the previous 3D targets. The numerator in the occlusion rate formula becomes larger due to the occluded parts.

[0121] Through the above steps 211 to 213, the occlusion rate of the 3D target can be calculated by using the projection frame circumscribed rectangle of the current 3D target, combined with the occupation mask image and the image segmentation model, to ensure the accuracy of the occlusion rate calculation. In addition, in the process of calculating the occlusion rate, the projection frame circumscribed rectangle used can also be used for subsequent matching degree calculation, so that there is no need to generate other information, which can improve the efficiency of judging whether the 3D target is occluded.

[0122] In an embodiment of the present application, the projection frame circumscribed rectangle of the 3D target is used as prompt information of the image segmentation model, and a segmentation mask image corresponding to the 3D target is obtained based on the image segmentation model. Specifically, in addition to being used to determine the occlusion rate, the segmentation mask image can also be used to correct the projection frame circumscribed rectangle. For example, the projection frame circumscribed rectangle is updated using the minimum circumscribed rectangle of the segmentation mask image.

[0123] For example, Figure 2 is a schematic diagram of updating the circumscribed rectangle of a projection frame provided in an embodiment of the present application, such as Figure 2 As shown, for a target, the enclosing rectangle of its projection frame can be updated to the minimum enclosing rectangle of the segmentation mask image.

[0124] By updating the projection frame circumscribed rectangle by segmenting the mask image, the matching success rate between the corrected projection frame circumscribed rectangle and the detection frame of the 2D target can be higher, which is beneficial to the matching of truncated targets and large targets.

[0125] After obtaining the occlusion rate of the current 3D target in the camera image, further, in step 22, it is determined whether the occlusion rate is less than a preset occlusion threshold. If it is less than the preset occlusion threshold, it means that the 2D target corresponding to the 3D target is relatively small in the camera image. It can be considered that the 3D target is not occluded in the camera image, and subsequent matching continues.

[0126] If the occlusion rate is greater than or equal to the preset occlusion threshold, it means that the 2D target corresponding to the 3D target is partially occluded in the camera image. It can be confirmed that the 3D target is occluded in the camera image. Continuing to match may lead to mismatching or matching failure. Therefore, the next 3D target can be directly projected and matched according to the sorting order of the predicted distance.

[0127] Through the above steps 21 and 22, before matching the current 3D target, it can be determined whether it is blocked in the camera image. If so, the matching can be canceled to avoid wrong matching or matching failure. At the same time, the matching efficiency can be improved.

[0128] Specifically, for the current 3D object, in addition to obtaining the matching degree between the 3D object and the unmatched 2D object, it is also necessary to determine the corresponding matching threshold according to the spatial position of the 3D object.

[0129] In a specific implementation, determining a matching threshold corresponding to the 3D object based on the spatial position of the 3D object includes:

[0130] Obtain the preset farthest perception distance, minimum matching threshold and maximum matching threshold; determine the matching threshold corresponding to the 3D target based on the predicted distance between the center of the 3D target and the center of the vehicle, the farthest perception distance, the minimum matching threshold and the maximum matching threshold.

[0131] The farthest perception distance may be a distance threshold set for the camera to ensure perception accuracy. The minimum matching threshold may be a minimum matching threshold for determining the matching degree between a 3D target and a 2D target, and the maximum matching threshold may be a maximum matching threshold for determining the matching degree between a 3D target and a 2D target.

[0132] Exemplarily, the matching threshold corresponding to the 3D target is determined based on the predicted distance, the farthest perceived distance, the minimum matching threshold, and the maximum matching threshold. The size of the 2D target corresponding to the 3D target in the camera image can be described by the ratio between the predicted distance and the farthest perceived distance, and the matching threshold is determined in combination with the minimum matching threshold and the maximum matching threshold, for example:

[0133] thresh=max(min_thresh,max_thresh-0.5×distance / max_disdance);

[0134] Where thresh is the matching threshold corresponding to the 3D target, min_thresh is the minimum matching threshold, max_thresh is the maximum matching threshold, max_distance is the farthest perceived distance, and distance is the predicted distance.

[0135] Through the above implementation, combined with the fact that 2D targets in the camera image conform to the rule of perspective projection that they are larger when closer and smaller when farther away, a smaller matching threshold can be given in the above manner for targets that appear larger in the camera image. For targets that appear smaller in the camera image, considering that the circumscribed rectangle of the projection frame is affected by the projection parameters in actual applications, the bumps during vehicle driving or inaccurate calibration of projection parameters may make it difficult to match distant targets. Therefore, a larger matching threshold can be given in the above manner to achieve adaptive adjustment of the matching threshold with distance to ensure the accuracy and reliability of matching.

[0136] After obtaining the matching threshold, further, the 2D target that matches the 3D target can be determined based on the matching threshold corresponding to the 3D target and the matching degree between the 2D targets that are not matched. It should be noted that after determining the 2D target that matches the 3D target, the matching state of the 2D target that matches the 3D target can be updated from unmatched to matched, so that when the next 3D target is matched in the sorting order, the matched 2D target is not included in the matching degree calculation, which can improve the matching efficiency and the matching accuracy.

[0137] S130 : Determine a 2D object that matches the 3D object based on a matching threshold and a matching degree corresponding to the 3D object.

[0138] Specifically, each 2D target whose matching degree is greater than the matching threshold can be screened out. If there is a 2D target whose matching degree is greater than the matching threshold, and the number is one, the 2D target can be directly determined as the 2D target matching the 3D target. If the number is multiple, the 2D target with the highest matching degree can be selected as the 2D target matching the 3D target.

[0139] If there is no 2D target with a matching degree greater than the matching threshold, it can be determined that the 3D target matching has failed, and then the next 3D target can be matched in the sorting order. Before matching, it can also be determined whether it is blocked based on the circumscribed rectangle of the projection frame.

[0140] In an embodiment of the present application, after determining the 2D target that matches the 3D target, the detection category of the 3D target can be updated according to the detection category of the 2D target, or the target detection frame of the matched 2D target can be used for 3D detection auxiliary training, or it can be used for matching continuous frames, or the tracking trajectory information can be used to update the historical frames and future frames through the optimal results in the matching.

[0141] In a specific implementation, while determining the detection frame position of each 2D target in the camera image based on the trained 2D detection model and determining the spatial position of each 3D target in the point cloud data based on the trained 3D detection model, it also includes:

[0142] Determine the detection category of each 2D target in the camera image based on the 2D detection model, and determine the detection category of each 3D target in the point cloud data based on the 3D detection model;

[0143] After determining the 2D target that matches the 3D target, it also includes:

[0144] Based on the detection category of the 2D target that matches the 3D target, the detection category of the 3D target is updated; and / or, for the 3D target that is not matched with the 2D target, the point cloud data and the spatial position of the 3D target are associated with the detection category and stored in a visual database, wherein the visual database is used for training the visual detection model.

[0145] For a 3D target that has been matched to a 2D target, the detection category of the 3D target may be updated according to the detection category of the 2D target.

[0146] For 3D targets that are not matched to 2D targets, the spatial position, detection category, and point cloud data of such 3D targets can be associated and stored for subsequent model training of pure vision solutions.

[0147] Through the above implementation, 3D targets that are matched to 2D targets and 3D targets that are not matched to 2D targets can be fully utilized to provide support for category updating and pure vision model training.

[0148] The embodiment of the present application provides a method for matching 2D targets and 3D targets. The method determines the detection frame position of each 2D target in the camera image based on the trained 2D detection model, and determines the spatial position of each 3D target in the point cloud data according to the trained 3D detection model. Then, for each 3D target, the matching degree is determined by the spatial position of the 3D target and the detection frame position of each 2D target, and the matching threshold corresponding to the 3D target is determined by the spatial position of the 3D target. Finally, the 2D target matching the 3D target is determined by the matching threshold corresponding to the 3D target and the matching degree with each 2D target, thereby realizing automatic matching between the 2D target and the 3D target. The method can determine the corresponding matching threshold based on the spatial position of the 3D target, realizes adaptive adjustment of the threshold based on the spatial position, and can dynamically adjust the matching threshold for targets of different sizes presented at different distances, which is beneficial to improving the accuracy of matching of long-distance targets, ensuring the reliability of matching of close-range targets, reducing the occurrence of false matching and missed matching, and thus helping to improve the quality of automatic labeling.

[0149] Figure 3 A schematic diagram of a matching process of a 2D target and a 3D target provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the camera image is passed through the 2D detection model to obtain a 2D detection result, and the lidar point cloud is passed through the 3D detection model to obtain a 3D detection result. It should be noted that the camera image and the millimeter-wave radar can be input into the 3D detection model as optional data.

[0150] Furthermore, a matching state table may be established for each camera image of each viewing angle, and the matching state table is used to describe the matching state of each 2D object in the camera image.

[0151] For the 3D detection results, each 3D target can be sorted according to the center distance, so as to judge whether it participates in subsequent matching based on the circumscribed rectangle of the projection box of the 3D target combined with the SAM segmentation model. If so, the adaptive threshold (i.e. the corresponding matching threshold) is further determined, and the unmatched 2D targets are queried from the matching status table. The matching degree between the 3D match and the adaptive threshold is calculated to achieve the matching of the 3D target and the 2D target. After the matching is completed, the matching status of the matched 2D target in the matching status table is updated.

[0152] The method provided in the embodiments of the present application has at least the following technical effects: 1. By matching 2D and 3D dynamic targets, the category label of the dynamic target detected in 3D can be corrected. At the same time, due to the different installation positions of the sensors, from the perspective of the image, the perspective occlusion information and more detailed labeling rules can be obtained from the 2D detection; 2. By generating the spatially sorted SAM occupancy mask and the adaptive threshold based on the spatial position, the accuracy of matching targets with different occlusions, different distances and different sizes can be effectively improved, and false matches and missed matches can be reduced, thereby improving the quality of automatic labeling, which is beneficial to both pure vision and multimodal model solutions for autonomous driving, and is conducive to promoting the development of automatic labeling and automated driving.

[0153] Figure 4 A schematic diagram of a 2D target and a 3D target matching device provided in an embodiment of the present application, the device includes a model output module 310, a matching degree calculation module 320 and a matching execution module 330, wherein:

[0154] A model output module 310, configured to determine the position of a detection frame of each 2D object in the camera image based on the trained 2D detection model, and to determine the spatial position of each 3D object in the point cloud data based on the trained 3D detection model;

[0155] A matching degree calculation module 320 is used to determine, for each of the 3D objects, a matching degree between the 3D object and each 2D object according to a spatial position of the 3D object and a detection frame position of each 2D object, and determine a matching threshold corresponding to the 3D object based on the spatial position of the 3D object;

[0156] The matching execution module 330 is configured to determine a 2D object that matches the 3D object based on a matching threshold corresponding to the 3D object and the matching degree.

[0157] Based on the above implementation, optionally, the matching degree calculation module 320 is specifically used for:

[0158] Determine a predicted distance between the center of each 3D object and the center of the vehicle based on the spatial position of each 3D object, and sort all 3D objects in ascending order of the predicted distance; determine a matching degree between the 3D object and the unmatched 2D object based on the spatial position of the 3D object and the detection frame position of the unmatched 2D object for each 3D object in the sorting result in the sorting order;

[0159] Correspondingly, the matching execution module 330 is further configured to update the matching status of the 2D object that matches the 3D object from unmatched to matched.

[0160] Based on the above implementation, optionally, the matching degree calculation module 320 includes a camera coordinate system conversion unit, which is used to convert the 3D target from the 3D space to the camera coordinate system based on the spatial position of each 3D target, and obtain the 3D frame coordinate point of the 3D target in the camera coordinate system; based on the 3D frame coordinate point in the camera coordinate system, determine the predicted distance between the center of the 3D target and the center of the vehicle.

[0161] On the basis of the above implementation, optionally, the matching degree calculation module 320 includes a projection frame determination unit, which is used to project the 3D target from the camera coordinate system to the camera image based on the 3D frame coordinate points of the 3D target in the camera coordinate system to obtain the projection frame circumscribed rectangle of the 3D target; based on the projection frame circumscribed rectangle of the 3D target and the detection frame position of the unmatched 2D target, determine the matching degree between the 3D target and the unmatched 2D target.

[0162] Based on the above implementation, optionally, the projection frame determination unit is further used to determine the occlusion rate of the 3D target in the camera image based on the projection frame circumscribed rectangle of the 3D target; if the occlusion rate is less than a preset occlusion threshold, an operation is performed to determine the degree of matching between the 3D target and the unmatched 2D target based on the projection frame circumscribed rectangle of the 3D target and the detection frame position of the unmatched 2D target.

[0163] On the basis of the above implementation, optionally, the projection frame determination unit is further used to obtain an occupancy mask image, wherein the occupancy mask image is constructed based on the camera image; using the circumscribed rectangle of the projection frame of the 3D target as prompt information of the image segmentation model, and segmenting the 2D target corresponding to the 3D target in the camera image based on the image segmentation model to obtain a segmentation mask image corresponding to the 3D target; determining the occupancy rate of the 3D target in the camera image based on the occupancy mask image and the segmentation mask image, and updating the occupancy mask image according to the segmentation mask image.

[0164] Based on the above implementation, optionally, the matching degree calculation module 320 includes an adaptive threshold unit, which is used to obtain a preset farthest perception distance, a minimum matching threshold and a maximum matching threshold; based on the predicted distance between the center of the 3D target and the center of the vehicle, the farthest perception distance, the minimum matching threshold and the maximum matching threshold, determine the matching threshold corresponding to the 3D target.

[0165] Based on the above implementation, optionally, the model output module 310 is further used to determine the detection category of each 2D target in the camera image based on the 2D detection model, and determine the detection category of each 3D target in the point cloud data based on the 3D detection model;

[0166] Accordingly, the apparatus further comprises a post-processing module, the post-processing module being configured to update the detection category of the 3D target based on the detection category of the 2D target matching the 3D target; and / or

[0167] For 3D targets that are not matched with 2D targets, the point cloud data and the spatial position of the 3D target are associated with the detection category and stored in a visual database, wherein the visual database is used for training a visual detection model.

[0168] The 2D target and 3D target matching device provided in the embodiment of the present application can execute the steps of the 2D target and 3D target matching method provided in the method embodiment of the present application, and the execution steps and beneficial effects are no longer repeated here.

[0169] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device 400 includes one or more processors 401 and a memory 402 .

[0170] The processor 401 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 400 to perform desired functions.

[0171] The memory 402 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 401 may run the program instructions to implement the matching method of the 2D target and the 3D target of any embodiment of the present application described above and / or other desired functions. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0172] In one example, the electronic device 400 may further include: an input device 403 and an output device 404, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown). The input device 403 may include, for example, a keyboard, a mouse, etc. The output device 404 may output various information to the outside, including early warning information, braking force, etc. The output device 404 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0173] Of course, to simplify, Figure 5 Only some of the components related to the present application in the electronic device 400 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 400 may also include any other appropriate components.

[0174] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the 2D target and 3D target matching method provided in any embodiment of the present application.

[0175] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0176] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the method for matching a 2D target with a 3D target provided in any embodiment of the present application.

[0177] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0178] It should be noted that the terms used in this application are only for describing specific embodiments, rather than limiting the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method or device including the elements.

[0179] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0180] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and its core ideas of this application. The above is only the preferred implementation method of this application. It should be pointed out that due to the limitations of textual expression and the objective existence of infinite specific structures, ordinary technicians in this technical field can make several improvements, modifications or changes without departing from the principles of this application, and can also combine the above technical features in an appropriate manner; these improvements, modifications, changes or combinations, or the direct application of the concept and technical solution of the invention to other occasions without improvement, should be regarded as the scope of protection of this application.

Claims

1. A method for matching a 2D object with a 3D object, characterized in that: include: Determine the detection box position of each 2D target in the camera image based on the trained 2D detection model, and determine the spatial position of each 3D target in the point cloud data based on the trained 3D detection model; For each of the 3D objects, determining a matching degree between the 3D object and each 2D object according to a spatial position of the 3D object and a detection frame position of each 2D object, and determining a matching threshold corresponding to the 3D object based on the spatial position of the 3D object; Based on the matching threshold corresponding to the 3D object and the matching degree, a 2D object matching the 3D object is determined.

2. The method according to claim 1, characterized in that For each of the 3D objects, determining a matching degree between the 3D object and each 2D object according to a spatial position of the 3D object and a detection frame position of each 2D object, including: Determine the predicted distance between the center of each 3D object and the center of the vehicle based on the spatial position of each 3D object, and sort all the 3D objects in order of predicted distance from small to large; For each 3D object in the sorting results, according to the sorting order, based on the spatial position of the 3D object and the position of the detection frame of the unmatched 2D object, determining a matching degree between the 3D object and the unmatched 2D object; After determining a 2D object matching the 3D object based on the matching threshold corresponding to the 3D object and the matching degree, the method further includes: The matching state of the 2D object matched with the 3D object is updated from unmatched to matched.

3. The method according to claim 2, characterized in that Determining a predicted distance between a center of each of the 3D objects and a center of the vehicle based on a spatial position of each of the 3D objects includes: For each of the 3D objects, based on the spatial position of the 3D object, the 3D object is converted from the 3D space to the camera coordinate system to obtain the 3D frame coordinate point of the 3D object in the camera coordinate system; The predicted distance between the center of the 3D object and the center of the vehicle is determined based on the 3D box coordinate points in the camera coordinate system.

4. The method according to claim 3, characterized in that Determining a degree of matching between the 3D object and the unmatched 2D object based on the spatial position of the 3D object and the detection frame position of the unmatched 2D object includes: Based on the 3D frame coordinate points of the 3D target in the camera coordinate system, projecting the 3D target from the camera coordinate system onto the camera image to obtain a projection frame circumscribed rectangle of the 3D target; Based on the projection frame circumscribed rectangle of the 3D object and the detection frame position of the unmatched 2D object, a matching degree between the 3D object and the unmatched 2D object is determined.

5. The method according to claim 4, characterized in that Before determining the degree of matching between the 3D object and the unmatched 2D object based on the projection frame circumscribed rectangle of the 3D object and the detection frame position of the unmatched 2D object, the method further includes: Determine an occlusion rate of the 3D object in the camera image based on a circumscribed rectangle of a projection frame of the 3D object; If the occlusion rate is less than a preset occlusion threshold, an operation of determining a degree of matching between the 3D object and the unmatched 2D object based on a projection frame circumscribed rectangle of the 3D object and a detection frame position of the unmatched 2D object is performed.

6. The method according to claim 5, characterized in that Determining the occlusion rate of the 3D object in the camera image based on the circumscribed rectangle of the projection frame of the 3D object includes: obtaining an occupancy mask image, wherein the occupancy mask image is constructed based on the camera image; Using the circumscribed rectangle of the projection frame of the 3D target as prompt information of an image segmentation model, segmenting a 2D target corresponding to the 3D target in the camera image based on the image segmentation model to obtain a segmentation mask image corresponding to the 3D target; An occlusion rate of the 3D object in the camera image is determined based on the occupancy mask image and the segmentation mask image, and the occupancy mask image is updated according to the segmentation mask image.

7. The method according to claim 2, characterized in that Determining a matching threshold corresponding to the 3D object based on the spatial position of the 3D object includes: Get the preset farthest sensing distance, minimum matching threshold and maximum matching threshold; A matching threshold corresponding to the 3D target is determined based on the predicted distance between the center of the 3D target and the center of the vehicle, the farthest perceived distance, the minimum matching threshold, and the maximum matching threshold.

8. The method according to claim 1, characterized in that While determining the detection frame position of each 2D target in the camera image based on the trained 2D detection model and determining the spatial position of each 3D target in the point cloud data based on the trained 3D detection model, it also includes: Determining a detection category of each 2D object in the camera image based on the 2D detection model, and determining a detection category of each 3D object in the point cloud data based on the 3D detection model; After determining the 2D object matching the 3D object, the method further includes: updating the detection category of the 3D object based on the detection category of the 2D object matching the 3D object; and / or, For 3D targets that are not matched with 2D targets, the point cloud data and the spatial position of the 3D target are associated with the detection category and stored in a visual database, wherein the visual database is used for training a visual detection model.

9. An electronic device, characterized in that: The electronic device comprises: Processor and memory; The processor is used to execute the steps of the method for matching a 2D object with a 3D object as described in any one of claims 1 to 8 by calling the program or instruction stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or an instruction, and the program or the instruction enables a computer to execute the steps of the method for matching a 2D object with a 3D object as claimed in any one of claims 1 to 8.

Images