Data matching method, device, electronic device and storage medium

Abstract

The present disclosure provides a data matching method, device, electronic device and storage medium, wherein the method comprises: detecting the point cloud data and image data to be matched respectively, obtaining the target detection result of the point cloud data and the target detection result of the image data; determining the target feature information corresponding to the point cloud data and the image data respectively according to the target detection result of the point cloud data; determining the target feature information corresponding to the image data according to the target detection result of the image data; the target feature information comprises the geometric feature information of the bounding box of the detected target object, and the appearance feature information of the target object in the bounding box; matching the bounding box in the point cloud data with the bounding box in the image data according to the target feature information. The embodiments of the present disclosure can improve the detection accuracy of the 3D target by combining the image data and the point cloud data to detect the 3D target, thereby obtaining a more accurate target detection result.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to a data matching method, device, electronic device, and storage medium. Background Art

[0002] The goal of 3D object detection is to identify the 3D bounding box information of an object, mainly including position, orientation, size, confidence and other information. In recent years, in the field of 3D object detection, single-modal methods based on lidar or camera sensors have made increasing progress. However, due to the characteristics of the data structure, the target detection effect of the image single-modal method is affected by the current environment. Exposed or too dark photos will affect the acquisition of target information, and are also affected by front and back occlusions; point cloud single-modal methods need to solve problems such as sparse, irregular point clouds, lack of texture and semantic information, and too few points for small objects and distant objects.

[0003] Based on this, there is an urgent need for a stable, reliable, accurate and robust multimodal matching algorithm. Summary of the invention

[0004] The embodiments of the present disclosure at least provide a data matching method, device, electronic device and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a data matching method, comprising: respectively detecting point cloud data and image data to be matched, and obtaining target detection results of the point cloud data and target detection results of the image data; wherein the target detection results include bounding box information of the detected target object; determining target feature information corresponding to the point cloud data based on the target detection results of the point cloud data; determining target feature information corresponding to the image data based on the target detection results of the image data; the target feature information includes geometric feature information of the bounding box of the detected target object, and appearance feature information of the target object within the bounding box; and matching the bounding box determined based on the point cloud data and the bounding box determined based on the image data based on the target feature information.

[0006] In the above implementation, by combining image data and point cloud data for target detection, the point cloud data can make up for the defect that image data is easily affected by illumination and occlusion, and the image data can make up for the defect that point cloud data is sparse and textureless. Therefore, combining image data and point cloud data to detect 3D targets can improve the detection accuracy of 3D targets, thereby obtaining more accurate target detection results.

[0007] In an optional embodiment, determining target feature information corresponding to the point cloud data based on target detection results of the point cloud data includes: projecting the target detection results of the point cloud data into the image data to obtain a projection frame of a bounding box corresponding to the point cloud data; and determining geometric feature information of the point cloud data based on pixel coordinates of vertices of the projection frame in the image data.

[0008] In the above embodiment, by projecting the 3D bounding box corresponding to the point cloud data into the image data to obtain a 2D projection box, the data format can be unified, so that the target object can be quickly matched according to the relative position between the 2D bounding box corresponding to the image data and the 2D projection box, thereby obtaining the corresponding matching result.

[0009] In an optional implementation, the geometric feature information includes position information and / or size information of the bounding box.

[0010] In the above embodiment, by expanding the position information and / or size information of the bounding box into the geometric feature information, the geometric feature information can be enriched, and the accuracy of data matching is further improved.

[0011] In an optional embodiment, determining the target feature information corresponding to the image data based on the target detection result of the image data includes: determining the target image data located within the detected bounding box in the image data; extracting image features of the target image data, and determining the extracted image features as the appearance feature information corresponding to the image data.

[0012] In the above implementation, by extracting the image features of the target image data in the bounding box corresponding to the image data, it is possible to further verify whether the objects in the bounding boxes with matching positions are the same object based on the appearance feature information on the basis of matching the bounding boxes based on the position information. Through the above processing method, the matching errors caused by the weak synchronization between the image data and the point cloud data can be compensated, thereby improving the accuracy of data matching and further improving the safety factor of autonomous driving.

[0013] In an optional embodiment, determining the target feature information corresponding to the point cloud data based on the target detection result of the point cloud data includes: determining the target point cloud data located within the detected bounding box in the point cloud data; wherein the target point cloud data includes: the number of target point clouds located within the bounding box and / or the coordinate information of the target point clouds in the point cloud coordinate system; based on the target point cloud data, determining the global point cloud features used to describe the overall features of the target point cloud, and determining the appearance feature information corresponding to the point cloud data based on the global point cloud features.

[0014] In the above implementation, by extracting the global point cloud features of the target point cloud data in the bounding box corresponding to the point cloud data, it is possible to further verify whether the objects in the bounding boxes with matching positions are the same object based on the appearance feature information on the basis of matching the bounding boxes according to the position information. Through the above processing method, the matching errors caused by the weak synchronization between the image data and the point cloud data can be compensated, thereby improving the accuracy of data matching and further improving the safety factor of autonomous driving.

[0015] In an optional embodiment, matching the bounding box determined based on the point cloud data and the bounding box determined based on the image data according to the target feature information includes: performing a correlation calculation on the target feature information of the image data and the target feature information of the point cloud data to obtain a correlation calculation result; and determining a matching result between the bounding box in the point cloud data and the bounding box in the image data according to the correlation calculation result.

[0016] In the above implementation, by combining geometric image information and appearance feature information, the correlation calculation is performed on the target feature information of the point cloud data and the target feature information of the image data. The correlation between each feature information corresponding to the M 2D bounding boxes and each feature information corresponding to the N 3D bounding boxes can be accurately determined, so that when the bounding boxes are matched according to the correlation calculation results, the matching accuracy of the bounding boxes is improved.

[0017] In an optional embodiment, the correlation calculation is performed on the target feature information of the image data and the target feature information of the point cloud data to obtain the correlation calculation result, including: splicing the geometric feature information corresponding to the image data and the appearance feature information corresponding to the image data to obtain the target image feature; splicing the geometric feature information corresponding to the point cloud data and the appearance feature information corresponding to the point cloud data to obtain the target point cloud feature; performing correlation operation on the target image feature and the target point cloud feature to obtain the correlation calculation result.

[0018] In the above implementation, the geometric feature information and the appearance feature information are spliced ​​to obtain the corresponding target image features and target point cloud features, and then the target image features and the target point cloud features are correlated, so as to determine the matching result of the bounding box according to the correlation calculation result. This can make up for the matching errors caused by the weak synchronization between the image data and the point cloud data, thereby improving the accuracy of data matching and further improving the safety factor of autonomous driving.

[0019] In an optional embodiment, determining the matching result between the bounding box in the point cloud data and the bounding box in the image data based on the correlation calculation result includes: performing a convolution calculation on the correlation calculation result to obtain a similarity matrix, wherein the similarity matrix is ​​used to characterize the degree of similarity between the bounding box in the point cloud data and the bounding box in the image data; performing an inverse calculation on the similarity matrix to obtain a matching cost matrix; and performing a bipartite graph matching process on the matching cost matrix to obtain a matching result between the bounding box in the point cloud data and the bounding box in the image data.

[0020] In the above implementation, by performing operations such as convolution, inversion and bipartite graph matching processing on the above correlation calculation results, the processing efficiency of the matching results can be improved and a matching result with higher accuracy can be obtained.

[0021] In an optional embodiment, the point cloud data and image data to be matched are detected separately to obtain the target detection results of the point cloud data and the target detection results of the image data, including: performing target detection on the point cloud data and the image data separately through a trained single-modal detection model to obtain the target detection results of the point cloud data and the target detection results of the image data.

[0022] In the above implementation, target detection is performed on point cloud data using a trained point cloud unimodal detection model, and target detection is performed on image data using an image unimodal detection model, so that the accuracy of the target detection result can be improved, thereby obtaining more accurate bounding box information that includes the complete target object.

[0023] In an optional embodiment, the unimodal detection model is trained according to the following steps to determine a training sample set containing multiple training samples; wherein each training sample includes: sample image data and sample point cloud data carrying a sample label; target detection is performed on the training sample set by the unimodal detection model to be trained to obtain sample target detection results; a label matching matrix is ​​determined based on the sample target detection results and the sample labels; a function value of a target loss function is calculated based on the label matching matrix, and model parameters of the unimodal detection model are adjusted based on the function value of the target loss function until preset conditions are met to obtain the trained unimodal detection model.

[0024] In the above implementation, by training the unimodal detection model in the manner described above, a unimodal detection model whose processing accuracy meets the accuracy requirements can be obtained. When target detection is performed according to the unimodal detection model, the accuracy of the target detection results can be improved, thereby improving the accuracy of data matching.

[0025] In an optional embodiment, determining a label matching matrix based on the sample target detection result and the sample label includes: calculating an intersection-and-union ratio between at least one predicted bounding box corresponding to the sample image data in the sample target detection result and a labeled bounding box corresponding to the sample image data in the sample label to obtain a first intersection-and-union ratio; and screening the at least one predicted bounding box according to the first intersection-and-union ratio to obtain a target predicted bounding box; calculating an intersection-and-union ratio between at least one predicted bounding box corresponding to the sample point cloud data in the sample target detection result and a labeled bounding box corresponding to the sample point cloud data in the sample label to obtain a second intersection-and-union ratio; and screening the at least one predicted bounding box according to the second intersection-and-union ratio to obtain a target predicted bounding box; matching the target predicted bounding box with the target predicted bounding box to obtain a label matching result, and determining the label matching matrix based on the label matching result.

[0026] In the above implementation, through the above processing method, the predicted bounding box and the predicted bounding box can be accurately matched to obtain a label matching matrix; when the function value of the target loss function is determined according to the label matching matrix, an accurate function value can be obtained, thereby improving the training accuracy of the unimodal detection model.

[0027] In an optional embodiment, determining a training sample set containing multiple training samples includes: acquiring a target tracking data sequence, wherein the target tracking data sequence includes image data and point cloud data acquired at each tracking moment; determining at least one data combination in the target tracking data sequence, wherein each of the data combinations includes: target image data and target point cloud data; a first tracking moment of the target image data and a second tracking moment of the target point cloud data are different, and the time interval between the first tracking moment and the second tracking moment is a preset interval; and using the data in each of the data combinations as the data in each training sample.

[0028] In the above-mentioned implementation mode, the technical solution disclosed in the present invention proposes a method for constructing a multimodal data set of weak synchronization between point clouds and images. This method can simulate the weak synchronization situations that may occur in actual scenarios of autonomous driving. When a single-modal detection model is trained by using the training sample set, the trained single-modal detection model can adapt to the weak synchronization scenarios of the multimodal data set.

[0029] In a second aspect, an embodiment of the present disclosure further provides a data matching device, comprising: an acquisition module: used to respectively detect point cloud data and image data to be matched, and obtain target detection results of the point cloud data and target detection results of the image data; wherein the target detection results include bounding box information of the detected target object; a determination module: used to determine target feature information corresponding to the point cloud data according to the target detection results of the point cloud data; and determine target feature information corresponding to the image data according to the target detection results of the image data; the target feature information includes geometric feature information of the bounding box of the detected target object, and appearance feature information of the target object within the bounding box; a matching module: used to match the bounding box determined based on the point cloud data and the bounding box determined based on the image data according to the target feature information.

[0030] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect, or any possible implementation of the first aspect are performed.

[0031] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0032] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.

[0034] Figure 1 A flow chart of a data matching method provided by an embodiment of the present disclosure is shown;

[0035] Figure 2A flow chart of another data matching method provided by an embodiment of the present disclosure is shown;

[0036] Figure 3 In a data matching method provided in an embodiment of the present disclosure, a flowchart of determining a label matching matrix according to sample target detection results and sample labels;

[0037] Figure 4 A schematic diagram of a data matching device provided by an embodiment of the present disclosure is shown;

[0038] Figure 5 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0040] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0041] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0042] Research has found that in the existing field of autonomous driving, when performing 3D target detection, a single-mode detection method based on a laser radar or camera sensor is usually used, wherein the above-mentioned single-mode detection methods usually include image single-mode methods and point cloud single-mode methods. However, due to the characteristics of the data structure, the target detection effect of the image single-mode method is affected by the current environment. Photos that are too exposed or too dark will affect the acquisition of target information, and are also affected by front and back occlusions; the point cloud single-mode method needs to solve problems such as sparse, irregular point clouds, lack of texture and semantic information, and too few points for small objects and distant objects.

[0043] Based on the above research, the present disclosure provides a data matching method, device, electronic device and storage medium. In the embodiments of the present disclosure, by combining image data and point cloud data for target detection, the defect that image data is easily affected by illumination and occlusion can be compensated by point cloud data, and the defect that point cloud data is sparse and textureless can be compensated by image data. Therefore, combining image data and point cloud data to detect 3D targets can improve the detection accuracy of 3D targets, thereby obtaining more accurate target detection results.

[0044] To facilitate understanding of this embodiment, a data matching method disclosed in the embodiment of the present disclosure is first introduced in detail. The executor of the data matching method provided in the embodiment of the present disclosure is generally an electronic device with certain computing capabilities.

[0045] See also Figure 1 FIG. 1 is a flow chart of a data matching method provided by an embodiment of the present disclosure, wherein the method includes steps S101 to S105, wherein:

[0046] S101: Detect the point cloud data and image data to be matched respectively to obtain target detection results of the point cloud data and the image data; wherein the target detection results include bounding box information of the detected target object.

[0047] In the embodiment of the present disclosure, image data can be collected by a camera device, and point cloud data can be collected by a laser radar sensor, wherein the camera device and the laser radar sensor are sensors pre-installed on a target vehicle. The target vehicle can be a vehicle with an automatic driving function, such as a minibus, a car, etc. The present disclosure does not specifically limit the type of the target vehicle.

[0048] For point cloud data, the target detection result of the point cloud data includes bounding box information of the detected target object, where the bounding box is a 3D bounding box. For example, if the number of target objects is N, then the bounding box information includes relevant information of the 3D bounding boxes of the N target objects.

[0049] For image data, the target detection result of the image data includes bounding box information of the detected target object, where the bounding box is a 2D bounding box. For example, if the number of target objects is M, then the bounding box information includes relevant information of the 2D bounding boxes of the M target objects.

[0050] It should be noted that the N target objects and the M target objects may include the same target objects or different target objects.

[0051] S103: Determine target feature information corresponding to the point cloud data based on the target detection result of the point cloud data; determine target feature information corresponding to the image data based on the target detection result of the image data; the target feature information includes geometric feature information of a bounding box of the detected target object and appearance feature information of the target object within the bounding box.

[0052] In the disclosed embodiment, the target feature information may be the target feature information corresponding to the above-mentioned point cloud data, or may be the target feature information corresponding to the image data.

[0053] For example, the target feature information corresponding to the point cloud data may include the geometric feature information of the bounding box of the detected target object, and the appearance feature information of the target object within the bounding box; the target feature information corresponding to the image data may also include the geometric feature information of the bounding box of the detected target object, and the appearance feature information of the target object within the bounding box.

[0054] In an embodiment of the present disclosure, the appearance feature information may be an attribute feature used to characterize the target object selected by the bounding box, and the attribute feature may be object category information of the target object, wherein the category information is a category label of the target object, such as a vehicle, a pedestrian, etc.

[0055] S105: Matching a bounding box determined based on the point cloud data and a bounding box determined based on the image data according to the target feature information.

[0056] In the disclosed embodiment, by combining image data and point cloud data for target detection, the defect that image data is easily affected by illumination and occlusion can be compensated by point cloud data, and the defect that point cloud data is sparse and textureless can be compensated by image data. Therefore, combining image data and point cloud data to detect 3D targets can improve the detection accuracy of 3D targets, thereby obtaining more accurate target detection results.

[0057] In the embodiments of the present disclosure, in order to solve the technical problem of poor accuracy of target detection in the existing single-modal detection method, an optional implementation method is a multi-modal detection method, where the multi-modal detection method refers to combining image data and point cloud data to detect the object. For example, the multi-modal detection method may include a point cloud projection method, an image back-projection method, and a similarity matrix method.

[0058] The point cloud projection method only considers the prediction results of the point cloud 3D candidate box, and is heavily dependent on the effect of the point cloud unimodal detector. The image projection method only considers the prediction results of the image 2D candidate box, and is heavily dependent on the effect of the image unimodal detector. The similarity matrix method does not explore the multimodal matching problem in the case of weak synchronization.

[0059] The prerequisite for the point cloud projection method, image back-projection method and similarity matrix method described above is strong synchronization between the lidar sensor and the camera device. Therefore, the above technical solution does not take into account the situation of weak synchronization between the lidar sensor and the camera device.

[0060] Here, strong synchronization between the camera device and the lidar sensor means that the acquisition time of the point cloud data and the image data is highly synchronized. It can also be understood that when the point cloud single-mode detector and the image single-mode detector detect the same object, the point cloud projection 2D frame will overlap with the image 2D frame, or the image back-projection 3D frame will overlap with the point cloud 3D frame. However, in the actual scene of autonomous driving, the camera device and the lidar sensor may have very weak synchronization due to response delays or complex road conditions. At this time, weak synchronization will cause synchronization errors between the image data and the point cloud data. Once a synchronization error occurs, according to the transferability of the error, the result of the projection of one party will also produce a corresponding synchronization error with the other party, that is, it can be understood that the bounding box of the object detected by the point cloud data and the bounding box of the object detected by the image data will not overlap. At this time, an erroneous matching result may be generated, and a poor matching effect will bring a worse detection result to multimodal fusion than a single mode.

[0061] Based on this, the present disclosure provides a data matching method, in which, by combining image data and point cloud data for target detection, the defect that image data is easily affected by illumination and occlusion can be compensated by point cloud data, and the defect that point cloud data is sparse and textureless can be compensated by image data. Therefore, combining image data and point cloud data to detect 3D targets can improve the detection accuracy of 3D targets, thereby obtaining more accurate target detection results.

[0062] In the embodiment of the present disclosure, the steps described in the above steps S101 to S105 will be described in detail, and the specific description process is as follows.

[0063] With respect to step S101, the point cloud data and the image data to be matched are detected respectively to obtain the target detection result of the point cloud data and the target detection result of the image data, which specifically includes the following process:

[0064] The trained unimodal detection model is used to perform target detection on the point cloud data and the image data respectively, so as to obtain the target detection result of the point cloud data and the target detection result of the image data.

[0065] Here, the unimodal detection module includes a point cloud unimodal detection model and an image unimodal detection model. The point cloud unimodal detection model is used to perform target detection on point cloud data to obtain corresponding target detection results; the image unimodal detection model is used to perform target detection on image data to obtain corresponding target detection results.

[0066] In the disclosed embodiment, point cloud data is first collected by a laser radar sensor, and image data is collected by a camera device. Then, target detection is performed on the point cloud data by a point cloud unimodal detection model, and target detection is performed on the image data by an image unimodal detection model.

[0067] It should be noted that point cloud unimodal detection models include but are not limited to SECOND, PointPillars, PointRCNN, and PV-RCNN, and image unimodal detection models include but are not limited to RRC, MSCNN, and Cascade R-CNN.

[0068] In the above implementation, target detection is performed on point cloud data using a trained point cloud unimodal detection model, and target detection is performed on image data using an image unimodal detection model, so that the accuracy of the target detection result can be improved, thereby obtaining more accurate bounding box information that includes the complete target object.

[0069] In step S103, after the target detection result is determined, the target feature information corresponding to the point cloud data and the target feature information corresponding to the image data can be determined. Therefore, step S103 can be described as the following process:

[0070] Step S1031, determining geometric feature information and appearance feature information corresponding to the point cloud data according to the target detection result of the point cloud data.

[0071] Step S1032, determining geometric feature information and appearance feature information corresponding to the image data according to the target detection result of the image data.

[0072] Here, there is no order between step S1031 and step S1032. Step S1031 and step S1032 may be performed simultaneously; step S1031 may be performed first and then step S1032; or step S1032 may be performed first and then step S1031. The above steps S1031 and S1032 will be described in detail below.

[0073] With respect to S1032, the geometric feature information corresponding to the image data may be determined through the following process, the specific process being described as follows:

[0074] (1) Determine the bounding box of the target object based on the target detection results of the image data.

[0075] (2) Determine the geometric feature information of the image data based on the bounding box of the target object.

[0076] In the embodiment of the present disclosure, it is assumed that the image data contains M target objects, and the target detection result of the image data includes the 2D bounding boxes of the M target objects. If the bounding box information of each 2D bounding box is recorded as I j , then the bounding box information of M 2D bounding boxes can be recorded as Among them, the bounding box information of each 2D bounding box can be expressed as: I j ={x j1 ,y j1 , x j2 ,y j2}.

[0077] At this time, the bounding box information of each 2D bounding box may be determined as the geometric feature information corresponding to the image data.

[0078] Here, (x j1 ,y j1 ) and (x j2 ,y j2 ) respectively represent the coordinate information of the upper left corner and the lower right corner of each 2D bounding box in the image data in the external reference coordinate system of the camera device.

[0079] With respect to S1031, determining target feature information corresponding to the point cloud data according to the target detection result of the point cloud data, the specific process is described as follows:

[0080] (1) Projecting the target detection result of the point cloud data into the image data to obtain a projection frame of the bounding box corresponding to the point cloud data.

[0081] (2) Determine the geometric feature information of the point cloud data according to the pixel coordinates of the vertices of the projection frame in the image data.

[0082] After performing target detection on the point cloud data through the point cloud unimodal detection model, the target detection result is obtained. Assume that the target detection result includes: 3D bounding boxes of N target objects, if the bounding box information of each 3D bounding box is recorded as P i , then the bounding box information of N 3D bounding boxes can be recorded as Among them, the bounding box information P of each 3D bounding box i ={x i ,y i , z i ,h i , w i , l i ,θ i}.

[0083] Here, x i ,y i , z i represents the coordinates of the center of the 3D bounding box in the laser radar sensor coordinate system, h i , w i , l i Represents the three-dimensional dimensions of the point cloud 3D bounding box, length, width and height, θ i Represents the orientation information of the point cloud 3D bounding box in the bird's-eye view, that is, the rotation angle around the Y axis of the lidar sensor coordinate system.

[0084] After obtaining the target detection result P of the point cloud data, the target detection result corresponding to the above point cloud data can be projected into the image data according to the external reference coordinate system of the camera device and the calibration relationship between the lidar sensor and the camera device to obtain the 2D projection frame of the bounding box corresponding to the point cloud data. in,

[0085] In the embodiment of the present disclosure, in the above 2D projection frame middle, is the pixel coordinates of the upper left corner and the lower right corner of the 2D projection frame in the image data. After the coordinate information of the 2D projection frame is obtained, the coordinate information can be used to determine the geometric feature information of the point cloud data.

[0086] It should be noted that the geometric feature information of the 2D projection frame of the bounding box corresponding to the above point cloud data can be recorded as in, At this point, it can be seen that the geometric feature information The data dimension D in G is 4, including The pixel coordinates of the upper left corner and the lower right corner in the point cloud data are as follows: The size of the geometric feature information of the 2D projection box of the bounding box corresponding to the above point cloud data can be recorded as N×D G .

[0087] Similarly, the geometric feature information of the 2D bounding box of the above image data can be recorded as in, At this point, it can be seen that the geometric feature information The data dimension D in G is 4, including The pixel coordinates of the upper left corner and the lower right corner of the image are as follows: G .

[0088] It should be noted here that the bounding box information in the geometric feature information may be the pixel coordinates of the lower left corner and the upper right corner in addition to the pixel coordinates of the upper left corner and the lower right corner, and the present disclosure does not make any specific limitation on this.

[0089] In the disclosed embodiment, the geometric feature information may further include position information and / or size information of the bounding box.

[0090] Here, after determining the geometric feature information of the point cloud data and the geometric feature information of the image data in the manner described above, the position information and / or size information of the bounding box can also be expanded into the geometric feature information. At this time, the expanded geometric feature information includes the position information and / or size information of the bounding box.

[0091] The following describes the expansion process of geometric feature information in two cases: expansion of geometric feature information of image data and expansion of geometric feature information of point cloud data.

[0092] Case 1: The process of expanding the geometric feature information of image data.

[0093] In the embodiment of the present disclosure, when expanding the geometric feature information of the above-mentioned image data, the pixel coordinates (x c ,y c ) is extended to the above In the G =6, then the expanded

[0094] In addition, the size information (h, w) of the 2D bounding box can also be expanded to the above In the G =8, then the expanded The size information h and w are respectively the height and width size information of the 2D bounding box.

[0095] Case 2: The process of expanding the geometric feature information of point cloud data.

[0096] In the embodiment of the present disclosure, when expanding the geometric feature information of the point cloud data, the pixel coordinates of the center point of the 2D projection frame can be Expanded to the above In the G =6, then the expanded In addition, the size information of the 2D projection frame can also be Also expanded to the above In the G =8, then the expanded Among them, the above size information They are respectively the height and width size information of the 2D projection frame.

[0097] In the above embodiment, by projecting the 3D bounding box corresponding to the point cloud data into the image data to obtain a 2D projection box, the data format can be unified, so that the target object can be quickly matched according to the relative position between the 2D bounding box corresponding to the image data and the 2D projection box, thereby obtaining the corresponding matching result.

[0098] In the above embodiment, by expanding the position information and / or size information of the bounding box into the geometric feature information, the geometric feature information can be enriched, and the accuracy of data matching is further improved.

[0099] With respect to S1032, when the target feature information includes appearance feature information, the target feature information corresponding to the image data is determined according to the target detection result of the image data. The specific process is described as follows:

[0100] (1) Determine target image data located within the detected bounding box in the image data.

[0101] (2) Extracting image features of the target image data, and determining the extracted image features as appearance feature information corresponding to the image data.

[0102] In the embodiment of the present disclosure, M 2D bounding boxes corresponding to the image data are determined in the image data. The determined image is cropped to obtain the target image data; for M 2D bounding boxes, M target image data can be obtained. After that, the cropped target image data can be scaled to obtain M RBG images with uniform pixels of r×r. The scaled target image data can be represented as Image, where The scaled target image data Image is used to represent the pixel value of each pixel point, and the size of the scaled target image data Image is r×r×3. After obtaining the target image data, the target image data Image can be input into a preset image feature extraction network to extract the target image data D A Dimensional image features Then, according to the image features Get the appearance feature information of the target image data Among them, the appearance feature information It can be expressed as a size of M×D A It should be noted that the above-mentioned image feature extraction network includes but is not limited to VGG-Net, ResNet, GoogleNet and other networks that can realize the above-mentioned image feature extraction.

[0103] In the above implementation, by extracting the image features of the target image data in the bounding box corresponding to the image data, it is possible to further verify whether the objects in the bounding boxes with matching positions are the same object based on the appearance feature information on the basis of matching the bounding boxes based on the position information. Through the above processing method, the matching errors caused by the weak synchronization between the image data and the point cloud data can be compensated, thereby improving the accuracy of data matching and further improving the safety factor of autonomous driving.

[0104] With respect to S1031, when the target feature information includes appearance feature information, step: determining the target feature information corresponding to the point cloud data according to the target detection result of the point cloud data, the specific process is described as follows:

[0105] (1) Determine the target point cloud data located within the detected bounding box in the point cloud data; wherein the target point cloud data includes: the number of target point clouds located within the bounding box and / or the coordinate information of the target point clouds in the point cloud coordinate system.

[0106] (2) Based on the target point cloud data, determine global point cloud features for describing overall features of the target point cloud, and determine appearance feature information of the point cloud data based on the global point cloud features.

[0107] In the disclosed embodiment, the above-mentioned laser radar sensor can scan the road conditions within the scanning range, thereby obtaining a number of point cloud data for characterizing the features of objects within the acquisition range. When determining the appearance feature information corresponding to the point cloud data, the target point cloud data located within the 3D bounding box can be determined in the point cloud data, wherein the target point cloud data includes the number L of target point clouds located within the bounding box and / or the coordinate information C of the target point clouds in the point cloud coordinate system (i.e., the laser radar coordinate system), where C=3.

[0108] Here, the extracted target point cloud data can be recorded as PC, where Then, the target point cloud data is input into the point cloud feature extraction network to extract the D A Dimensional global point cloud features Then, according to the global point cloud features Get the appearance feature information of the point cloud data Among them, the appearance feature information It can be expressed as a size of N×D A in vector form.

[0109] It should be noted that the above-mentioned point cloud feature extraction network includes but is not limited to Pointnet, Pointnet++, PointSIFT and other extraction networks that can realize the above-mentioned point cloud feature extraction.

[0110] In the above implementation, by extracting the global point cloud features of the target point cloud data in the bounding box corresponding to the point cloud data, it is possible to further verify whether the objects in the bounding boxes with matching positions are the same object based on the appearance feature information on the basis of matching the bounding boxes according to the position information. Through the above processing method, the matching errors caused by the weak synchronization between the image data and the point cloud data can be compensated, thereby improving the accuracy of data matching and further improving the safety factor of autonomous driving.

[0111] For step S105, according to the target feature information, the bounding box determined based on the point cloud data and the bounding box determined based on the image data are matched, and the specific process is described as follows:

[0112] (1) performing correlation calculation on the target feature information of the image data and the target feature information of the point cloud data to obtain a correlation calculation result.

[0113] (2) Determine a matching result between the bounding box in the point cloud data and the bounding box in the image data according to the correlation calculation result.

[0114] In the embodiment of the present disclosure, after the target feature information is determined, the target feature information of the image data and the target feature information of the point cloud data may be correlated with each other according to a preset correlation algorithm to obtain a correlation calculation result.

[0115] Assume that the target feature information of the image data includes feature information corresponding to M 2D bounding boxes, and the target feature information of the point cloud data includes feature information corresponding to N 3D bounding boxes. In this case, the correlation calculation result can be understood as the correlation between each feature information in the feature information corresponding to the M 2D bounding boxes and each feature information in the feature information corresponding to the N 3D bounding boxes.

[0116] After obtaining the above correlation calculation result, the matching result between the bounding box determined based on the point cloud data and the bounding box determined based on the image data can be determined according to the correlation calculation result.

[0117] Specifically, the matching result indicates whether the bounding box determined based on the point cloud data matches the bounding box determined based on the image data, wherein the matching result may be an N*M matching matrix. When an element in the matching matrix is ​​1, it indicates that the two bounding boxes are matched bounding boxes, and when an element in the matching matrix is ​​0, it indicates that the two bounding boxes are not matched bounding boxes.

[0118] In the above implementation, by combining geometric image information and appearance feature information, the correlation calculation is performed on the target feature information of the point cloud data and the target feature information of the image data. The correlation between each feature information corresponding to the M 2D bounding boxes and each feature information corresponding to the N 3D bounding boxes can be accurately determined, so that when the bounding boxes are matched according to the correlation calculation results, the matching accuracy of the bounding boxes is improved.

[0119] In the embodiment of the present disclosure, in the above steps: performing correlation calculation on the target feature information of the image data and the target feature information of the point cloud data to obtain a correlation calculation result, the specific process is described as follows:

[0120] (1) The geometric feature information corresponding to the image data and the appearance feature information corresponding to the image data are spliced ​​to obtain target image features.

[0121] (2) The geometric feature information corresponding to the point cloud data and the appearance feature information corresponding to the point cloud data are spliced ​​to obtain the target point cloud features.

[0122] (3) performing a correlation operation on the target image features and the target point cloud features to obtain the correlation calculation result.

[0123] In the embodiment of the present disclosure, the geometric feature information of the above-mentioned image data and the appearance feature information of the image data can be spliced ​​to obtain the target image feature of the image data. A Appearance feature vector (Appearance feature information of image data) and size is M×D G The geometric eigenvector of (geometric feature information of image data) to obtain a size of M×(D A +D G )’s image feature vector F img , that is, the target image feature. In the embodiment of the present disclosure, the geometric feature information of the point cloud data and the appearance feature information of the point cloud data can be spliced ​​to obtain the target point cloud feature of the point cloud data. A Appearance feature vector (Appearance feature information of point cloud data) and size is N×D G The geometric eigenvector F pc (geometric feature information of point cloud data) to obtain a size of N×(D A +D G ) point cloud feature vector F pc , that is, the target point cloud features. Afterwards, the target image features and the target point cloud features can be correlated using a preset correlation algorithm. For example, the image feature vector F img and image feature vector F pc Calculate and get N×M×(D A +D G )'s correlation matrix F correlation (That is, the above correlation calculation result).

[0124] In an optional implementation, the preset correlation algorithm may be an algorithm corresponding to any of the following calculation formulas:

[0125]

[0126]

[0127]

[0128]

[0129]

[0130] In the above implementation, the geometric feature information and the appearance feature information are spliced ​​to obtain the corresponding target image features and target point cloud features, and then the target image features and the target point cloud features are correlated, so as to determine the matching result of the bounding box according to the correlation calculation result. This can make up for the matching errors caused by the weak synchronization between the image data and the point cloud data, thereby improving the accuracy of data matching and further improving the safety factor of autonomous driving.

[0131] In the embodiment of the present disclosure, in the above steps: determining the matching result between the bounding box in the point cloud data and the bounding box in the image data according to the correlation calculation result, the specific process is described as follows:

[0132] (1) performing convolution calculation on the correlation calculation result to obtain a similarity matrix, wherein the similarity matrix is ​​used to represent the similarity between the bounding box in the point cloud data and the bounding box in the image data.

[0133] (2) Perform inverse calculation on the similarity matrix to obtain a matching cost matrix.

[0134] (3) Performing bipartite graph matching processing on the matching cost matrix to obtain a matching result between the bounding box in the point cloud data and the bounding box in the image data.

[0135] In the embodiment of the present disclosure, after obtaining the correlation calculation result, the above correlation calculation result (that is, the correlation matrix F correlation ) is input into several two-dimensional convolutional networks for convolution calculation to obtain a similarity matrix of size N×M×1. Each element in the similarity matrix represents the similarity between each 3D bounding box in the N 3D bounding boxes and each 2D bounding box in the M 2D bounding boxes.

[0136] Here, the similarity degree includes: the similarity degree determined based on geometric feature information, and the similarity degree determined based on appearance feature information.

[0137] For example, if the geometric feature information of the nth 3D bounding box and the mth 2D bounding box are highly similar, and the appearance feature information of the objects framed by the nth 3D bounding box and the mth 2D bounding box are highly similar, it can be determined that the nth 3D bounding box and the mth 2D bounding box are matching bounding boxes.

[0138] After obtaining the similarity matrix, the similarity matrix can be inverted to obtain a matching cost matrix; then, the matching cost matrix is ​​subjected to bipartite graph matching to obtain a matching result between the bounding box in the point cloud data and the bounding box in the image data.

[0139] Assume that between two single-modal detection results (i.e., the target detection result of image data and the target detection result of point cloud data), each detection result can only constitute at most one match, and the detection results under the same modality are different from each other.

[0140] Here, each detection result can constitute at most one match, which can be understood as: a 2D bounding box determined based on the image data can be matched to at most one 3D bounding box determined based on the point cloud data.

[0141] At this time, the matching problem of two unimodal detection results can be regarded as a bipartite graph matching problem. Specifically, in an undirected graph, the target detection results of image data and the target detection results of point cloud data can be divided into two subsets, for example, the target detection results of image data are used as one subset, and the target detection results of point cloud data are used as another subset. In each subset, multiple vertices are included, each vertex corresponds to a bounding box, and the vertices in each subset do not intersect each other, and all the vertices associated with the edges in the undirected graph belong to two different sets. For a bipartite graph, the number of matches formed can be different, and the matching goal is to make the two subsets match each other as accurately as possible. Therefore, through the matching algorithm, the similarity matrix is ​​inverted one by one as a matching cost matrix, and then the matching threshold δ is set. The matching cost is higher than δ and does not participate in the matching, and then the final multimodal matching matrix (that is, the matching result between the bounding box in the point cloud data and the bounding box in the image data) is calculated.

[0142] It should be noted that the matching algorithms described above include but are not limited to the Hungarian matching algorithm and the Kuhn-Munkres matching algorithm.

[0143] In the above implementation, by performing operations such as convolution, inversion and bipartite graph matching processing on the above correlation calculation results, the processing efficiency of the matching results can be improved and a matching result with higher accuracy can be obtained.

[0144] In a weakly synchronized multimodal dataset, for geometric feature information, since the objects are matched through the position information of the projection frames of the 2D bounding box and the 3D bounding box, the geometric feature information will also have a weak synchronization problem. In particular, for small objects, the deviation of geometric feature information will lead to a serious decline in matching results, so the existing commonly used IOU similarity matrix matching algorithm cannot solve the multimodal matching problem under weak synchronization. The appearance feature information is always extracted by following the objects in the 3D frame and the 2D frame. Therefore, the appearance feature information will not be affected by the weak synchronization problem, so the appearance feature information helps to correct the errors caused by weak synchronization.

[0145] Based on the above analysis, the data matching method proposed in the embodiment of the present disclosure can be applied to the matching process of many-to-many multimodal data in both strong synchronization and weak synchronization situations.

[0146] In the embodiments of the present disclosure, Figure 2 As shown, a flow chart of another data matching method is also provided, and the method is described in detail as follows:

[0147] (1) Determine the target detection result.

[0148] The image data to be matched is collected by a camera device; and the image data is detected by an image unimodal detection model to obtain a target detection result A1, wherein the target detection result A1 contains the 2D bounding boxes of the M objects contained in the image data.

[0149] The point cloud data to be matched is collected by the laser radar sensor; the point cloud data is detected by the point cloud single-mode detection model to obtain the target detection result A2, where the target detection result A2 contains the 3D bounding boxes of the N objects sensed in the point cloud data.

[0150] (2) Determine the target feature information corresponding to the image data based on the target detection result A1.

[0151] The 2D bounding box of M objects The position information is determined as the geometric feature information in the target feature information corresponding to the image data. The target image data located in each 2D bounding box is determined in the image data, and the image features of the above target image data are extracted through the image feature extraction network, and the extracted image features are determined as the appearance feature information M×D in the target feature information corresponding to the above image data. A .

[0152] (3) Determine the target feature information corresponding to the point cloud data based on the target detection result A2.

[0153] The target detection results corresponding to the point cloud data are projected into the image data to obtain the projection frame of the 3D bounding box corresponding to the point cloud data; the geometric feature information of the point cloud data is determined according to the pixel coordinates of the vertices of the projection frame in the image data.

[0154] Determine the target point cloud data located within the detected 3D bounding box in the point cloud data; wherein the target point cloud data includes: the number of target point clouds located within the 3D bounding box and / or the coordinate information of the target point clouds in the point cloud coordinate system; based on the target point cloud data, determine the global point cloud features used to describe the overall features of the target point cloud, and determine the appearance feature information of the point cloud data based on the global point cloud features.

[0155] (4) Correlation calculation.

[0156] The geometric feature information corresponding to the image data and the appearance feature information corresponding to the image data are spliced ​​to obtain the target image features; the geometric feature information corresponding to the point cloud data and the appearance feature information corresponding to the point cloud data are spliced ​​to obtain the target point cloud features; the target image features and the target point cloud features are correlated to obtain the correlation calculation results.

[0157] (5) Data matching process.

[0158] Perform convolution calculation on the above correlation calculation results to obtain a similarity matrix N×M×1; perform inversion calculation on the above similarity matrix to obtain a matching cost matrix N×M; perform bipartite graph matching processing on the above matching cost matrix to obtain a matching result between the bounding box in the above point cloud data and the bounding box in the above image data, wherein the matching result can be a matching matrix of size N×M.

[0159] From the above description, it can be seen that in the disclosed embodiment, a data matching method is proposed for the weak synchronization situation between the camera device and the lidar sensor due to response delay or complex road conditions. The method uses the bounding boxes predicted by the point cloud unimodal detection model and the image unimodal detection model to first obtain the geometric feature information of the projected 2D bounding box of the 3D bounding box corresponding to the point cloud data and the 2D bounding box corresponding to the image data, and then extracts the appearance feature information in the corresponding bounding box through the point cloud feature extraction network and the image feature extraction network. Finally, the joint features of the geometric feature information and the appearance feature information are used to predict the similarity matrix between the target detection results of the point cloud data and the target detection results of the image data.

[0160] In the embodiment of the present disclosure, before performing target detection on the point cloud data and the image data respectively by using the trained unimodal detection model to obtain the target detection result of the point cloud data and the target detection result of the image data, it is necessary to train the unimodal detection model according to the following steps:

[0161] (1) Determine a training sample set including multiple training samples; wherein each training sample includes: sample image data and sample point cloud data carrying a sample label.

[0162] (2) Performing target detection on the training sample set using the unimodal detection model to be trained to obtain a sample target detection result.

[0163] (3) Determine a label matching matrix based on the sample target detection results and the sample labels.

[0164] (4) Calculate the function value of the target loss function according to the label matching matrix, and adjust the model parameters of the unimodal detection model according to the function value of the target loss function until the preset conditions are met, thereby obtaining the trained unimodal detection model.

[0165] In the embodiment of the present disclosure, when training a unimodal detection model, it is first necessary to construct a training sample set containing multiple training samples, that is, a sample collection of sample image data or sample point cloud data containing sample labels.

[0166] In the disclosed embodiment, the above-mentioned unimodal detection model can be trained to recognize the sample labels in the above-mentioned point cloud data and image data respectively by inputting the above-mentioned training sample set into the unimodal detection model to be trained, so as to obtain the sample target detection result.

[0167] After obtaining the sample target detection result, the label matching matrix can be determined according to the sample label and the sample target detection result. Then, the target loss function is calculated according to the label matching matrix, and the model parameters of the unimodal detection model are adjusted according to the target loss function until the preset conditions are met to obtain the trained unimodal detection model, wherein the preset conditions may be that the number of trainings of the unimodal detection model meets the preset number requirements, and / or the training accuracy of the unimodal detection model meets the preset accuracy requirements.

[0168] It should be noted that the target loss function includes but is not limited to mean square error loss (MSE), absolute error loss (MAE), cross entropy loss (BCE) and other algorithms that can implement the above-mentioned single-modal detection model training.

[0169] It can be seen from the above description that the unimodal detection model includes a point cloud unimodal detection model and an image unimodal detection model. When training the point cloud unimodal detection model and the image unimodal detection model, the image unimodal detection model can be trained based on a sample training set containing sample image data, and the point cloud unimodal detection model can be trained based on a sample training set containing sample point cloud data. The specific training process is as described above and will not be described separately here.

[0170] In the above implementation, by training the unimodal detection model in the manner described above, a unimodal detection model whose processing accuracy meets the accuracy requirements can be obtained. When target detection is performed according to the unimodal detection model, the accuracy of the target detection results can be improved, thereby improving the accuracy of data matching.

[0171] In the embodiments of the present disclosure, Figure 3 As shown, the above steps: determine the label matching matrix according to the sample target detection result and the sample label, and the specific process is described as follows:

[0172] (1) Calculate the intersection-and-union ratio between at least one predicted bounding box corresponding to the sample image data in the sample target detection result and the labeled bounding box corresponding to the sample image data in the sample label to obtain a first intersection-and-union ratio; and filter the at least one predicted bounding box according to the first intersection-and-union ratio to obtain a target predicted bounding box.

[0173] Specifically, in an embodiment of the present disclosure, the sample image data contained in the training sample can be input into an image unimodal detection model to obtain a sample target detection result including at least one predicted bounding box, wherein the at least one predicted bounding box can also be called a predicted 2D bounding box.

[0174] After that, the IOU (Intersection Over Union) between each predicted 2D bounding box and each labeled bounding box in the sample image data is calculated to obtain multiple first IOU values; then, at least one predicted bounding box is screened according to the multiple first IOU values ​​to obtain the target predicted bounding box. The specific screening process is described as follows:

[0175] First, for each predicted 2D bounding box, determine whether the multiple first intersection-and-union ratios satisfy an intersection-and-union ratio greater than or equal to a preset threshold. If it is determined to be included, the predicted 2D bounding box is determined as the target predicted bounding box. At this time, the marked bounding box corresponding to the maximum intersection-and-union ratio can be determined among the multiple first intersection-and-union ratios, and the marked bounding box is determined as the bounding box that matches the predicted 2D bounding box. If it is determined not to be included, the predicted 2D bounding box is discarded.

[0176] (2) Calculate the intersection-and-union ratio between at least one predicted bounding box corresponding to the sample point cloud data in the sample target detection result and the labeled bounding box corresponding to the sample point cloud data in the sample label to obtain a second intersection-and-union ratio; and filter the at least one predicted bounding box according to the second intersection-and-union ratio to obtain a target predicted bounding box.

[0177] Specifically, in an embodiment of the present disclosure, the sample point cloud data contained in the training sample can be input into a point cloud unimodal detection model to obtain a sample target detection result including at least one predicted bounding box, wherein the at least one predicted bounding box can also be called a predicted 3D bounding box.

[0178] After that, the IOU (Intersection Over Union) between each predicted 3D bounding box and each labeled bounding box in the sample point cloud data is calculated to obtain multiple second IOU values; then, at least one predicted bounding box is screened according to the multiple second IOU values ​​to obtain the target predicted bounding box. The specific screening process is described as follows:

[0179] First, for each predicted 3D bounding box, determine whether the multiple second intersection-and-union ratios satisfy an intersection-and-union ratio greater than or equal to a preset threshold. If it is determined to be included, the predicted 3D bounding box is determined as the target predicted bounding box. At this time, the marked bounding box corresponding to the maximum intersection-and-union ratio can be determined among the multiple second intersection-and-union ratios, and the marked bounding box is determined as the bounding box that matches the predicted 3D bounding box. If it is determined not to be included, the predicted 3D bounding box is discarded.

[0180] (3) Matching the target prediction bounding box with the target prediction bounding box to obtain a label matching result, and determining the label matching matrix according to the label matching result.

[0181] After determining the target prediction bounding box and the target prediction bounding box, the target prediction bounding box and the target prediction bounding box corresponding to the same object in the target prediction bounding box are taken as a label matching pair; and the corresponding position of the label matching matrix is ​​set to 1, and the unmatched position is set to 0, thereby obtaining a label matching matrix.

[0182] In the above implementation, through the above processing method, the predicted bounding box and the predicted bounding box can be accurately matched to obtain a label matching matrix; when the function value of the target loss function is determined according to the label matching matrix, an accurate function value can be obtained, thereby improving the training accuracy of the unimodal detection model.

[0183] In the embodiment of the present disclosure, a training sample set including multiple training samples is determined, and the specific process is described as follows:

[0184] (1) Acquire a target tracking data sequence, wherein the target tracking data sequence includes image data and point cloud data acquired at each tracking moment.

[0185] (2) Determine at least one data combination in the target tracking data sequence, wherein each of the data combinations includes: target image data and target point cloud data; a first tracking moment of the target image data and a second tracking moment of the target point cloud data are different, and a time interval between the first tracking moment and the second tracking moment is a preset interval.

[0186] (3) Using the data in each of the data combinations as the data in each training sample.

[0187] In an embodiment of the present disclosure, a target tracking data sequence including image data and point cloud data is first obtained respectively, wherein the target tracking data sequence includes data sufficient for tracking and training the single-modal detection model.

[0188] In the embodiment of the present disclosure, first, in the same target tracking data sequence, the target image data image at the first tracking moment is selected. k Then, according to the preset interval, the second tracking moment is determined at intervals of several frames and the target point cloud data PC at that moment is selected k+n , its construction principle is weak synchronization transmission, the target image data image of the current frame k Image data with intervals of several frames k+n There will be weak synchronization of time and space, then the image data image k+n Corresponding strong synchronization target point cloud data PC k+n It will also be combined with the target image data image k Produce weak synchronization in space and time.

[0189] In the embodiment of the present disclosure, the target image data image k and target point cloud data PC k+n They are respectively determined as sample image data and sample point cloud data in the training samples.

[0190] In the above-mentioned implementation mode, the technical solution disclosed in the present invention proposes a method for constructing a multimodal data set of weak synchronization between point clouds and images. This method can simulate the weak synchronization situations that may occur in actual scenarios of autonomous driving. When a single-modal detection model is trained by using the training sample set, the trained single-modal detection model can adapt to the weak synchronization scenarios of the multimodal data set.

[0191] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0192] Based on the same inventive concept, a data matching device corresponding to the data matching method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned data matching method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0193] Reference Figure 4 FIG. 4 is a schematic diagram of a data matching device provided by an embodiment of the present disclosure, wherein the device comprises: an acquisition module 41, a determination module 42, and a matching module 43; wherein:

[0194] An acquisition module 41 is used to detect the point cloud data and the image data to be matched respectively, and obtain the target detection result of the point cloud data and the target detection result of the image data; wherein the target detection result includes the bounding box information of the detected target object;

[0195] Determination module 42: used to determine target feature information corresponding to the point cloud data according to the target detection result of the point cloud data; determine target feature information corresponding to the image data according to the target detection result of the image data; the target feature information includes geometric feature information of the bounding box of the detected target object and appearance feature information of the target object in the bounding box;

[0196] Matching module 43: used for matching the bounding box determined based on the point cloud data and the bounding box determined based on the image data according to the target feature information.

[0197] In the disclosed embodiment, by combining image data and point cloud data for target detection, the point cloud data can make up for the defect that image data is easily affected by illumination and occlusion, and the image data can make up for the defect that point cloud data is sparse and textureless. Therefore, combining image data and point cloud data to detect 3D targets can improve the detection accuracy of 3D targets, thereby obtaining more accurate target detection results.

[0198] In a possible implementation, the determination module 42 is also used to: project the target detection result of the point cloud data into the image data to obtain a projection frame of the bounding box of the point cloud data; and determine the geometric feature information of the point cloud data according to the pixel coordinates of the vertices of the projection frame in the image data.

[0199] In a possible implementation, the geometric feature information includes position information and / or size information of the bounding box.

[0200] In a possible implementation, the determination module 42 is further used to: determine the target image data in the image data that is within the detected bounding box; extract image features of the target image data, and determine the extracted image features as appearance feature information corresponding to the image data.

[0201] In a possible implementation, the determination module 42 is further used to: determine the target point cloud data located within the detected bounding box in the point cloud data; wherein the target point cloud data includes: the number of target point clouds located within the bounding box and / or the coordinate information of the target point cloud in the point cloud coordinate system; based on the target point cloud data, determine the global point cloud features used to describe the overall features of the target point cloud, and determine the appearance feature information corresponding to the point cloud data based on the global point cloud features.

[0202] In a possible implementation, the matching module 43 is further used to: perform correlation calculation on the target feature information of the image data and the target feature information of the point cloud data to obtain a correlation calculation result; and determine the matching result between the bounding box in the point cloud data and the bounding box in the image data based on the correlation calculation result.

[0203] In a possible implementation, the matching module 43 is further used to: splice the geometric feature information corresponding to the image data and the appearance feature information corresponding to the image data to obtain target image features; splice the geometric feature information corresponding to the point cloud data and the appearance feature information corresponding to the point cloud data to obtain target point cloud features; perform correlation calculations on the target image features and the target point cloud features to obtain the correlation calculation results.

[0204] In one possible implementation, the matching module 43 is further used to: perform convolution calculation on the correlation calculation result to obtain a similarity matrix, wherein the similarity matrix is ​​used to characterize the degree of similarity between the bounding box in the point cloud data and the bounding box in the image data; perform inverse calculation on the similarity matrix to obtain a matching cost matrix; and perform bipartite graph matching processing on the matching cost matrix to obtain a matching result between the bounding box in the point cloud data and the bounding box in the image data.

[0205] In a possible implementation, the matching module 43 is further used to perform target detection on the point cloud data and the image data respectively through a trained unimodal detection model to obtain a target detection result of the point cloud data and a target detection result of the image data.

[0206] In a possible implementation, the device is also used to: train the unimodal detection model according to the following steps: determine a training sample set containing multiple training samples; wherein each training sample contains: sample image data and sample point cloud data carrying a sample label; perform target detection on the training sample set through the unimodal detection model to be trained to obtain sample target detection results; determine a label matching matrix based on the sample target detection results and the sample labels; calculate a target loss function based on the label matching matrix, and adjust the model parameters of the unimodal detection model based on the target loss function until preset conditions are met to obtain the trained unimodal detection model.

[0207] In a possible implementation, the device is further used to: calculate the intersection-and-union ratio between at least one predicted bounding box corresponding to the sample image data in the sample target detection result and the labeled bounding box corresponding to the sample image data in the sample label to obtain a first intersection-and-union ratio; and filter the at least one predicted bounding box according to the first intersection-and-union ratio to obtain a target predicted bounding box; calculate the intersection-and-union ratio between at least one predicted bounding box corresponding to the sample point cloud data in the sample target detection result and the labeled bounding box corresponding to the sample point cloud data in the sample label to obtain a second intersection-and-union ratio; and filter the at least one predicted bounding box according to the second intersection-and-union ratio to obtain a target predicted bounding box; match the target predicted bounding box with the target predicted bounding box to obtain a label matching result, and determine the label matching matrix according to the label matching result.

[0208] In a possible implementation, the device is further used to: obtain a target tracking data sequence, wherein the target tracking data sequence includes image data and point cloud data acquired at each tracking moment; determine at least one data combination in the target tracking data sequence, wherein each of the data combinations includes: target image data and target point cloud data; a first tracking moment of the target image data and a second tracking moment of the target point cloud data are different, and the time interval between the first tracking moment and the second tracking moment is a preset interval; and use the data in each of the data combinations as the data in each training sample.

[0209] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0210] Corresponds to Figure 1 The data matching method in the present disclosure also provides an electronic device 500, such as Figure 5 FIG. 5 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present disclosure, including:

[0211] Processor 51, memory 52, and bus 53; memory 52 is used to store execution instructions, including memory 521 and external memory 522; memory 521 here is also called internal memory, which is used to temporarily store operation data in processor 51 and data exchanged with external memory 522 such as hard disk. Processor 51 exchanges data with external memory 522 through memory 521. When the electronic device 500 is running, the processor 51 communicates with the memory 52 through bus 53, so that the processor 51 executes the following instructions:

[0212] The point cloud data and image data to be matched are detected respectively to obtain target detection results of the point cloud data and the image data; wherein the target detection results include bounding box information of the detected target object; according to the target detection results of the point cloud data, target feature information corresponding to the point cloud data is determined; according to the target detection results of the image data, target feature information corresponding to the image data is determined; the target feature information includes geometric feature information of the bounding box of the detected target object and appearance feature information of the target object in the bounding box; according to the target feature information, the bounding box determined based on the point cloud data and the bounding box determined based on the image data are matched.

[0213] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0214] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0215] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0216] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0217] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A data matching method, characterized in that: include: Detecting the point cloud data and the image data to be matched respectively, and obtaining the target detection result of the point cloud data and the target detection result of the image data; wherein the target detection result includes the bounding box information of the detected target object; Determine target feature information corresponding to the point cloud data according to the target detection result of the point cloud data; determine target feature information corresponding to the image data according to the target detection result of the image data; the target feature information includes geometric feature information of a bounding box of the detected target object and appearance feature information of the target object within the bounding box; the appearance feature information is used to characterize attribute features of the target object selected by the bounding box; According to the target feature information, matching a bounding box determined based on the point cloud data with a bounding box determined based on the image data; The matching of the bounding box determined based on the point cloud data and the bounding box determined based on the image data according to the target feature information includes: Performing correlation calculation on the target feature information of the image data and the target feature information of the point cloud data to obtain a correlation calculation result; Performing convolution calculation on the correlation calculation result to obtain a similarity matrix, wherein the similarity matrix is ​​used to represent the similarity between the bounding box in the point cloud data and the bounding box in the image data; Performing an inverse calculation on the similarity matrix to obtain a matching cost matrix; A bipartite graph matching process is performed on the matching cost matrix to obtain a matching result between the bounding box in the point cloud data and the bounding box in the image data.

2. The method according to claim 1, characterized in that Determining target feature information corresponding to the point cloud data according to the target detection result of the point cloud data includes: Projecting the target detection result of the point cloud data into the image data to obtain a projection frame of the bounding box corresponding to the point cloud data; The geometric feature information of the point cloud data is determined according to the pixel coordinates of the vertices of the projection frame in the image data.

3. The method according to claim 1, characterized in that The geometric feature information includes position information and / or size information of the bounding box.

4. The method according to claim 1, characterized in that: The determining, according to the target detection result of the image data, target feature information corresponding to the image data comprises: Determining target image data in the image data that is within the detected bounding box; Image features of the target image data are extracted, and the extracted image features are determined as appearance feature information corresponding to the image data.

5. The method according to claim 1, characterized in that Determining target feature information corresponding to the point cloud data according to the target detection result of the point cloud data includes: Determine the target point cloud data located within the detected bounding box in the point cloud data; wherein the target point cloud data includes: the number of target point clouds located within the bounding box and / or the coordinate information of the target point clouds in the point cloud coordinate system; Based on the target point cloud data, a global point cloud feature for describing the overall feature of the target point cloud is determined, and appearance feature information corresponding to the point cloud data is determined according to the global point cloud feature.

6. The method according to claim 1, characterized in that The performing correlation calculation on the target feature information of the image data and the target feature information of the point cloud data to obtain a correlation calculation result includes: splicing the geometric feature information corresponding to the image data and the appearance feature information corresponding to the image data to obtain target image features; splicing the geometric feature information corresponding to the point cloud data and the appearance feature information corresponding to the point cloud data to obtain target point cloud features; A correlation operation is performed on the target image feature and the target point cloud feature to obtain the correlation calculation result.

7. The method according to any one of claims 1 to 6, characterized in that The detecting the point cloud data and the image data to be matched respectively to obtain the target detection result of the point cloud data and the target detection result of the image data includes: The trained unimodal detection model is used to perform target detection on the point cloud data and the image data respectively, so as to obtain target detection results of the point cloud data and target detection results of the image data.

8. The method according to claim 7, characterized in that The unimodal detection model is trained according to the following steps: Determine a training sample set including a plurality of training samples; wherein each training sample includes: sample image data or sample point cloud data carrying a sample label; Performing target detection on the training sample set using the unimodal detection model to be trained to obtain a sample target detection result; Determining a label matching matrix according to the sample target detection result and the sample label; The function value of the target loss function is calculated according to the label matching matrix, and the model parameters of the unimodal detection model are adjusted according to the function value of the target loss function until a preset condition is met, thereby obtaining the trained unimodal detection model.

9. The method according to claim 8, characterized in that Determining a label matching matrix according to the sample target detection result and the sample label includes: Calculating an intersection-and-union ratio between at least one predicted bounding box corresponding to the sample image data in the sample target detection result and a labeled bounding box corresponding to the sample image data in the sample label to obtain a first intersection-and-union ratio; and screening the at least one predicted bounding box according to the first intersection-and-union ratio to obtain a target predicted bounding box; Calculate the intersection-and-union ratio between at least one predicted bounding box corresponding to the sample point cloud data in the sample target detection result and the labeled bounding box corresponding to the sample point cloud data in the sample label to obtain a second intersection-and-union ratio; and filter the at least one predicted bounding box according to the second intersection-and-union ratio to obtain a target predicted bounding box; The target prediction bounding box is matched with the target prediction bounding box to obtain a label matching result, and the label matching matrix is ​​determined according to the label matching result.

10. The method according to claim 8 or 9, characterized in that: The step of determining a training sample set including a plurality of training samples comprises: Acquire a target tracking data sequence, wherein the target tracking data sequence includes image data and point cloud data acquired at each tracking moment; Determine at least one data combination in the target tracking data sequence, wherein each of the data combinations includes: target image data and target point cloud data; a first tracking moment of the target image data and a second tracking moment of the target point cloud data are different, and a time interval between the first tracking moment and the second tracking moment is a preset interval; The data in each of the data combinations is used as the data in each training sample.

11. A data matching device, characterized in that: include: Acquisition module: used to detect the point cloud data and image data to be matched respectively, and obtain the target detection result of the point cloud data and the target detection result of the image data; wherein the target detection result includes the bounding box information of the detected target object; Determination module: used to determine target feature information corresponding to the point cloud data according to the target detection result of the point cloud data; determine target feature information corresponding to the image data according to the target detection result of the image data; the target feature information includes geometric feature information of the bounding box of the detected target object and appearance feature information of the target object in the bounding box; the appearance feature information is used to characterize the attribute features of the target object selected by the bounding box; A matching module: used for matching a bounding box determined based on the point cloud data with a bounding box determined based on the image data according to the target feature information; The matching module, when matching the bounding box determined based on the point cloud data and the bounding box determined based on the image data according to the target feature information, is used to: Performing correlation calculation on the target feature information of the image data and the target feature information of the point cloud data to obtain a correlation calculation result; Performing convolution calculation on the correlation calculation result to obtain a similarity matrix, wherein the similarity matrix is ​​used to represent the similarity between the bounding box in the point cloud data and the bounding box in the image data; Performing an inverse calculation on the similarity matrix to obtain a matching cost matrix; A bipartite graph matching process is performed on the matching cost matrix to obtain a matching result between the bounding box in the point cloud data and the bounding box in the image data.

12. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the data matching method as described in any one of claims 1 to 10 are performed.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data matching method according to any one of claims 1 to 10 are executed.

Images