A data processing method and device, server, and storage medium

Abstract

The embodiment of the present invention provides a method and device, a server, and a storage medium for data processing, the method comprising: obtaining a camera feature vector for a camera target and a radar feature vector for a radar target; generating camera heat map data corresponding to the camera feature vector and radar heat map data corresponding to the radar feature vector; using a preset data model to perform model processing on the camera heat map data and the radar heat map data to obtain nested data; generating a camera nested vector and a radar nested vector based on the nested data; using the camera nested vector and the radar nested vector to train the data model, or generating a connection relationship for the camera target and the radar target. Through the embodiment of the present invention, the fusion algorithm is driven by data to learn the fusion mode and rules, avoiding dependence on artificial empirical rules, and obtaining a fusion result with strong robustness and high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data processing method and device, a server, and a storage medium. Background Art

[0002] At present, the technology for the fusion of two sensors, camera and millimeter-wave radar, is based on traditional methods, mainly with manually defined logic. It uses logic programming to form an algorithm for fusion by using the fusion rules in multiple scenarios summarized by human experience.

[0003] However, considering the complexity and infinity of autonomous driving scenarios, this method of manually defining logic has great limitations. As the amount of data increases, the performance of the algorithm cannot be continuously improved. Summary of the invention

[0004] In view of the above problems, a data processing method and device, server, and storage medium are proposed to overcome the above problems or at least partially solve the above problems, including:

[0005] A method for data processing, the method comprising:

[0006] Obtaining a camera feature vector for a camera target and a radar feature vector for a radar target;

[0007] Generate camera heat map data corresponding to the camera feature vector and radar heat map data corresponding to the radar feature vector;

[0008] Using a preset data model, model processing is performed on the camera thermal map data and the radar thermal map data to obtain nested data;

[0009] Generate a camera nesting vector and a radar nesting vector according to the nesting data;

[0010] The camera nesting vector and the radar nesting vector are used to train the data model, or to generate a connection relationship between a camera target and a radar target.

[0011] Optionally, the generating the camera heat map data corresponding to the camera feature vector and the radar heat map data corresponding to the radar feature vector includes:

[0012] Determine a camera target position of the camera target in a camera plane, and generate camera heat map data corresponding to the camera feature vector according to the camera target position;

[0013] A radar target position where the radar target is projected onto a camera plane is determined, and radar heat map data corresponding to the radar feature vector is generated according to the radar target position.

[0014] Optionally, generating a camera nesting vector and a radar nesting vector according to the nesting data includes:

[0015] determining a camera nesting vector from the nesting data according to the camera target position;

[0016] A radar nesting vector is determined from the nesting data according to the radar target position.

[0017] Optionally, the acquiring a camera feature vector for a camera target and a radar feature vector for a radar target includes:

[0018] Acquiring camera data and determining radar data that matches the camera data;

[0019] Performing target detection on the camera data to obtain a camera target, and performing target detection on the radar data to obtain a radar target;

[0020] The camera target is characterized and expressed to obtain a camera feature vector for the camera target, and the radar target is characterized and expressed to obtain a radar feature vector for the radar target.

[0021] Optionally, the using of a preset data model to perform model processing on the radar heat map data and the camera heat map data to obtain nested data includes:

[0022] Splicing the radar thermal map data and the camera thermal map data to obtain spliced ​​data;

[0023] The preset data model is used to perform model processing on the spliced ​​data to obtain nested data.

[0024] Optionally, before generating the camera heat map data corresponding to the camera feature vector and the radar heat map data corresponding to the radar feature vector, the method further includes:

[0025] The camera feature vector and the radar feature vector are aligned.

[0026] Optionally, the adopting the camera nested vector and the radar nested vector to train the data model includes:

[0027] A loss function is constructed using the camera nested vector and the radar nested vector; wherein the camera target corresponding to the camera nested vector and the radar target corresponding to the radar nested vector are the same target;

[0028] The data model is trained according to the loss function.

[0029] Optionally, the using the camera nesting vector and the radar nesting vector to generate a connection relationship between a camera target and a radar target includes:

[0030] Using the camera nesting vector and the radar nesting vector, construct a similarity matrix for camera targets and radar targets;

[0031] A connection relationship between the camera target and the radar target is generated according to the similarity matrix.

[0032] A data processing device, comprising:

[0033] A feature vector acquisition module, used to acquire a camera feature vector for a camera target and a radar feature vector for a radar target;

[0034] A heat map data generation module, used to generate camera heat map data corresponding to the camera feature vector and radar heat map data corresponding to the radar feature vector;

[0035] A nested data obtaining module, used to use a preset data model to perform model processing on the camera thermal map data and the radar thermal map data to obtain nested data;

[0036] A nested vector generation module, used to generate a camera nested vector and a radar nested vector according to the nested data;

[0037] A connection relationship generating module is used to train the data model by using the camera nesting vector and the radar nesting vector, or to generate a connection relationship between a camera target and a radar target.

[0038] A server comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the data processing method as described above when executed by the processor.

[0039] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method described above is implemented.

[0040] The embodiments of the present invention have the following advantages:

[0041] In an embodiment of the present invention, by acquiring a camera feature vector for a camera target and a radar feature vector for a radar target, camera heat map data corresponding to the camera feature vector and radar heat map data corresponding to the radar feature vector are generated, and then a preset data model is used to perform model processing on the camera heat map data and the radar heat map data to obtain nested data, and a camera nested vector and a radar nested vector are generated according to the nested data, and then the camera nested vector and the radar nested vector are used to train the data model, or a connection relationship between the camera target and the radar target is generated, so that the fusion algorithm is driven by data to learn the fusion method and rules, so that the performance of the algorithm can be improved with the skills of the training data, avoiding dependence on artificial empirical rules, and being able to obtain a fusion result with strong robustness and high accuracy. At the same time, this deep learning-based association method can be used to automatically jointly annotate a large number of image radar data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0043] Figure 1 is a flowchart of a data processing method provided by an embodiment of the present invention;

[0044] Figure 2 is a flowchart of another data processing method provided by an embodiment of the present invention;

[0045] Figure 3 is a flowchart of another data processing method provided by an embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of an example of a target matching process provided by an embodiment of the present invention;

[0047] Figure 5 It is a structural diagram of a data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] Reference Figure 1 , shows a flowchart of a data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0050] Step 101, obtaining a camera feature vector for a camera target and a radar feature vector for a radar target;

[0051] The camera feature vector may be a vector representing the features of a camera target; the radar feature vector may be a vector representing the features of a radar target. For example, a set of vectors may be used to describe the features of each target.

[0052] In the process of fusing camera signals and radar signals, the camera target and the radar target can be obtained, and a set of vectors can be used to describe the characteristics of each camera target or radar target, so as to obtain a camera feature vector for the camera target and a radar feature vector for the radar target.

[0053] For example, target detection may be performed on camera data and radar data respectively, and the detected targets may be characterized, that is, a set of vectors is used to describe the characteristics of each detected target.

[0054] In autonomous driving and advanced driver assistance systems, cameras and millimeter-wave radars can be configured to achieve perception of the surrounding environment and target detection. Since cameras and millimeter-wave radars are two different sensors, the camera signals and radar signals obtained are of different properties and have different focuses. The two signals can be fused to achieve a complementary effect to enhance the perception capability of the autonomous driving system. The detection result of the camera target is two-dimensional information on the camera plane. After being fused with the detection result of the millimeter-wave radar, the two-dimensional information can be expanded to three-dimensional information, which can increase its role in the autonomous driving technology stack.

[0055] In one example, the fusion technology for cameras and radars can be implemented using a fusion algorithm based on deep learning. The algorithm can use neural networks to perform autonomous learning to learn the relationship between radar targets and camera targets. The fusion results are obtained by training and learning on a large amount of real driving data. This can achieve sensor fusion with strong robustness and high accuracy, and the performance of the algorithm can be continuously improved with the skills of the training data.

[0056] In another example, the radar target detection module can output a target, and the camera target detection module can also output a target. Each target can contain the sensor's perception and understanding of the surrounding environment. If the two targets are fused, the autonomous driving system's overall perception and understanding of the surrounding environment can be enhanced. For radar targets and camera targets, each target can find the object that generates the target in the time environment, and then the corresponding radar target and camera target generated by the same object in the actual scene can be matched to achieve fusion of the two targets.

[0057] Step 102, generating camera heat map data corresponding to the camera feature vector and radar heat map data corresponding to the radar feature vector;

[0058] After obtaining the camera feature vector and the radar feature vector, the camera feature vector may be used to generate corresponding camera heat map data, and the radar feature vector may be used to generate corresponding radar heat map data, so as to further process the camera heat map data and the radar heat map data.

[0059] For example, the camera feature vector can be converted into different latitudes outside the camera plane according to the camera plane and the detected camera target position, and can be further processed by Gaussian Smooth to construct a camera heat map; the radar feature vector can be converted into different channels of the camera plane image according to the camera plane and the projection of the radar target on the camera plane, and can be further processed by Gaussian Smooth to construct a radar heat map, which can give full play to the role of deep learning (such as convolutional neural networks).

[0060] Step 103, using a preset data model to perform model processing on the camera thermal map data and the radar thermal map data to obtain nested data;

[0061] As an example, a preset data model may be used to perform representation learning on camera heat map data and radar heat map data, for example, by performing representation learning through a neural network model.

[0062] After obtaining the camera thermal map data and the radar thermal map data, a neural network model can be used for representation learning, and then nested data processed by the neural network model can be obtained, such as nested image-radar joint data.

[0063] For example, the spliced ​​camera heat map and radar heat map can be subjected to representation learning using a neural network based on deep learning, and after being processed by the neural network, an embedding map based on the camera plane can be obtained.

[0064] In one example, the nested image may have high-level, global semantic information after being processed by a neural network. The nested image may be directly used for classification judgment, which is different from an ordinary heat map composed of radar targets and camera targets, which is low-level, localized information.

[0065] In another example, representation learning can be performed in a variety of ways. Representation learning can be performed based on a deep learning neural network. The neural network can be based on an encoder-decoder architecture and can be combined with a feature pyramid network to enhance the representation learning capability.

[0066] Step 104, generating a camera nesting vector and a radar nesting vector according to the nesting data;

[0067] After the nested image-radar joint data is obtained, the camera embedding vector and the radar embedding vector for the detected target can be extracted from the nested image-radar joint data. For example, in the camera plane, the embedding vector of each detected target can be extracted in the nested image according to the projection position of the radar target on the camera plane and the position of the camera target.

[0068] Step 105: Use the camera nesting vector and the radar nesting vector to train the data model, or generate a connection relationship between the camera target and the radar target.

[0069] After obtaining the camera nested vector and the radar nested vector, the camera nested vector and the radar nested vector can be used to construct a loss function in the neural network training stage, and then the data model can be trained. In the neural network inference stage, the camera nested vector and the radar nested vector can be used to infer the connection relationship between the radar target and the camera target. For example, the correlation relationship between the radar target and the camera target can be obtained.

[0070] In one example, a nested vector can be further obtained by nesting images. The nested vector can be high-level, global information, and then a matching judgment can be made on the connection relationship between the camera target and the radar target by calculating the distance.

[0071] Compared with traditional fusion technology, the fusion algorithm of the present invention can be driven by data and continuously learn fusion methods and rules from data, thereby getting rid of the dependence on artificial empirical rules, and then obtaining a fusion result with strong robustness and high accuracy. Moreover, since the fusion algorithm can be driven by data, the performance of the fusion algorithm can be continuously improved with the accumulation of data, thereby continuously improving the performance of autonomous driving technology.

[0072] In an embodiment of the present invention, by acquiring a camera feature vector for a camera target and a radar feature vector for a radar target, camera heat map data corresponding to the camera feature vector and radar heat map data corresponding to the radar feature vector are generated, and then a preset data model is used to perform model processing on the camera heat map data and the radar heat map data to obtain nested data, and a camera nested vector and a radar nested vector are generated according to the nested data, and then the camera nested vector and the radar nested vector are used to train the data model, or a connection relationship between the camera target and the radar target is generated, so that the fusion algorithm is driven by data to learn the fusion method and rules, so that the performance of the algorithm can be improved with the skills of the training data, avoiding dependence on artificial empirical rules, and being able to obtain a fusion result with strong robustness and high accuracy. At the same time, this deep learning-based association method can be used to automatically jointly annotate a large number of image radar data sets.

[0073] Reference Figure 2 , shows a flowchart of another data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0074] Step 201, obtaining a camera feature vector for a camera target and a radar feature vector for a radar target;

[0075] In the process of fusing camera signals and radar signals, the camera target and the radar target can be obtained, and a set of vectors can be used to describe the characteristics of each camera target or radar target, so as to obtain a camera feature vector for the camera target and a radar feature vector for the radar target.

[0076] Step 202, determining a camera target position of the camera target in a camera plane, and generating camera heat map data corresponding to the camera feature vector according to the camera target position;

[0077] After obtaining the camera feature vector, the camera target position on the camera plane can be determined for the camera target, and then the camera target position and the camera feature vector can be used to generate camera heat map data.

[0078] For example, the camera feature vector can be converted into different channels of the camera plane image according to the camera plane and the detected camera target position, and Gaussian smoothing can be further applied to construct a camera heat map.

[0079] Step 203, determining a radar target position where the radar target is projected onto a camera plane, and generating radar heat map data corresponding to the radar feature vector according to the radar target position;

[0080] After obtaining the radar feature vector, the radar target position projected in the camera plane can be determined for the radar target, and then the radar target position and the radar feature vector can be used to generate radar heat map data.

[0081] For example, the radar feature vector can be converted into different channels of the camera plane image according to the camera plane and the projection of the radar target on the camera plane, and Gaussian smoothing can be further applied to construct a radar heat map.

[0082] Step 204, splicing the radar thermal map data and the camera thermal map data to obtain spliced ​​data;

[0083] After obtaining the radar thermal map data and the camera thermal map data, the radar thermal map data and the camera thermal map data can be spliced ​​to obtain spliced ​​data, for example, spliced ​​image-radar joint data, which can be obtained by splicing the constructed camera thermal map and radar thermal map together (image concatenation) after the camera thermal map and the radar thermal map are constructed.

[0084] In one example, a camera heat map and a radar heat map are stitched together to obtain a feature map, which is different from an image in the conventional sense.

[0085] Step 205, using a preset data model to perform model processing on the spliced ​​data to obtain nested data;

[0086] After obtaining the camera thermal map data and the radar thermal map data, a neural network model can be used for representation learning, and then nested data processed by the neural network model can be obtained, such as nested image-radar joint data.

[0087] In one example, point cloud expression can be used to replace heat map expression, and data splicing and training can also be performed on the data expressed in point cloud form to achieve the same expression effect as heat map expression.

[0088] Step 206, determining a camera nesting vector from the nesting data according to the camera target position;

[0089] After the nested image-radar joint data is obtained, the camera nesting vector for the detected target can be extracted from the nested image-radar joint data according to the camera target position. For example, the nesting vector of each detected target can be extracted from the nested image according to the position of the camera target in the camera plane.

[0090] Step 207, determining a radar nesting vector from the nesting data according to the radar target position;

[0091] After the nested image-radar joint data is obtained, the camera nesting vector for the detected target can be extracted from the nested image-radar joint data according to the radar target position. For example, in the camera plane, the nesting vector of each detected target can be extracted from the nested image according to the projection position of the radar target on the camera plane.

[0092] In one example, each radar target and each camera target may have a respective pixel position, and then a nested vector for each target may be obtained in the channel dimension of the nested image according to the pixel position of the target.

[0093] Step 208: Use the camera nesting vector and the radar nesting vector to train the data model, or generate a connection relationship between the camera target and the radar target.

[0094] After obtaining the camera nested vector and the radar nested vector, the camera nested vector and the radar nested vector can be used to construct a loss function in the neural network training stage, and then the data model can be trained. In the neural network inference stage, the camera nested vector and the radar nested vector can be used to infer the connection relationship between the radar target and the camera target.

[0095] In an embodiment of the present invention, a camera feature vector for a camera target and a radar feature vector for a radar target are obtained to determine a camera target position of the camera target in a camera plane, and camera heat map data corresponding to the camera feature vector is generated according to the camera target position, a radar target position where the radar target is projected onto the camera plane is determined, and radar heat map data corresponding to the radar feature vector is generated according to the radar target position, and the radar heat map data and the camera heat map data are spliced ​​to obtain spliced ​​data, and then a preset data model is used to perform model processing on the spliced ​​data to obtain nested data, and the corresponding camera target position is determined from the nested data according to the camera target position. The camera nested vector is determined from the nested data according to the radar target position, and the camera nested vector and the radar nested vector are used to train the data model, or the connection relationship between the camera target and the radar target is generated. The fusion algorithm is driven by data to learn the fusion method and rules. By expressing and splicing the radar data and the camera data with heat maps, and then using neural networks for representation learning, the performance of the algorithm can be improved with the skills of the training data, and a fusion result with strong robustness and high accuracy can be obtained. At the same time, this association method based on deep learning can be used to automatically jointly annotate a large number of image radar data sets.

[0096] Reference Figure 3 , shows a flowchart of a data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0097] Step 301, acquiring camera data, and determining radar data matching the camera data;

[0098] In the process of fusing camera signals and radar signals, the camera data can be obtained, and then the radar data can be matched with the camera data.

[0099] For example, a camera sensor may be used to collect signals, and camera photos may be processed to obtain camera data. A radar sensor may be used to collect signals, and radar signals may be subjected to DSP (Digital Signal Processing) processing. Then, for any frame of camera data, the camera data may be used to find the radar data that best matches in time.

[0100] Step 302, performing target detection on the camera data to obtain a camera target, and performing target detection on the radar data to obtain a radar target;

[0101] After acquiring the camera data and the radar data, target detection may be performed on the camera data to obtain the camera target, and target detection may be performed on the radar data to obtain the radar target. For example, 2D target detection may be performed on the camera data, and target detection may be performed on the radar data to obtain the camera target and the radar target.

[0102] Step 303, characterizing the camera target to obtain a camera feature vector for the camera target, and characterizing the radar target to obtain a radar feature vector for the radar target;

[0103] After obtaining the camera target and the radar target, the camera target can be characterized and expressed, and then a camera feature vector for the camera target can be obtained. The radar target can also be characterized and expressed, and then a radar feature vector for the radar target can be obtained.

[0104] For example, detected camera targets and radar targets may be characterized, that is, a set of vectors may be used to describe the characteristics of each detected target.

[0105] Step 304, aligning the camera feature vector and the radar feature vector;

[0106] After obtaining the camera feature vector and the radar feature vector, the camera feature vector and the radar feature vector may be time aligned and space processed, so as to further use the aligned and calibrated data for subsequent processing.

[0107] For example, the radar data can be aligned with the camera data by projecting the speed of each radar point in time. For example, the radar and the camera can be time-aligned by motion compensation. The position of the radar point at different time points can be estimated and predicted by the radar speed, thereby eliminating the error in the acquisition time of the radar data and the camera data.

[0108] For another example, the radar target can be projected onto the camera plane through the calibration relationship between the radar sensor and the camera sensor. For example, the radar can be spatially aligned with the camera coordinate system through coordinate system transformation, thereby spatially aligning the data.

[0109] Step 305, generating camera heat map data corresponding to the camera feature vector and radar heat map data corresponding to the radar feature vector;

[0110] After obtaining the camera feature vector and the radar feature vector, the camera feature vector may be used to generate corresponding camera heat map data, and the radar feature vector may be used to generate corresponding radar heat map data, so as to further process the camera heat map data and the radar heat map data.

[0111] Step 306, using a preset data model to perform model processing on the camera thermal map data and the radar thermal map data to obtain nested data;

[0112] After obtaining the camera thermal map data and the radar thermal map data, a neural network model can be used for representation learning, and then nested data processed by the neural network model can be obtained, such as nested image-radar joint data.

[0113] Step 307, generating a camera nesting vector and a radar nesting vector according to the nesting data;

[0114] After the nested image-radar joint data is obtained, the camera nesting vector and the radar nesting vector for the detected target can be extracted from the nested image-radar joint data. For example, in the camera plane, the nesting vector of each detected target can be extracted in the nested image according to the projection position of the radar target on the camera plane and the position of the camera target.

[0115] In one example, through nested vector expression, a set of optimal vectors can be found through machine learning to describe each target (radar target and camera target), which can characterize the global, high-level environmental perception and understanding capabilities, and thus can better distinguish whether the radar target and the camera target come from the same object.

[0116] Step 308: Use the camera nesting vector and the radar nesting vector to train the data model, or generate a connection relationship between the camera target and the radar target.

[0117] After obtaining the camera nested vector and the radar nested vector, the camera nested vector and the radar nested vector can be used to construct a loss function in the neural network training stage, and then the data model can be trained. In the neural network inference stage, the camera nested vector and the radar nested vector can be used to infer the connection relationship between the radar target and the camera target. For example, the correlation relationship between the radar target and the camera target can be obtained.

[0118] In one embodiment of the present invention, step 308 may include the following sub-steps:

[0119] A loss function is constructed using the camera nested vector and the radar nested vector; wherein a camera target corresponding to the camera nested vector and a radar target corresponding to the radar nested vector are the same target; and the data model is trained according to the loss function.

[0120] In a specific implementation, after obtaining the camera nesting vector and the radar nesting vector, the camera target corresponding to the camera nesting vector and the radar target corresponding to the radar nesting vector can be the same target. In the neural network training stage, the camera nesting vector and the radar nesting vector are used to construct a loss function, and then the data model can be trained according to the loss function.

[0121] For example, in the neural network training stage, a loss function can be constructed for the corresponding nested vectors of each target according to the matching relationship between the radar target and the camera target. The loss function can be used to achieve the goal of making the nested vectors that should be matched closer and the nested vectors that should not be matched farther. There are many ways to design the loss function, and a push and pull loss function can be adopted, which can pull the nested vectors that can be matched closer and push the nested vectors that cannot be matched farther.

[0122] In one example, different loss functions may be used to construct the neural network training phase, such as a push / pull loss function, a triplet loss function, and the like.

[0123] In another example, since there is a matching relationship between the camera target group and the radar target group, in the neural network training stage, by adopting a pre-labeled matching relationship, for example, any pair of matched camera targets and radar targets can be taken, their respective corresponding nested vectors can be found, and then the distance between the nested vectors can be calculated, so that the distance between the nested vectors that should be matched is close enough; similarly, any pair of camera targets and radar targets that should not be matched can be taken, their respective corresponding nested vectors can be found, and then the distance between the nested vectors can be calculated, so that the distance between the nested vectors that should not be matched is far enough.

[0124] Since traditional fusion technology cannot utilize the soft rules in the fusion into the fusion algorithm, such as the fused ordinal information, the present invention can integrate the soft rules into the algorithm based on the deep learning neural network, and can encode the soft rules into the neural network through a loss function to achieve it, for example, an ordinal loss function, thereby making the fusion algorithm of the present invention based on the deep learning neural network more powerful and flexible.

[0125] In one embodiment of the present invention, step 308 may include the following sub-steps:

[0126] The camera nesting vector and the radar nesting vector are used to construct a similarity matrix for the camera target and the radar target; and a connection relationship for the camera target and the radar target is generated according to the similarity matrix.

[0127] In a specific implementation, after obtaining the camera nesting vector and the radar nesting vector, the camera nesting vector and the radar nesting vector can be used in the neural network inference stage to construct a similarity matrix for the camera target and the radar target, and then the connection relationship between the radar target and the camera target can be obtained according to the similarity matrix. For example, the association relationship between the radar target and the camera target can be obtained.

[0128] For example, in the neural network reasoning stage, the affinity matrix of the radar target and the camera target can be constructed according to the nested vector of each target obtained by the neural network processing, and then matching can be performed according to the affinity matrix. There can be many designs for the matching algorithm. A greedy algorithm can be used to select the radar target and the camera target with the closest matching distance as a pair, that is, until the matching distance exceeds a preset distance threshold, so that the radar target and the camera target with a matching relationship can be screened out through the threshold, and at the same time, it can also avoid misjudging the radar target and the camera target that should not be matched as a matching relationship.

[0129] In one example, a variant algorithm based on a greedy algorithm, such as the Hungarian Algorithm, may be used to similarly achieve the purpose of obtaining the connection relationship between the radar target and the camera target.

[0130] In order to enable those skilled in the art to better understand the above steps, Figure 4 The embodiments of the present invention are described exemplarily, but it should be understood that the embodiments of the present invention are not limited thereto.

[0131] For radar modules:

[0132] 1. Collect signals through radar sensors;

[0133] 2. For radar signal processing (DSP) and target detection;

[0134] 3. Characterize the radar target;

[0135] For camera modules:

[0136] 7. Collect signals through camera sensors;

[0137] 8. For camera photo processing and 2D target detection;

[0138] 9. Characterize the camera target;

[0139] For the Deep Fusion module:

[0140] 4. Align the radar to the camera coordinate system (coordinate system transformation, etc.);

[0141] 5. Time alignment of radar to camera (motion compensation, etc.);

[0142] 6. Heat map representation of radar targets based on the 2D position of camera plane objects;

[0143] 10. Camera target is a heat map representation of the 2D position of the camera plane object;

[0144] 11. Perform thermal image stitching for radar and camera;

[0145] 12. Representation learning based on deep learning neural network;

[0146] 13. Obtain a nested image based on the camera plane;

[0147] 14. Extract radar nesting vectors based on radar target positions from the nested images;

[0148] 15. Extract the camera nesting vector based on the camera target position from the nested image;

[0149] 16. Express the nested vector of radar target;

[0150] 17. Express the camera target nested vector;

[0151] 18. Determine whether to enter the neural network training stage or the neural network inference stage;

[0152] 19. During the neural network training phase, a loss function can be constructed based on the radar camera target matching relationship;

[0153] 20. During the neural network inference phase, the similarity matrix between radar targets and camera targets can be constructed;

[0154] 21. During the neural network inference stage, the connection relationship between the radar target and the camera target can be constructed based on the greedy algorithm according to the similarity matrix.

[0155] In an embodiment of the present invention, by acquiring camera data and determining radar data matching the camera data, performing target detection on the camera data to obtain a camera target, performing target detection on the radar data to obtain a radar target, and characterizing the camera target to obtain a camera feature vector for the camera target, characterizing the radar target to obtain a radar feature vector for the radar target, and then aligning the camera feature vector and the radar feature vector to generate camera heat map data corresponding to the camera feature vector and radar heat map data corresponding to the radar feature vector, using a preset data model, performing model processing on the camera heat map data and the radar heat map data to obtain nested data, and then generating a camera nested vector and a radar nested vector according to the nested data, using the camera nested vector and the radar nested vector to train the data model, or generating a connection relationship between the camera target and the radar target, the fusion algorithm is driven by data to learn the fusion mode and rules, and the radar target detection result and the camera target detection result are characterized and then data are processed, so that a fusion result with strong robustness and high accuracy can be obtained. At the same time, this deep learning-based association method can be used to automatically jointly annotate a large number of image radar data sets.

[0156] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0157] Reference Figure 5 , shows a schematic diagram of the structure of a data processing device provided by an embodiment of the present invention, which may specifically include the following modules:

[0158] A feature vector acquisition module 501 is used to acquire a camera feature vector for a camera target and a radar feature vector for a radar target;

[0159] A heat map data generating module 502, configured to generate camera heat map data corresponding to the camera feature vector and radar heat map data corresponding to the radar feature vector;

[0160] The nested data obtaining module 503 is used to use a preset data model to perform model processing on the camera thermal map data and the radar thermal map data to obtain nested data;

[0161] A nesting vector generating module 504, configured to generate a camera nesting vector and a radar nesting vector according to the nesting data;

[0162] The connection relationship generating module 505 is used to train the data model by using the camera nesting vector and the radar nesting vector, or to generate a connection relationship between a camera target and a radar target.

[0163] In one embodiment of the present invention, the heat map data generation module 502 includes:

[0164] A camera heat map data generating submodule, used to determine a camera target position of the camera target in a camera plane, and generate camera heat map data corresponding to the camera feature vector according to the camera target position;

[0165] The radar heat map data generating submodule is used to determine the radar target position where the radar target is projected onto the camera plane, and generate radar heat map data corresponding to the radar feature vector according to the radar target position.

[0166] In one embodiment of the present invention, the nested vector generation module 504 includes:

[0167] A camera nesting vector generating submodule, used for determining a camera nesting vector from the nesting data according to the camera target position;

[0168] The radar nesting vector generating submodule is used to determine the radar nesting vector from the nesting data according to the radar target position.

[0169] In one embodiment of the present invention, the feature vector acquisition module 501 includes:

[0170] A camera data and radar data acquisition submodule, used to acquire camera data and determine radar data matching the camera data;

[0171] A target acquisition submodule, used to perform target detection on the camera data to obtain a camera target, and to perform target detection on the radar data to obtain a radar target;

[0172] The feature vector obtaining submodule is used to perform feature expression on the camera target to obtain a camera feature vector for the camera target, and to perform feature expression on the radar target to obtain a radar feature vector for the radar target.

[0173] In one embodiment of the present invention, the nested data obtaining module 503 includes:

[0174] A data splicing submodule is used to splice the radar thermal map data and the camera thermal map data to obtain spliced ​​data;

[0175] The nested data obtaining submodule is used to adopt a preset data model to perform model processing on the spliced ​​data to obtain nested data.

[0176] In one embodiment of the present invention, the device further comprises:

[0177] An alignment module is used to align the camera feature vector and the radar feature vector.

[0178] In one embodiment of the present invention, the connection relationship generating module 505 includes:

[0179] A loss function construction submodule, used to construct a loss function using the camera nesting vector and the radar nesting vector; wherein the camera target corresponding to the camera nesting vector and the radar target corresponding to the radar nesting vector are the same target;

[0180] A training submodule is used to train the data model according to the loss function.

[0181] In one embodiment of the present invention, the connection relationship generating module 505 includes:

[0182] A similarity matrix construction submodule, used to construct a similarity matrix for camera targets and radar targets using the camera nesting vector and the radar nesting vector;

[0183] The connection relationship generating submodule is used to generate a connection relationship for the camera target and the radar target according to the similarity matrix.

[0184] In an embodiment of the present invention, by acquiring a camera feature vector for a camera target and a radar feature vector for a radar target, camera heat map data corresponding to the camera feature vector and radar heat map data corresponding to the radar feature vector are generated, and then a preset data model is used to perform model processing on the camera heat map data and the radar heat map data to obtain nested data, and a camera nested vector and a radar nested vector are generated according to the nested data, and then the camera nested vector and the radar nested vector are used to train the data model, or a connection relationship between the camera target and the radar target is generated, so that the fusion algorithm is driven by data to learn the fusion method and rules, so that the performance of the algorithm can be improved with the skills of the training data, avoiding dependence on artificial empirical rules, and being able to obtain a fusion result with strong robustness and high accuracy. At the same time, this deep learning-based association method can be used to automatically jointly annotate a large number of image radar data sets.

[0185] An embodiment of the present invention further provides a server, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor, and the computer program implements the above data processing method when executed by the processor.

[0186] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above data processing method is implemented.

[0187] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0188] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0189] Those skilled in the art will appreciate that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0190] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0194] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0195] The above provides a detailed introduction to a data processing method and device, server, and storage medium. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for data processing, characterized in that: The method comprises: Obtaining a camera feature vector for a camera target and a radar feature vector for a radar target; Generate camera heat map data corresponding to the camera feature vector and radar heat map data corresponding to the radar feature vector; Using a preset data model, model processing is performed on the camera thermal map data and the radar thermal map data to obtain nested data; Generate a camera nesting vector and a radar nesting vector according to the nesting data; Using the camera nesting vector and the radar nesting vector to train the data model, or to generate a connection relationship between a camera target and a radar target; The generating the camera heat map data corresponding to the camera feature vector and the radar heat map data corresponding to the radar feature vector includes: Determine a camera target position of the camera target in a camera plane, and generate camera heat map data corresponding to the camera feature vector according to the camera target position; A radar target position where the radar target is projected onto a camera plane is determined, and radar heat map data corresponding to the radar feature vector is generated according to the radar target position.

2. The method according to claim 1, characterized in that The step of generating a camera nesting vector and a radar nesting vector according to the nesting data includes: determining a camera nesting vector from the nesting data according to the camera target position; A radar nesting vector is determined from the nesting data according to the radar target position.

3. The method according to claim 1 or 2, characterized in that: The obtaining of the camera feature vector for the camera target and the radar feature vector for the radar target includes: Acquiring camera data and determining radar data that matches the camera data; Performing target detection on the camera data to obtain a camera target, and performing target detection on the radar data to obtain a radar target; The camera target is characterized and expressed to obtain a camera feature vector for the camera target, and the radar target is characterized and expressed to obtain a radar feature vector for the radar target.

4. The method according to claim 1, characterized in that: The method adopts a preset data model to perform model processing on the radar heat map data and the camera heat map data to obtain nested data, including: Splicing the radar thermal map data and the camera thermal map data to obtain spliced ​​data; The preset data model is used to perform model processing on the spliced ​​data to obtain nested data.

5. The method according to claim 1, characterized in that Before generating the camera heat map data corresponding to the camera feature vector and the radar heat map data corresponding to the radar feature vector, the method further includes: The camera feature vector and the radar feature vector are aligned.

6. The method according to claim 1, characterized in that The adopting the camera nested vector and the radar nested vector to train the data model includes: A loss function is constructed using the camera nested vector and the radar nested vector; wherein the camera target corresponding to the camera nested vector and the radar target corresponding to the radar nested vector are the same target; The data model is trained according to the loss function.

7. The method according to claim 1, characterized in that The step of using the camera nesting vector and the radar nesting vector to generate a connection relationship between a camera target and a radar target includes: Using the camera nesting vector and the radar nesting vector, construct a similarity matrix for camera targets and radar targets; A connection relationship between the camera target and the radar target is generated according to the similarity matrix.

8. A data processing device, characterized in that: The device comprises: A feature vector acquisition module, used to acquire a camera feature vector for a camera target and a radar feature vector for a radar target; A heat map data generation module, used to generate camera heat map data corresponding to the camera feature vector and radar heat map data corresponding to the radar feature vector; A nested data obtaining module, used to use a preset data model to perform model processing on the camera thermal map data and the radar thermal map data to obtain nested data; A nested vector generation module, used to generate a camera nested vector and a radar nested vector according to the nested data; A connection relationship generating module, used to train the data model by using the camera nesting vector and the radar nesting vector, or to generate a connection relationship between a camera target and a radar target; The heat map data generation module includes: A camera heat map data generating submodule, used to determine a camera target position of the camera target in a camera plane, and generate camera heat map data corresponding to the camera feature vector according to the camera target position; The radar heat map data generating submodule is used to determine the radar target position where the radar target is projected onto the camera plane, and generate radar heat map data corresponding to the radar feature vector according to the radar target position.

9. A server, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for data processing according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for data processing according to any one of claims 1 to 7 is implemented.

Images