Millimeter wave radar and monocular camera fused information acquisition device and method

Abstract

The invention discloses a millimeter wave radar and monocular camera fused information acquisition device and method, and relates to the technical field of target detection. The device comprises a millimeter wave radar, a monocular camera and an adjusting bracket, wherein the adjusting bracket can adjust the positions and angles of the radar and the camera. And the data fusion module performs time synchronization and space alignment on the radar information and the image information, and obtains a target recognition result through a preset target recognition model. The acquisition method comprises the steps of timestamp matching, time compensation, joint calibration, coordinate transformation, image distortion correction and the like. Further, radar physical features are extracted by using a CFAR algorithm, image features are extracted by using a deep learning model, and effective features are obtained through PCA dimension reduction. And finally, an accurate target recognition result is obtained through feature splicing and optimization by using a support vector machine (SVM). According to the method, efficient fusion and target identification of millimeter wave radar and monocular camera information are realized, and the method has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to an information acquisition device and method that integrates a millimeter-wave radar and a monocular camera. Background Art

[0002] With the development of science and technology, information collection technology is of great significance in the fields of intelligent transportation, unmanned driving, robot navigation, etc. Existing information collection devices mainly include millimeter wave radar and monocular camera, but both have certain limitations in the information collection process. Millimeter wave radar has good penetration and distance measurement capabilities, but it has deficiencies in angle resolution and target recognition; monocular camera has advantages in target recognition and scene understanding, but is greatly affected by lighting and weather conditions. How can the information collection device effectively integrate the advantages of the two, make up for their respective shortcomings, and improve the comprehensive performance of the perception system is a key issue facing the current intelligent perception technology. Summary of the invention

[0003] In order to overcome the shortcomings of the above-mentioned existing information collection devices in how to effectively integrate the advantages of both, the main purpose of the present invention is to provide an information collection device and method that integrates millimeter-wave radar and a monocular camera.

[0004] To achieve the above object, the present invention adopts the following technical solution, a millimeter wave radar and monocular camera fusion information acquisition device, characterized in that it includes:

[0005] At least one millimeter wave radar and at least one monocular camera; the millimeter wave radar is used to collect radar information of the target object; the monocular camera is used to collect image information of the target surrounding environment;

[0006] The adjusting bracket includes a fixing component 1, a direction adjustment component 2 and a plurality of height adjustment components 3, wherein the plurality of height adjustment components 3 are respectively fixedly connected to the direction adjustment component 2, and the millimeter wave radar and the monocular camera are respectively fixedly mounted on the plurality of height adjustment components 3 to collect radar information and image information at different heights and directions;

[0007] The data fusion module is used to control the rotation and lifting of the adjustment component, receive radar information and image information at different heights and directions, synchronize the time and align the space of the radar information and image information, extract effective features, and obtain the target recognition result through the preset target recognition model.

[0008] The fixing component 1 is a metal frame fixed by bolts;

[0009] The direction adjustment assembly 2 includes a turntable and a motor for driving the turntable to rotate;

[0010] The plurality of height adjustment components 3 include a plurality of hydraulic lifting columns.

[0011] The acquisition method of the information acquisition device integrating millimeter wave radar and monocular camera, wherein the radar information and the image information are synchronized in time and aligned in space, specifically includes:

[0012] The timestamp of the time when the radar information and image information are obtained;

[0013] Obtain the corresponding frames with the closest timestamps in the radar information and image information as candidate pairs for time synchronization, perform timestamp matching, and obtain matching results;

[0014] Obtain the time difference between radar information and image information, perform time compensation, and obtain compensated data; obtain verified synchronized data by comparing the timing characteristics of synchronized data;

[0015] Jointly calibrate the millimeter-wave radar and monocular camera to determine the relative position and attitude relationship and obtain the calibration parameters;

[0016] According to the calibration parameters, the transformation relationship between the radar coordinate system and the image coordinate system is established in combination with the synchronized data;

[0017] Convert the target points detected by the radar into coordinates in the world coordinate system;

[0018] Perform distortion correction on the image to obtain a corrected image;

[0019] Based on the coordinate transformation matrix, the target point in the world coordinate system is projected into the image coordinate system to obtain the aligned radar data and image data.

[0020] It is characterized in that the obtaining of effective features specifically includes:

[0021] Select the CFAR algorithm to extract the aligned radar data and obtain the radar physical characteristics;

[0022] After preprocessing the aligned image data, the deep learning model is used to extract features and obtain image features;

[0023] PCA is used to reduce the dimension of radar physical features and images, and the reduced dimension radar physical features and image features are obtained as effective features.

[0024] The target recognition result is obtained by using a preset target recognition model, specifically including: fusing the radar physical features and image features after dimensionality reduction by feature splicing, and optimizing them by machine learning to obtain the target recognition result.

[0025] The step of fusing the radar physical features and image features after dimensionality reduction by feature splicing includes the following steps:

[0026] The radar physical features and image features after dimension reduction are processed by minimum-maximum normalization to obtain standard radar physical features and image features;

[0027] Standard radar physical features and image features are weightedly spliced ​​by dimension to obtain splicing features.

[0028] The machine learning selects a support vector machine;

[0029] The support vector machine optimization is used to obtain the target recognition result. The following steps are included:

[0030] Divide the concatenated features into training and test sets;

[0031] The kernel function of the support vector machine is selected as the Gaussian radial basis kernel, the decision boundary is learned by the support vector machine using the training set, the optimization goal is determined to be the maximum classification interval, the penalty factor and the kernel function are adjusted, and the training is verified by the test machine to obtain the trained support vector machine;

[0032] The radar information and image information of the target are input into the trained support vector machine to obtain the target recognition result.

[0033] Compared with the prior art, the beneficial effects of the present invention are:

[0034] 1. The present invention can effectively utilize the penetration ability of radar and the high-resolution image information of the camera by integrating millimeter-wave radar and monocular camera. Compared with a single sensor system, the present invention greatly improves the accuracy of target detection and recognition in complex environments, especially the target recognition ability in low visibility, at night or in severe weather conditions is significantly enhanced.

[0035] 2. The present invention solves the problem of inconsistency between radar data and image data in time and space through precise time synchronization and space alignment technology, ensures efficient fusion and consistency of data between radar and camera, and thus improves the accuracy and reliability of target recognition.

[0036] 3. The CFAR algorithm is used to extract physical features from radar data, and the deep learning model is combined to extract features from image data. The PCA dimensionality reduction technology is further used to reduce data redundancy and improve processing efficiency. At the same time, the accuracy of effective features is ensured, effectively improving the accuracy of target recognition.

[0037] 4. The present invention ensures the accuracy of the relative position and posture relationship between the millimeter-wave radar and the camera through joint calibration technology, so that the robustness of target recognition in dynamic environments is enhanced, and it can better adapt to the changes of moving targets or complex scenes. By splicing the radar and image features after dimensionality reduction and optimizing them with support vector machines (SVM), the present invention realizes accurate recognition of targets in complex scenes. The SVM model has strong robustness and generalization ability when processing high-dimensional features, and can better adapt to target recognition tasks in different scenes.

[0038] 5. The present invention avoids the common complex hardware matching and high cost problems in multi-sensor systems by adopting a fusion solution of a monocular camera and a millimeter-wave radar, reduces the hardware requirements of the system, and improves the accuracy and efficiency of target recognition through a data fusion method, thereby reducing the development and maintenance costs of the overall system.

[0039] In summary, the information acquisition device and method for integrating millimeter-wave radar and monocular camera of the present invention have the advantages of high precision, strong robustness, high integration, wide adaptability, etc., and can effectively improve the performance of the target recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0041] Figure 1 It is a schematic diagram of the structure of the adjusting bracket of the present invention;

[0042] Figure 2 It is a schematic diagram of the process framework of the present invention;

[0043] Figure 3 It is a partial flow diagram of the present invention;

[0044] Figure 4 It is a flow chart of obtaining target recognition results of the present invention.

[0045] 1. Fixed components, 2. Direction adjustment components, 3. Multiple height adjustment components. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0047] Example:

[0048] The information collection device of the millimeter wave radar and monocular camera fusion provided in this embodiment includes at least one radar sensor of the millimeter wave radar model XYZ123 and at least one high-definition camera of the monocular camera model ABC456. The millimeter wave radar is used to collect radar information of the target object, and the monocular camera is used to collect image information of the target surrounding environment.

[0049] The information collection device includes an adjustment bracket. The fixing part of the bracket is a metal frame, which can firmly fix the millimeter-wave radar and the monocular camera in the specified position. The direction adjustment component of the bracket is designed as a mechanical structure that can rotate around a vertical axis, and its direction is adjusted by a motor drive. Multiple height adjustment components support the adjustment of the height of the millimeter-wave radar and the monocular camera through an electric control device, so that they can adapt to different installation environments and target collection requirements. Specifically, the height adjustment component of the bracket includes multiple hydraulic lifting columns that can be adjusted within the range of 0 to 2 meters.

[0050] The data fusion module is responsible for time synchronization and spatial alignment of radar information and image information collected by the millimeter wave radar and monocular camera. The specific process is as follows:

[0051] Both radar information and image information have timestamps. The data fusion module first obtains the timestamps of these two data sources.

[0052] On this basis, the radar data frame and image data frame with the closest timestamps are selected as candidate pairs for time synchronization and timestamp matching is performed.

[0053] By calculating the time difference and performing time compensation, the consistency of the two on the time axis is ensured. At this point, the accuracy of the synchronization data can be further verified by comparing the timing characteristics.

[0054] In order to accurately fuse the detection data of the millimeter-wave radar and the monocular camera, the millimeter-wave radar and the monocular camera are jointly calibrated in this embodiment. The calibration process is carried out through a static calibration plate, which contains multiple marking points with known positions. By collecting the radar information and image information of these points, the relative position and attitude relationship between the radar coordinate system and the camera coordinate system is calculated using the least squares method. Through this calibration, the calibration parameters obtained include the rotation matrix and the displacement vector.

[0055] Subsequently, the data fusion module combines the synchronized radar data and image data according to the calibration parameters and uses the coordinate transformation matrix to transform the target points detected by the radar from the radar coordinate system to the world coordinate system, and then projects the target points in the world coordinate system into the camera coordinate system, thus achieving spatial alignment of the radar data and the image data.

[0056] After the image and radar data are aligned, effective features are extracted in the following way:

[0057] First, the common CFAR algorithm is used to preprocess the radar data to extract the physical characteristics of the target object such as the target's speed and distance.

[0058] By preprocessing the image data such as denoising and histogram equalization, deep learning models such as convolutional neural networks are used to extract high-level features such as the shape and color of the target from the image.

[0059] The extracted radar physical features and image features will be processed by PCA for dimensionality reduction to reduce redundant information and improve feature processing efficiency. The radar physical features and image features after PCA dimensionality reduction are used as effective features for subsequent target recognition.

[0060] Based on the radar physical features and image features after dimensionality reduction, feature fusion is performed through feature splicing. The specific steps are as follows:

[0061] First, the radar features and image features after dimensionality reduction are normalized by min-max so that they have the same scale.

[0062] Then, the radar physical features and image features are weighted and spliced ​​by dimension to obtain a comprehensive spliced ​​feature vector.

[0063] The concatenated feature vector is input into the support vector machine (SVM) classifier for target recognition. The kernel function of the support vector machine is selected as the Gaussian radial basis kernel function, and the classifier is trained using the training set. During the training process, the optimization goal is to maximize the classification interval, while adjusting the penalty factor and kernel function parameters to improve the accuracy of recognition.

[0064] Finally, the trained support vector machine (SVM) model is applied to the target recognition task of real-time data. The newly acquired radar information and image information are input into the SVM model, and the final target recognition result is output. The recognition result includes the category of the target such as pedestrians, vehicles, etc. and its position in the image.

[0065] Embodiment 2:

[0066] In this embodiment, the calibration method is further improved. In addition to the static calibration plate, a dynamic calibration method is introduced. Through real-time motion calibration, the error that may be caused by the fixed calibration plate is overcome. This method is suitable for mobile platforms, especially in vehicle-mounted or unmanned driving systems, and can automatically perform real-time correction to improve the accuracy and robustness of the system.

[0067] This embodiment achieves efficient and accurate target recognition by integrating millimeter-wave radar and monocular camera, combining time synchronization, spatial alignment, joint calibration, feature extraction, dimensionality reduction and support vector machine optimization technology. This information collection device can not only be widely used in fields such as autonomous driving, drones and intelligent monitoring, but also can further improve the accuracy and real-time performance of recognition through optimization algorithms.

[0068] It should be noted that, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0069] The above embodiments are merely examples of the present invention and do not limit the protection scope of the present invention. All designs that are the same or similar to the present invention fall within the protection scope of the present invention.

Claims

1. An information collection device integrating millimeter wave radar and monocular camera, characterized in that: include: At least one millimeter-wave radar and at least one monocular camera; The millimeter wave radar is used to collect radar information of the target object; The monocular camera is used to collect image information of the target's surrounding environment; The adjustment bracket comprises a fixing component (1), a direction adjustment component (2) and a plurality of height adjustment components (3), wherein the plurality of height adjustment components (3) are respectively fixedly connected to the direction adjustment component (2), and the millimeter wave radar and the monocular camera are respectively fixedly mounted on the plurality of height adjustment components (3) to collect radar information and image information at different heights and directions; The data fusion module is used to control the rotation and lifting of the adjustment component, receive radar information and image information at different heights and directions, synchronize the time and align the space of the radar information and image information, extract effective features, and obtain the target recognition result through the preset target recognition model.

2. The information acquisition device integrating millimeter wave radar and monocular camera as claimed in claim 1, characterized in that: include: The fixing component (1) is a metal frame fixed by bolts; The direction adjustment component (2) comprises a turntable and a motor for driving the turntable to rotate; The plurality of height adjustment components (3) include a plurality of hydraulic lifting columns.

3. The acquisition method of the information acquisition device of the millimeter wave radar and monocular camera fusion as claimed in claim 1, characterized in that: The time synchronization and spatial alignment of radar information and image information specifically include: The timestamp of the time when the radar information and image information are obtained; Obtain the corresponding frames with the closest timestamps in the radar information and image information as candidate pairs for time synchronization, perform timestamp matching, and obtain matching results; Obtain the time difference between radar information and image information, perform time compensation, and obtain compensated data; obtain verified synchronized radar information and image information by comparing the timing characteristics of synchronized data; Jointly calibrate the millimeter-wave radar and monocular camera to determine the relative position and attitude relationship and obtain the calibration parameters; According to the calibration parameters, the transformation relationship between the radar coordinate system and the image coordinate system is established in combination with the verified synchronous radar information and image information; Convert the target points detected by the radar into coordinates in the world coordinate system; Perform distortion correction on the image to obtain a corrected image; Based on the coordinate transformation matrix, the target point in the world coordinate system is projected into the image coordinate system to obtain the aligned radar data and image data.

4. The acquisition method of the information acquisition device of the millimeter wave radar and the monocular camera fusion as claimed in claim 3 is characterized in that: The obtaining of effective features specifically includes: Select the CFAR algorithm to extract the aligned radar data and obtain the radar physical characteristics; After preprocessing the aligned image data, the deep learning model is used to extract features and obtain image features; PCA is used to reduce the dimension of radar physical features and images, and the radar physical features and image features after dimension reduction are obtained as effective features.

5. The acquisition method of the information acquisition device of the millimeter wave radar and monocular camera fusion as claimed in claim 1, characterized in that: The target recognition result is obtained by using a preset target recognition model, specifically including: fusing the radar physical features and image features after dimensionality reduction by feature splicing, and optimizing them by machine learning to obtain the target recognition result.

6. The acquisition method of the information acquisition device of the millimeter wave radar and monocular camera fusion as claimed in claim 4, characterized in that: The step of fusing the radar physical features and image features after dimensionality reduction by feature splicing includes the following steps: The radar physical features and image features after dimension reduction are processed by minimum-maximum normalization to obtain standard radar physical features and image features; Standard radar physical features and image features are weightedly spliced ​​by dimension to obtain splicing features.

7. The acquisition method of the information acquisition device of the millimeter wave radar and the monocular camera fusion as claimed in claim 4, characterized in that: The machine learning selects a support vector machine; The method of using support vector machine optimization to obtain target recognition results includes the following steps: Divide the concatenated features into training and test sets; The kernel function of the support vector machine is selected as the Gaussian radial basis kernel, the decision boundary is learned by the support vector machine using the training set, the optimization goal is determined to be the maximum classification interval, the penalty factor and the kernel function are adjusted, and the training is verified by the test machine to obtain the trained support vector machine; The radar information and image information of the target are input into the trained support vector machine to obtain the target recognition result.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 3 to 7 is implemented.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of claims 3 to 7 is implemented.

Images