A target positioning method and device for visual and radar fusion

Abstract

The present invention relates to a target positioning method integrating vision and radar. The method includes: S100: Obtain radar point cloud data and first visible light image data, unify the modalities of the radar point cloud data and the first visible light image data to form initial data; S200: Obtain the initial data, design a depth completion network to output a dense depth map; obtain the two-dimensional bounding box information of the target object, combine the dense depth map with the two-dimensional bounding box information, and extract the depth information of the target object on the dense depth map; S300: Use the extracted depth information, the internal parameters of the surround-view camera, and the external parameters between the vehicle-mounted lidar and the surround-view camera to perform three-dimensional position calculation to obtain the coordinate position of the target object in the three-dimensional space; By using the technology of visual and radar data fusion, the radar point cloud information is supplemented and enhanced, and a denser depth map is generated, thereby achieving the effects of enhancing target recognition and precise positioning.

Description

Technical Field

[0001] The present invention relates to the technical field of road traffic, and specifically to a target positioning method and device that integrates vision and radar. Background Art

[0002] The application prospect of autonomous driving technology is very broad. In urban traffic, autonomous vehicles can relieve road congestion, reduce traffic accidents, and improve the utilization rate of social resources. In the logistics field, autonomous driving technology can also improve transportation efficiency, reduce costs, and provide more intelligent solutions for the express delivery industry, logistics industry, etc.

[0003] Chinese Patent Application No. CN202111113213.1 provides a positioning method, device, terminal and medium for a target object. The method includes: during the movement of the target object, acquiring target point cloud data collected by a radar arranged within a set range corresponding to the target object; based on the target point cloud data, acquiring the target point cloud distribution information within the target range corresponding to the current position of the target object; based on the target point cloud distribution information, determining, from the point cloud distribution information included in a pre-established map, the candidate point cloud distribution information that matches the target point cloud distribution information; based on the candidate point cloud distribution information, the candidate point cloud distribution information with the smallest function value of the target error function after the corresponding point cloud data is rotated and / or translated and is the same as the target point cloud data.

[0004] However, relying on radar point cloud data for positioning, as the distance increases, the laser beam emitted by the radar spreads in space, resulting in a decrease in the number of effective returned point clouds. Due to the sparse point cloud data, the radar system may not be able to accurately identify information such as the shape, size, and position of the target, thus leading to difficult or incorrect target recognition. Summary of the Invention

[0005] The present invention aims at the technical problems existing in the prior art and provides a target positioning method and device that integrates vision and radar.

[0006] The technical solution of the present invention to solve the above technical problems is as follows: A target positioning method that integrates vision and radar, the method includes:

[0007] Based on an in-vehicle lidar and an in-vehicle camera group, S100: Acquire radar point cloud data and first visible light image data, unify the modalities of the radar point cloud data and the first visible light image data to form initial data;

[0008] S200: Acquire the initial data, design a depth completion network, and output a dense depth map; acquire the two-dimensional bounding box information of the target object, combine the dense depth map with the two-dimensional bounding box information, and extract the depth information of the target object on the dense depth map;

[0009] S300: Using the extracted depth information, the internal parameters of the surround-view camera, and the external parameters between the vehicle-mounted lidar and the surround-view camera, perform three-dimensional position calculation to obtain the coordinate position of the target object in the three-dimensional space;

[0010] Among them, the two-dimensional image coordinates are converted into normalized coordinates using the internal parameters of the surround-view camera, and the normalized coordinates are converted into three-dimensional world coordinates in combination with the external parameters between the vehicle-mounted lidar and the surround-view camera. The depth information provides the dimensional information during the conversion process.

[0011] Furthermore, it also includes: S400: Obtain the coordinate position of the target object in the three-dimensional space, match the coordinate position with the pre-made high-precision map to determine the final position of the target object on the map; among them, the coordinate position of the target object is converted into the same global coordinate system as the high-precision map, use the sensor to obtain the motion information, the motion information includes the speed and direction information of the current target object, use the map matching algorithm to simultaneously match the converted coordinates and motion information with multiple road networks in the high-precision map to obtain the matching compliance, and then determine the final position of the target object on the map.

[0012] Furthermore, if there is only a unique value above 96% for the matching compliance, the position of this road network is the final position; if there are two or more values above 96% for the matching compliance, it is marked as a suspected position.

[0013] Furthermore, obtain the second visible light image data and the radar point cloud data, extract the key features such as the edges, corners, and textures of the second visible light image data, use the key features to distinguish from the first visible light image data and unify the modality with the radar point cloud data at the corresponding positions to form the re-inspection data; use the re-inspection data to perform matching through the map matching algorithm on the suspected positions to form a new matching degree, and the one with the highest matching degree is the final position.

[0014] Furthermore, obtain the second visible light image data and divide it into front image data and rear image data. Respectively perform preprocessing and stabilization processing on the front image data and the rear image data, extract the key feature points from the processed front image data and rear image data, use the feature matching algorithm to match the feature points in the front image data and the rear image data, establish the corresponding relationship between the images, and form the third visible light image data.

[0015] Further, analyze the motion of feature points between consecutive frames of the front-view image data and the rear-view image data to estimate the motion state of the target object, fuse the information of multiple consecutive frames of images, improve the accuracy and robustness of state estimation, track the feature points, and obtain the motion trajectory between consecutive frames; use the obtained motion trajectory to estimate the state vectors such as the speed, acceleration, and direction change of the target object, and use the Kalman filter algorithm to predict the state of the target object at the next moment according to the state vectors, so as to achieve more accurate positioning.

[0016] Further, the steps of the Kalman filter algorithm include:

[0017] S501: Initialize parameters, the initial state vector and covariance matrix; the state vector contains the estimated value of the system state and the current speed value parameter; the covariance matrix represents the uncertainty of the state vector, its size is the same as the dimension of the state vector, and the diagonal elements of the covariance matrix represent the variances of each state variable;

[0018] S502: Use the dynamic model of the system to predict the state at the next moment, apply the state at the previous moment and the control input to the model to obtain the predicted state; when predicting the state, it is necessary to update the covariance matrix to reflect the increased uncertainty caused by the process noise.

[0019] Further, the steps also include: S503: Calculate the measurement residual and the Kalman gain by using the features extracted from the radar point cloud data and the third visible light image data, and combining the predicted value of the state vector; the measurement residual is the difference between the actual measurement value and the predicted value; the Kalman gain is used as a weighting factor to balance the predicted state and the current measurement value; the Kalman gain is calculated according to the covariance matrix and the measurement noise;

[0020] S504: Use the Kalman gain and the measurement residual to update the state vector, fuse the predicted and measured information to obtain the optimal state estimate at the current moment; update the covariance matrix to reflect the change in the state estimation uncertainty caused by the measurement noise and the Kalman gain, continuously fuse the radar point cloud data and the third visible light image data, and update the state estimation of the vehicle.

[0021] A target positioning device that fuses vision and radar includes:

[0022] An acquisition unit, configured to acquire visible light image data and radar point cloud data and form initial data, wherein, the surround-view camera acquires the first visible light image data, the front camera and the rear camera acquire the second visible light image data, and the lidar acquires the radar point cloud data;

[0023] A processing unit is used to design a depth completion network to process initial data, generate a dense depth map, obtain two-dimensional bounding box information of the target object, extract depth information, and perform fixed storage. During subsequent large target positioning, the above information is directly read and used without repeated calculation.

[0024] A prediction unit is used to predict the state of the target object at the next moment. Among them, the Kalman filtering algorithm is used for data fusion to predict the target object in a high-speed motion state.

[0025] A positioning unit is used to generate the final position of the target object on the map. Among them, three-dimensional position calculation is used to obtain the coordinate position in three-dimensional space, and the map matching algorithm is used to obtain the matching compliance to determine the final position of the target object on the map.

[0026] A computer-readable storage medium stores a computer software program. When the computer software program is executed by a processor, the above-mentioned target positioning method for vision and radar fusion is implemented.

[0027] The beneficial effects of the present invention are as follows: By adopting the technology of vision and radar data fusion, the acquired image data is fused with the radar point cloud data to supplement and enhance the radar point cloud information. A depth completion network is designed to interpolate the sparse point cloud data through an algorithm to generate a denser depth map, thereby achieving the effects of enhancing target recognition and accurate positioning. Description of the Drawings

[0028] Figure 1 It is the overall flowchart of a target positioning method for vision and radar fusion of the present invention;

[0029] Figure 2 It is the algorithm flowchart of a target positioning method for vision and radar fusion of the present invention;

[0030] Figure 3 It is the architecture diagram of a target positioning device for vision and radar fusion of the present invention. Detailed Embodiments

[0031] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0032] In the description of the present application, the terms "first" and "second" are used only for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the present application, "a plurality" means two or more, unless otherwise specifically defined.

[0033] In the description of the present application, the term "for example" is used to mean "serving as an example, illustration, or explanation". Any embodiment described as "for example" in the present application is not necessarily construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the present invention. In the following description, details are set forth for purposes of explanation. It should be understood that those of ordinary skill in the art can recognize that the present invention can be implemented without the use of these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed in the present application.

[0034] Embodiment 1

[0035] As Figure 1 shown, a target positioning method for visual and radar fusion in the present application, the method comprising: based on an on-vehicle lidar and an on-vehicle camera group, S100: acquiring radar point cloud data and first visible light image data, and performing modality unification on the radar point cloud data and the first visible light image data to form initial data.

[0036] Among them, the on-vehicle camera group includes surround-view cameras to achieve 360-degree environmental perception of the target object, and the scanning area is smaller than that of the lidar. The surround-view cameras are used to collect the first visible light image data.

[0037] Specifically, the modality unifies time synchronization, spatial calibration, and data preprocessing for radar point cloud data and the first visible light image data. For time synchronization, a unified clock source is used to synchronize each sensor. Meanwhile, by analyzing the timestamps of the sensor data streams, the closest time points are found to approximately synchronize the data, ensuring that the data captured by different sensors is obtained at the same or similar time points. For spatial calibration, a unified reference coordinate system is selected, and clearly visible feature objects are used as calibration points. The rotation matrix R and translation vector T between the radar point cloud data and the first visible light image data are calculated and optimized. The optimized rotation matrix R and translation vector T are used to align the radar point cloud data and the first visible light image data to the reference coordinate system centered on the calibration points. For data preprocessing, the radar point cloud data is denoised and filtered, and the first visible light image data is enhanced in brightness and contrast and corrected for distortion.

[0038] S200: Obtain initial data, design a depth completion network, and output a dense depth map; obtain the two-dimensional bounding box information of the target object, combine the dense depth map with the two-dimensional bounding box information, and extract the depth information of the target object on the dense depth map.

[0039] Obtain initial data, design a depth completion network, collect and label a set of initial data as labeled data, generate a sparse depth map as the input of the depth completion network using the labeled data, and use the remaining initial data as the true label during training to output a dense depth map.

[0040] Among them, the sparse depth map is generated by projecting the radar point cloud onto the image plane. The dense depth map is the same size as the input image, and each pixel point has a corresponding depth value, representing the distance between a certain point in the scene and the camera.

[0041] Apply a two-dimensional object detection algorithm to the visible light image in the initial data to identify and locate the target object, obtain the two-dimensional bounding box information of the target object, and combine the dense depth map processed by the depth completion network with the two-dimensional bounding box information to extract the depth information of the target object on the dense depth map.

[0042] Specifically, the two-dimensional object detection algorithm identifies the objects existing in the image by performing feature extraction, classification, and localization on the image, and uses a rectangular box to label the target object. The two-dimensional bounding box is used to represent the position and size of the target object in the image.

[0043] S300: Use the extracted depth information, the internal parameters of the surround camera, and the external parameters between the vehicle-mounted lidar and the surround camera to perform three-dimensional position calculation to obtain the coordinate position of the target object in the three-dimensional space.

[0044] Among them, in step S300, the position of the target object in the three-dimensional space is calculated through projective geometric relationships. The perspective projection method is used for three-dimensional position calculation. The internal parameters of the surround-view camera are used to convert the two-dimensional image coordinates into normalized coordinates, and the external parameters between the vehicle-mounted lidar and the surround-view camera are combined to convert the normalized coordinates into three-dimensional world coordinates. The depth information provides the dimensional information during the conversion process.

[0045] S400: Obtain the coordinate position of the target object in the three-dimensional space, match it with a pre-made high-precision map according to the coordinate position, and determine the final position of the target object on the map; among them, convert the coordinate position of the target object into the same global coordinate system as the high-precision map, use the sensor to obtain the motion information, and the motion information includes the speed and direction information of the current target object. Use the map matching algorithm to simultaneously match the converted coordinates and motion information with multiple road networks in the high-precision map to obtain the matching compliance, and then determine the final position of the target object on the map.

[0046] Among them, the map matching algorithm receives the converted coordinates and motion information, scans the position on the map through the point cloud, calculates the error and distance between the scanned points and the corresponding points on the high-precision map at each position, then sums the squares of the errors, and finds the candidate road sections similar to the target object according to the sum of the squared errors. Calculate the distance, angle difference, and trajectory information between the roads through the coordinate data to obtain the error value range, and obtain the matching compliance according to the error value range. The smaller the error value range, the higher the matching compliance.

[0047] The technical solutions in the above embodiments of the present application have at least the following technical effects or advantages:

[0048] The present application adopts the technology of visual and radar data fusion. By fusing the acquired image data with the radar point cloud data, the radar point cloud information is supplemented and enhanced. A depth completion network is designed to interpolate the sparse point cloud data through an algorithm to generate a denser depth map, thereby achieving the effects of enhancing target recognition and precise positioning.

[0049] Embodiment 2

[0050] In Embodiment 1, the technology of visual and radar fusion is used to achieve precise positioning. However, when identifying the surrounding area of the target object, there are a large number of highly similar road areas, resulting in positioning errors.

[0051] If there is only a unique value above 96% for the matching compliance, the position of this road network is the final position; if there are two or more values above 96% for the matching compliance, it is marked as a suspected position.

[0052] Among them, the vehicle-mounted camera group further includes a front camera and a rear camera. The front camera and the rear camera respectively perform long-distance scanning on the front and rear of the target object. The scanning areas are fan-shaped with the target object as the center. The scanning ranges of the front camera and the rear camera are both larger than that of the surround-view camera. The area beyond the scanning area of the surround-view camera is the difference area, and the second visible light image data is collected for the difference area.

[0053] Obtain the second visible light image data and the radar point cloud data, extract the key features such as the edges, corner points and textures of the second visible light image data, use the key features to distinguish from the first visible light image data and unify the modality with the radar point cloud data at the corresponding position to form re-inspection data; use the re-inspection data to match the suspected position through the map matching algorithm to form a new matching degree, and the one with the highest matching degree is the final position.

[0054] The technical solutions in the above embodiments of the present application at least have the following technical effects or advantages:

[0055] The present application uses the front camera and the rear camera to perform long-distance scanning to obtain the second visible light image data, performs secondary recognition on the front and rear of the target object to expand the scanning area, and separately recognizes the suspected position through the map matching algorithm, thereby achieving the effect of accurately positioning highly similar road areas.

[0056] Embodiment 3

[0057] In the above embodiment, secondary recognition is performed on the front and rear areas to achieve accurate positioning of highly similar road areas. However, when the target object is moving at a high speed and the lidar is used to scan the target object, the position of the first laser beam and the position of the last laser beam (after the lidar scans a circle) have a relative displacement. The distance of the relative displacement is determined by the vehicle speed. The point clouds of the target object at two different times show different information in the coordinate system, resulting in the lag of positioning and causing errors in positioning.

[0058] Obtain the second visible light image data and divide it into front image data and rear image data. Respectively perform preprocessing and stabilization processing on the front image data and the rear image data. Extract key feature points from the processed front image data and rear image data. Use the feature matching algorithm to match the feature points in the front image data and the rear image data to establish the corresponding relationship between the images and form the third visible light image data.

[0059] Analyze the motion of feature points between consecutive frames of the front-view image data and the rear-view image data to estimate the motion state of the target object. Integrate the information of multiple consecutive frames of images to improve the accuracy and robustness of state estimation. Track the feature points to obtain the motion trajectories between consecutive frames. Use the obtained motion trajectories to estimate the state vectors such as the speed, acceleration, and direction change of the target object. Use the Kalman filter algorithm to predict the state of the target object at the next moment based on the state vectors, so as to achieve more accurate positioning.

[0060] Among them, the front camera collects the front-view image data, and the rear camera collects the rear-view image data. Preprocess the collected front-view images and rear-view images, including denoising, distortion correction, brightness and contrast adjustment, to improve the image quality. For high-speed dynamic environments, perform stabilization processing through image registration to reduce the impact of fast motion on the images. The motion state of the target object includes speed, acceleration, angular velocity, and angular acceleration.

[0061] The front-view image and the rear-view image obtained at the same moment are grouped together. Collect multiple groups of front-view images and rear-view images. Extract the feature and state vector of the rear-view image data as the input of the Kalman filter algorithm. Use the feature and state vector of the front-view image data at the same moment to verify with the state predicted by the Kalman filter algorithm to improve the adaptability of the Kalman filter algorithm to the image data in the current scene.

[0062] As Figure 2 shown, the steps of the Kalman filter algorithm include:

[0063] S501: Initialize the parameters, the initial state vector and the covariance matrix; the state vector contains the estimated value of the system state and the current speed value parameter; the covariance matrix represents the uncertainty of the state vector, and its size is the same as the dimension of the state vector. The diagonal elements of the covariance matrix represent the variances of each state variable.

[0064] Specifically, the covariance matrix is a tool for describing the correlation between the components of a multi-dimensional random variable. In the Kalman filter, it is used to represent the uncertainty of the state vector and the correlation between each state variable. By continuously updating the covariance matrix, the Kalman filter algorithm can dynamically adjust the uncertainty of the state estimation, so as to achieve more accurate state estimation. At initialization, set the covariance matrix according to the initial uncertainty of the system state. The specific value depends on the prior knowledge of the system state and set different variance values for the uncertainty of the initial vector parameters. For example, if the uncertainty of the initial position is large and the uncertainty of the speed is small, a larger position variance and a smaller speed variance can be set.

[0065] S502: Use the dynamic model of the system to predict the state at the next moment. Apply the state at the previous moment and the control input to the model to obtain the predicted state. When predicting the state, the covariance matrix needs to be updated to reflect the increased uncertainty caused by process noise.

[0066] Specifically, the dynamic model uses the vehicle's motion equation to describe the vehicle's dynamic behavior. It is established based on the vehicle's physical characteristics and kinematic constraints to achieve the prediction of the vehicle's state at the next moment. Generate the state transition matrix and the control input matrix according to the initial state vector of the target object. The state transition matrix represents the process of how the state evolves over time without control input. The control input matrix describes how the control input affects the change of the state.

[0067] Among them, the process noise represents the uncertainty of the model and external disturbances. The process noise is modeled as zero-mean Gaussian white noise, and its covariance matrix is used to describe the uncertainty of the noise. The selection of the covariance matrix is adjusted according to the actual application scenario and experience. The covariance matrix is set as a diagonal matrix, where the diagonal elements represent the variances of the process noise of each state variable.

[0068] S503: Use the features extracted from the radar point cloud data and the third visible light image data, combined with the predicted value of the state vector, to calculate the measurement residual and the Kalman gain. The measurement residual is the difference between the actual measurement value and the predicted value. The Kalman gain is used as a weighting factor to balance the predicted state and the current measurement value. The Kalman gain is calculated based on the covariance matrix and the measurement noise.

[0069] S504: Use the Kalman gain and the measurement residual to update the state vector, fuse the predicted and measured information to obtain the optimal state estimate at the current moment. Update the covariance matrix to reflect the change in the state estimate uncertainty caused by the measurement noise and the Kalman gain. Continuously fuse the radar point cloud data and the third visible light image data and update the state estimate of the vehicle.

[0070] Specifically, use the Kalman filtering algorithm to fuse the third visible light image data and the radar point cloud data, predict the state of the target object at the next moment, and use the map matching algorithm in the above embodiments for matching according to the predicted data state information at the next moment to obtain the final position of the target object on the map.

[0071] Specifically, multiple groups of front images and rear images acquired at the same moment are used as control groups and input in batches. The feature data and state vectors of the front images at the same moment are verified with the states predicted by the Kalman filtering algorithm, and the real-time state vectors of the target objects are continuously updated to improve the performance of the Kalman filtering algorithm in this scenario. The third visible light image data formed by feature fusion of subsequent front images and rear images can display the association between the feature states of the front images and rear images to the greatest extent, further improving the accuracy of positioning.

[0072] The technical solutions in the embodiments of the present application at least have the following technical effects or advantages:

[0073] The present application uses the Kalman filtering algorithm for data fusion to predict the state of the target object at the next moment, ensuring that effective data information can be obtained for positioning in a high-speed motion state. Multiple groups of front images and rear images acquired at the same moment are used as control groups and input in batches to continuously improve the adaptability of the Kalman filtering algorithm in this scenario, thereby achieving the effect of accurate positioning of the predicted motion state.

[0074] Embodiment 4

[0075] Based on the above embodiments, as Figure 3 shown, the embodiments of the present application also provide a target positioning device for visual and radar fusion, including: an acquisition unit for acquiring visible light image data and radar point cloud data and forming initial data, wherein the panoramic camera collects the first visible light image data, the front camera and the rear camera collect the second visible light image data, and the lidar collects the radar point cloud data;

[0076] A processing unit for designing a depth completion network to process the initial data, generate a dense depth map, obtain the two-dimensional bounding box information of the target object, extract the depth information, and save it fixedly. The above information is directly read and used in the subsequent large target positioning process without repeated calculation;

[0077] A prediction unit for predicting the state of the target object at the next moment, wherein the Kalman filtering algorithm is used for data fusion to predict the target object in a high-speed motion state;

[0078] A positioning unit for generating the final position of the target object on the map, wherein the three-dimensional position solution is used to obtain the coordinate position in the three-dimensional space, and the map matching algorithm is used to obtain the matching degree to determine the final position of the target object on the map.

[0079] The embodiments of the present application also provide a computer-readable storage medium, in which a computer software program is stored, and when the computer software program is executed by a processor, the above-mentioned target positioning method for visual and radar fusion is implemented.

[0080] It should be noted that in the above embodiments, the descriptions of each embodiment have their own focuses. For the parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0081] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram can be realized by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the specified functions in Figure 1 one or more of the processes Figure 1 or multiple processes and / or blocks

[0083] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means realizes the specified functions in Figure 1 one or more of the processes Figure 1 or multiple processes and / or blocks

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for realizing the specified functions in Figure 1 one or more of the processes Figure 1 or multiple processes and / or blocks

[0085] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once they learn of the basic inventive concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention.

[0086] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A target positioning method that fuses vision and radar, characterized in that The method includes: based on an on-vehicle lidar and an on-vehicle camera group, the on-vehicle camera group includes surround-view cameras, and using the surround-view cameras to collect first visible light image data; S100: obtaining lidar point cloud data and first visible light image data, unifying the modalities of the lidar point cloud data and the first visible light image data to form initial data; S200: obtaining the initial data, designing a depth completion network to output a dense depth map; obtaining two-dimensional bounding box information of the target object, combining the dense depth map with the two-dimensional bounding box information to extract the depth information of the target object on the dense depth map; S300: using the extracted depth information, the internal parameters of the surround-view camera, and the external parameters between the on-vehicle lidar and the surround-view camera to perform three-dimensional position calculation to obtain the coordinate position of the target object in three-dimensional space; S400: obtaining the coordinate position of the target object in three-dimensional space, matching the coordinate position with a pre-made high-precision map to determine the final position of the target object on the map; obtaining a matching compliance; if there are two or more values of the matching compliance above 96%, it is marked as a suspected position; Obtaining second visible light image data and lidar point cloud data, extracting key features of the second visible light image data, using the key features to distinguish from the first visible light image data and unifying the modalities with the lidar point cloud data at the corresponding positions to form re-inspection data; using the re-inspection data to perform matching on the suspected positions through a map matching algorithm to form a new matching degree, and the one with the highest matching degree is the final position; Among them, the two-dimensional image coordinates are converted into normalized coordinates by using the internal parameters of the surround-view camera, and the normalized coordinates are converted into three-dimensional world coordinates by combining the external parameters between the on-vehicle lidar and the surround-view camera, and the depth information provides the dimension information in the conversion process.

2. The object positioning method that fuses vision and radar according to claim 1, characterized in that, Converting the coordinate position of the target object into the same global coordinate system as the high-precision map, using sensors to obtain motion information, the motion information includes the speed and direction information of the current target object, and using a map matching algorithm to match the converted coordinates and motion information with multiple road networks in the high-precision map simultaneously to obtain a matching compliance, and further determining the final position of the target object on the map.

3. A target positioning method integrating vision and radar according to claim 1, characterized in that, If there is only a single value of the matching compliance above 96%, the position of the road network is the final position.

4. A target positioning method integrating vision and radar according to claim 1, characterized in that, Obtaining second visible light image data and performing division, dividing it into front image data and rear image data, respectively performing preprocessing and stabilization processing on the front image data and the rear image data, extracting key feature points from the processed front image data and rear image data, using a feature matching algorithm to match the feature points in the front image data and the rear image data to establish a corresponding relationship between the images to form third visible light image data.

5. A target positioning method that fuses vision and radar according to claim 4, characterized in that, Analyze the motion of feature points between consecutive frames of pre-image data and rear-view image data to estimate the motion state of the target object, fuse the information of multiple consecutive frames of images to improve the accuracy and robustness of state estimation, track the feature points, and obtain the motion trajectory between consecutive frames; use the obtained motion trajectory to estimate the state vectors such as the speed, acceleration, and direction change of the target object, and use the Kalman filter algorithm to predict the state of the target object at the next moment according to the state vectors to achieve more accurate positioning.

6. The method for target positioning by fusing vision and radar according to claim 5, characterized in that, The steps of the Kalman filter algorithm include: S501: Initialize parameters, the initial state vector and covariance matrix; the state vector includes the estimated value of the system state and the current speed value parameter; the covariance matrix represents the uncertainty of the state vector, its size is the same as the dimension of the state vector, and the diagonal elements of the covariance matrix represent the variances of each state variable; S502: Use the dynamic model of the system to predict the state at the next moment, apply the state at the previous moment and the control input to the model to obtain the predicted state; when predicting the state, it is necessary to update the covariance matrix to reflect the increased uncertainty caused by process noise.

7. A target positioning method integrating vision and radar according to claim 6, characterized in that, The steps further include: S503: Calculate the measurement residual and Kalman gain by using the features extracted from the radar point cloud data and the third visible light image data, in combination with the predicted value of the state vector; the measurement residual is the difference between the actual measurement value and the predicted value; the Kalman gain is used as a weighting factor to balance the predicted state and the current measurement value; the Kalman gain is calculated according to the covariance matrix and the measurement noise; S504: Use the Kalman gain and the measurement residual to update the state vector, fuse the predicted and measured information to obtain the optimal state estimate at the current moment; update the covariance matrix to reflect the change in the state estimation uncertainty caused by the measurement noise and the Kalman gain, continuously fuse the radar point cloud data and the third visible light image data, and update the state estimation of the vehicle.

8. An object positioning device that integrates vision and radar, characterized in that, It includes: An acquisition unit for acquiring radar point cloud data and first visible light image data, unifying the modalities of the radar point cloud data and the first visible light image data to form initial data; Acquire the second visible light image data and the radar point cloud data to form re-inspection data; among them, the surround-view camera acquires the first visible light image data, the front camera and the rear camera acquire the second visible light image data, and the lidar acquires the radar point cloud data; A processing unit for designing a depth completion network to process the initial data to generate a dense depth map, applying a two-dimensional object detection algorithm to the visible light image in the initial data to identify and locate the target object, obtain the two-dimensional bounding box information of the target object, extract the depth information, and perform fixed storage; A prediction unit for predicting the state of the target object at the next moment, where the Kalman filter algorithm is used for data fusion to predict the target object in a high-speed motion state; A positioning unit for generating the final position of the target object on the map, where the three-dimensional position solution is used to obtain the coordinate position in the three-dimensional space, and the map matching algorithm is used to obtain the matching compliance to determine the final position of the target object on the map.

9. A computer-readable storage medium, characterized in that, The computer software program is stored in the storage medium, and when the computer software program is executed by a processor, it implements the visual and radar fusion-based target localization method according to any one of claims 1-7.

Images