Multi-modal three-dimensional target recognition and positioning method applied to automatic driving

Abstract

The present application relates to the technical field of automatic driving recognition, and especially relates to a multi-modal three-dimensional target recognition and positioning method applied to automatic driving; the present application fuses multi-modal data of various sensors such as laser radar, camera, millimeter wave radar and the like, realizes complementation of information, improves accuracy of target recognition and positioning, meets the demand of automatic driving vehicles for surrounding environment perception, provides reliable guarantee for safe driving of the automatic driving vehicles, meanwhile, the automatic driving vehicles are accompanied by classification of the categories of targets and driving autonomous planning process after the classification, so as to improve driving safety and autonomous planning flexibility of the automatic driving vehicles, and the three-dimensional target recognition and positioning model of the automatic driving vehicles is comprehensively evaluated from two dimensions of positioning performance and recognition and positioning performance, so as to improve target recognition and positioning accuracy and autonomous learning management effect of the automatic driving vehicles.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving recognition technology, and in particular to a multimodal three-dimensional target recognition and localization method for autonomous driving. Background Technology

[0002] With the rapid development of autonomous driving technology, accurate perception of the surrounding environment is the key to achieving safe autonomous driving. Target detection plays a crucial role in the entire autonomous driving system. Its main task is to accurately identify and locate key targets such as vehicles, pedestrians, and traffic signs in complex road environments, thereby providing important input information for the vehicle's decision-making system.

[0003] Currently, autonomous vehicles typically use a single sensor or a few sensors for target recognition and localization. Their target recognition capabilities are limited, which makes it difficult for autonomous vehicles to accurately identify and locate targets and plan routes reasonably. This affects the normal operation of autonomous vehicles and makes it difficult to manage traffic reasonably based on the type of target, resulting in a decrease in the reliability of target recognition and localization. Therefore, relying on a single sensor is insufficient to meet the high-precision requirements of autonomous vehicles for target recognition and localization in complex environments.

[0004] To address the aforementioned technical shortcomings, a solution is proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal three-dimensional target recognition and localization method for autonomous driving, in order to solve the aforementioned technical defects. This invention makes full use of the advantages of different sensors, realizes information complementarity, and provides a reliable guarantee for the safe driving of autonomous vehicles. At the same time, it is accompanied by the autonomous vehicle's classification of target categories and the autonomous driving planning process after classification, so as to improve the driving safety and autonomous planning flexibility of autonomous vehicles, and improve the target recognition and localization accuracy and autonomous learning management effect of autonomous vehicles.

[0006] The objective of this invention can be achieved through the following technical solution: a multimodal three-dimensional target recognition and localization method for autonomous driving, comprising the following steps:

[0007] S1: The process of multimodal data acquisition and vehicle coordinate system construction for autonomous vehicles;

[0008] S2: The process of preprocessing multimodal data and acquiring multimodal fusion features;

[0009] S3: Target recognition and positioning analysis process based on the combination of a 3D target recognition and positioning model and a vehicle coordinate system;

[0010] S4: The target classification process, which is to determine whether the category in the basic information is dynamic. If it is, it is determined to be a dynamic target; otherwise, it is determined to be a static target.

[0011] S5: The process of controlling and managing the driving route of autonomous vehicles based on the classification of targets;

[0012] S6: Autonomous planning, management and analysis process of a three-dimensional target recognition and positioning model based on a comprehensive evaluation of recognition and positioning performance and characteristics.

[0013] Preferably, the process of S1 is as follows: establish the vehicle coordinate system of the autonomous vehicle, collect point cloud data within a pre-set detection range through the lidar on the autonomous vehicle, collect RGB image data within a pre-set detection range through the high-definition camera, and collect radar echo data within a pre-set detection range through the millimeter-wave radar.

[0014] Point cloud data, RGB image data, and radar echo data are collectively referred to as multimodal data.

[0015] Preferably, the process of S2 is as follows: preprocessing the multimodal data, performing time calibration on the preprocessed multimodal data using a linear interpolation method, calibrating the multimodal data to the same time point and transforming it to the vehicle coordinate system to achieve spatial calibration.

[0016] Preferably, the S3 process is as follows: inputting the multimodal fusion features into the retrieved 3D target recognition and localization model to obtain the basic information of each target output by the 3D target recognition and localization model, including the vehicle's 3D coordinates, size, and category.

[0017] Preferably, the process of S5 is as follows: S5 includes two steps, S51 and S52;

[0018] S51: When the target is a static target, the driving route of the autonomous vehicle is obtained, and the collision between the driving route and the static target is determined. If the collision occurs, a route adjustment signal is generated; otherwise, a pass signal is generated.

[0019] S52: When the target is a dynamic target, acquire the operating data of the dynamic target and the autonomous vehicle. The operating data includes the operating direction and the operating speed. Then, determine whether the operating directions of the dynamic target and the autonomous vehicle are the same. If they are, generate a same-direction signal; otherwise, generate a different-direction signal.

[0020] Preferably, S51 includes S511: when generating a route adjustment signal: based on the driving route, the safe distance of the autonomous vehicle passing through the static target is obtained, the static target is set as the center of the circle, and the safe distance is set as the radius, so as to draw a circle around the static target;

[0021] S512: Obtain the two intersection points of the drawn circle and the driving route of the autonomous vehicle, set the two intersection points of the drawn circle and the driving route of the autonomous vehicle as obstacle avoidance points, and set the arc between the two obstacle avoidance points as the obstacle avoidance route.

[0022] S513: Combine the obstacle avoidance route with the driving route of the autonomous vehicle, and replace the initial route between obstacle avoidance points on the driving route with the obstacle avoidance route to obtain a dynamically controlled route.

[0023] Preferably, S52 includes S521: when generating a unidirectional signal, determining whether the dynamic target is located in the direction of travel of the autonomous vehicle; if yes, generating a feedback command; if no, generating a normal signal.

[0024] When a feedback command is generated, the distance between the autonomous vehicle and the dynamic target is obtained and set as the following distance. The system then determines whether the following distance exceeds the preset following distance. If it does, a driving signal is generated; otherwise, a following control signal is generated.

[0025] Preferably, S52 includes S522: when a non-co-directional signal is generated, the driving routes of the dynamic target and the autonomous vehicle are obtained, the minimum straight-line distance between the dynamic target and the autonomous vehicle is obtained based on the driving routes, and the minimum straight-line distance is judged to see if it is greater than a preset safe distance. If it is, a pass signal is generated, and the autonomous vehicle drives normally through the dynamic target according to the driving route. If not, an avoidance signal is generated.

[0026] Preferably, the process of S6 is as follows: obtain the historical recognition and positioning information of the three-dimensional target recognition and positioning model, including the total number of recognitions and the recognition response time;

[0027] The response duration feature curve is constructed based on time series. The maximum peak value and minimum trough value are obtained from the response duration feature curve. The response interval is constructed based on the maximum peak value and minimum trough value. The ratio between the non-overlapping length of the response interval and the preset response interval and the length of the response interval is obtained and set as the response performance index.

[0028] Based on historical identification and positioning information, the percentage of the total number of identifications in which the identified target came into contact with the autonomous vehicle or the number of times the autonomous vehicle normally performed preset operations after the target was identified is obtained, and this percentage is set as the execution plan fit.

[0029] The preset weight factor coefficients of the response performance index and the execution plan fit are obtained, and the sum of the response performance index and the execution plan fit multiplied by the corresponding preset weight factor coefficients is set as the target recognition and positioning score. The target recognition and positioning score is then processed to obtain the autonomous update signal or data collection signal.

[0030] The beneficial effects of this invention are as follows:

[0031] (1) This invention integrates multimodal data from multiple sensors such as lidar, camera, and millimeter-wave radar, making full use of the advantages of different sensors, achieving information complementarity, improving the accuracy of target recognition and positioning, and simultaneously outputting information such as target category, location, and size through target detection and three-dimensional coordinate regression, meeting the needs of autonomous vehicles for perceiving the surrounding environment, and providing reliable protection for the safe driving of autonomous vehicles.

[0032] (2) This invention is accompanied by the autonomous vehicle classifying the target category and the autonomous driving planning process after classification, so as to improve the driving safety and autonomous planning flexibility of the autonomous vehicle. At the same time, it conducts an in-depth comprehensive evaluation of the three-dimensional target recognition and positioning model of the autonomous vehicle from the dual dimensions of positioning performance and recognition and positioning performance, thereby improving the target recognition and positioning accuracy and autonomous learning management effect of the autonomous vehicle. Attached Figure Description

[0033] The invention will now be further described with reference to the accompanying drawings;

[0034] Figure 1 This is a reference diagram of the method of the present invention;

[0035] Figure 2 This is a reference diagram of method S5 of the present invention;

[0036] Figure 3 This is a partial analysis diagram of Embodiment 2 of the present invention. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments;

[0039] Example 1:

[0040] Please see Figures 1 to 3 As shown, this invention provides a multimodal 3D target recognition and localization method for autonomous driving, comprising the following steps:

[0041] S1: The process of multimodal data acquisition and vehicle coordinate system construction for autonomous vehicles;

[0042] S2: The process of preprocessing multimodal data and acquiring multimodal fusion features;

[0043] S3: Target recognition and positioning analysis process based on the combination of a 3D target recognition and positioning model and a vehicle coordinate system;

[0044] S4: The target classification process, which is to determine whether the category in the basic information is dynamic. If it is, it is determined to be a dynamic target; otherwise, it is determined to be a static target.

[0045] S5: The process of controlling and managing the driving route of autonomous vehicles based on the classification of targets;

[0046] S6: Autonomous planning, management and analysis process of a three-dimensional target recognition and positioning model based on a comprehensive evaluation of recognition and positioning performance and recognition and positioning characteristics;

[0047] S1: The process of multimodal data acquisition and vehicle coordinate system construction for autonomous vehicles, namely, establishing the vehicle coordinate system of autonomous vehicles, collecting point cloud data within a pre-set detection range through the lidar on the autonomous vehicle, collecting RGB image data within a pre-set detection range through the high-definition camera, and collecting radar echo data within a pre-set detection range through the millimeter-wave radar.

[0048] Point cloud data, RGB image data, and radar echo data are collectively referred to as multimodal data;

[0049] S2: The process of preprocessing multimodal data and acquiring multimodal fusion features, namely, preprocessing multimodal data, performing time calibration on the preprocessed multimodal data using linear interpolation methods, calibrating the multimodal data to the same time point and transforming it to the vehicle coordinate system to achieve spatial calibration;

[0050] In this embodiment of the invention, the point cloud data collected by the lidar is preprocessed: noise points in the point cloud data are removed by using a statistical filtering algorithm to calculate the average distance from each point to its neighboring points. Points with an average distance greater than a set threshold are regarded as noise points and removed. By removing noise points, the interference of noise on subsequent feature extraction and target recognition can be reduced.

[0051] The denoised point cloud data is downsampled using a voxel grid downsampling method, which divides the point cloud space into a voxel grid of a certain size. Each voxel grid retains a designated representative point. Downsampling can reduce the amount of point cloud data, improve processing efficiency, and preserve the overall structural features of the point cloud.

[0052] Preprocessing of image data acquired by camera: Distortion correction of image data is performed. Based on the camera's intrinsic parameter matrix and distortion coefficients, the coordinates of pixels in the image are corrected. Due to factors such as the manufacturing process of the camera lens, the acquired image may have distortion. Correction can improve the accuracy of the image.

[0053] The distortion-corrected image is scaled and cropped to meet the input requirements of the subsequent network. Unifying the image size facilitates network processing and feature extraction.

[0054] Preprocessing of radar echo data acquired by millimeter-wave radar: Clutter removal is performed on the radar echo data, and a constant false alarm rate detection algorithm is used to remove static clutter, which can interfere with the detection of dynamic targets. Removing static clutter can improve the effectiveness of radar data.

[0055] The radar data after clutter removal is transformed to convert the data from the radar coordinate system to the autonomous vehicle coordinate system so that it can be fused with data from other sensors.

[0056] Feature extraction is performed on the preprocessed multimodal data to obtain the multimodal fusion features of the multimodal data. The multimodal fusion features are obtained by fusing the local / global features of point cloud data, the semantic / texture features extracted from RGB image data, and the motion / distance features extracted from radar echo data.

[0057] The multimodal feature fusion network described above can fully leverage the complementarity of LiDAR, cameras, and millimeter-wave radar to output fused features that include spatial structure, semantic category, and motion state, providing high-precision feature support for subsequent 3D target recognition and localization.

[0058] Example 2:

[0059] In this embodiment of the invention, target recognition and localization are performed through a model. The three-dimensional target recognition and localization model is based on the Transformer architecture and uses a self-attention mechanism to globally model features, capture the spatial relationships and dependencies between different targets, and then outputs the three-dimensional coordinates, size, category and other information of the target through regression and classification operations, so as to achieve accurate recognition and localization of three-dimensional targets in the surrounding environment.

[0060] S3: The target recognition and positioning analysis process based on the combination of the three-dimensional target recognition and positioning model and the vehicle coordinate system, that is, the multi-modal fusion features are input into the retrieved three-dimensional target recognition and positioning model to obtain the basic information of each target output by the three-dimensional target recognition and positioning model. The basic information includes the vehicle's three-dimensional coordinates, size, category, etc.

[0061] S4: The target classification process, which is to determine whether the category in the basic information is dynamic. If it is, it is determined to be a dynamic target; otherwise, it is determined to be a static target.

[0062] S5: The process of controlling and managing the driving route of autonomous vehicles based on the classification of targets;

[0063] S5 includes two steps: S51 and S52.

[0064] S51: When the target is a static target, the driving route of the autonomous vehicle is obtained, and the collision between the driving route and the static target is determined. If so, a route adjustment signal is generated; otherwise, a pass signal is generated. When a pass signal is generated, the autonomous vehicle drives normally through the static target according to the driving route.

[0065] S511: When generating a route adjustment signal: Based on the driving route, obtain the safe distance of the autonomous vehicle passing through the static target, set the static target as the center and the safe distance as the radius, and draw a circle around the static target;

[0066] S512: Obtain the two intersection points of the drawn circle and the driving route of the autonomous vehicle, set the two intersection points of the drawn circle and the driving route of the autonomous vehicle as obstacle avoidance points, and set the arc between the two obstacle avoidance points as the obstacle avoidance route.

[0067] S513: Combine the obstacle avoidance route with the driving route of the autonomous vehicle, and replace the initial route between obstacle avoidance points on the driving route with the obstacle avoidance route to obtain the dynamic control route. Then the autonomous vehicle drives according to the dynamic control route.

[0068] S52: When the target is a dynamic target, the operation data of the dynamic target and the autonomous vehicle are obtained. The operation data includes the direction of operation, speed of operation, etc., and the dynamic target and the autonomous vehicle are judged to see if their directions of operation are the same. If they are, a same-direction signal is generated; otherwise, a different-direction signal is generated.

[0069] S521: When a unidirectional signal is generated, the system determines whether the dynamic target is located in the direction of travel of the autonomous vehicle. If so, a feedback instruction is generated; otherwise, a normal signal is generated, and the vehicle travels normally according to its route.

[0070] When a feedback command is generated, the distance between the autonomous vehicle and the dynamic target is obtained and set as the following distance. The system then determines whether the following distance exceeds the preset following distance. If it does, a driving signal is generated and the vehicle drives normally according to the autonomous vehicle's operating data. If not, a following control signal is generated and the preset warning operation corresponding to the following control signal is executed to adjust the autonomous vehicle and improve its driving safety.

[0071] S522: When a non-co-directional signal is generated, the driving routes of the dynamic target and the autonomous vehicle are obtained. Based on the driving routes, the minimum straight-line distance between the dynamic target and the autonomous vehicle is obtained. The minimum straight-line distance is judged to see if it is greater than the preset safe distance. If it is, a pass signal is generated, and the autonomous vehicle drives normally through the dynamic target according to the driving route. If not, an avoidance signal is generated, and the preset warning operation corresponding to the avoidance signal is executed to avoid the autonomous vehicle.

[0072] Example 3:

[0073] S6: The autonomous planning, management and analysis process of the three-dimensional target recognition and positioning model based on the comprehensive evaluation of recognition and positioning performance and recognition and positioning characteristics. That is, to obtain the historical recognition and positioning information of the three-dimensional target recognition and positioning model, including the total number of recognitions, recognition response time, etc.

[0074] The response duration feature curve is constructed based on time series. The maximum peak value and minimum trough value are obtained from the response duration feature curve. The response interval is constructed based on the maximum peak value and minimum trough value. The response interval is compared and analyzed with the preset response interval to obtain the ratio between the non-overlapping length of the response interval and the length of the response interval, and this ratio is set as the response performance index.

[0075] Based on historical identification and positioning information, the percentage of the total number of identifications in which the identified target came into contact with the autonomous vehicle or the number of times the autonomous vehicle normally performed preset operations after the target was identified is obtained, and this percentage is set as the execution plan fit.

[0076] The system obtains the preset weight factor coefficients of the response performance index and the execution plan fit, and sets the sum of the product of the response performance index and the execution plan fit with the corresponding preset weight factor coefficients as the target recognition and positioning score. The target recognition and positioning score is then processed for discrimination. If the target recognition and positioning score is greater than or equal to the preset target recognition and positioning score threshold, an autonomous update signal is generated. If the target recognition and positioning score is less than the preset target recognition and positioning score threshold, a data collection signal is generated. The preset warning operation corresponding to the autonomous update signal or the data collection signal is then executed immediately. The preset warning operation corresponding to the autonomous update signal is to optimize the data of the 3D target recognition and positioning model to improve the recognition and positioning effect of the 3D target recognition and positioning model. The preset warning operation corresponding to the data collection signal is to perform data collection operation during the operation of the autonomous vehicle to provide data support for subsequent route planning, 3D target recognition and positioning model, etc.

[0077] In summary, this invention integrates multimodal data from various sensors, including LiDAR, cameras, and millimeter-wave radar, fully leveraging the advantages of different sensors to achieve information complementarity and improve the accuracy of target recognition and localization. Simultaneously, through target detection and 3D coordinate regression, it can output information such as target category, location, and size, meeting the environmental perception needs of autonomous vehicles and providing reliable assurance for their safe operation. Furthermore, the invention facilitates the autonomous vehicle's classification of targets and subsequent autonomous driving planning, enhancing driving safety and autonomous planning flexibility. Finally, it provides an in-depth comprehensive evaluation of the autonomous vehicle's 3D target recognition and localization model from both localization performance and recognition / localization capabilities, thereby improving the accuracy of target recognition and localization and the effectiveness of autonomous learning management.

[0078] The threshold is set for comparative analysis of results to determine whether they are good or bad. The value of the threshold is determined by a combination of large-scale model analysis of sample data and human experience. It can also be adjusted appropriately based on seasonal or common-sense influencing factors.

[0079] The size of the coefficient is a specific value obtained by quantifying each parameter to facilitate subsequent comparison. The size of the coefficient depends on the amount of sample data and the corresponding operating coefficient initially set by those skilled in the art for each set of sample data; as long as it does not affect the proportional relationship between the parameter and the quantified value.

[0080] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal three-dimensional target recognition and localization method for autonomous driving, characterized in that, Includes the following steps: S1: The process of multimodal data acquisition and vehicle coordinate system construction for autonomous vehicles; S2: The process of preprocessing multimodal data and acquiring multimodal fusion features; S3: Target recognition and positioning analysis process based on the combination of a 3D target recognition and positioning model and a vehicle coordinate system; S4: The target classification process, which is to determine whether the category in the basic information is dynamic. If it is, it is determined to be a dynamic target; otherwise, it is determined to be a static target. S5: The process of controlling and managing the driving route of autonomous vehicles based on the classification of targets; S6: Autonomous planning, management and analysis process of a three-dimensional target recognition and positioning model based on a comprehensive evaluation of recognition and positioning performance and recognition and positioning characteristics; The S1 process is as follows: establish the vehicle coordinate system of the autonomous vehicle, collect point cloud data within a pre-set detection range through the lidar on the autonomous vehicle, collect RGB image data within a pre-set detection range through the high-definition camera, and collect radar echo data within a pre-set detection range through the millimeter-wave radar. Point cloud data, RGB image data, and radar echo data are collectively referred to as multimodal data; The S2 process is as follows: preprocess the multimodal data, perform time calibration on the preprocessed multimodal data using a linear interpolation method, calibrate the multimodal data to the same time point and transform it to the vehicle coordinate system to achieve spatial calibration; The S3 process is as follows: The multimodal fusion features are input into the retrieved 3D target recognition and localization model to obtain the basic information of each target output by the 3D target recognition and localization model. The basic information includes the vehicle's 3D coordinates, size, and category. The process of S5 is as follows: S5 includes two steps, S51 and S52. S51: When the target is a static target, the driving route of the autonomous vehicle is obtained, and the collision between the driving route and the static target is determined. If the collision occurs, a route adjustment signal is generated; otherwise, a pass signal is generated. S52: When the target is a dynamic target, obtain the operation data of the dynamic target and the autonomous vehicle. The operation data includes the running direction and running speed. Then, determine whether the running directions of the dynamic target and the autonomous vehicle are the same. If they are, generate a same-direction signal; otherwise, generate a different-direction signal. S51 includes S511: When generating a route adjustment signal: Based on the driving route, the safe distance of the autonomous vehicle passing through the static target is obtained, the static target is set as the center of the circle, and the safe distance is set as the radius, so as to draw a circle around the static target; S512: Obtain the two intersection points of the drawn circle and the driving route of the autonomous vehicle, set the two intersection points of the drawn circle and the driving route of the autonomous vehicle as obstacle avoidance points, and set the arc between the two obstacle avoidance points as the obstacle avoidance route. S513: Combine the obstacle avoidance route with the driving route of the autonomous vehicle, and replace the initial route between obstacle avoidance points on the driving route with the obstacle avoidance route to obtain a dynamically controlled route. S52 includes S521: When a unidirectional signal is generated, it determines whether the dynamic target is located in the direction of travel of the autonomous vehicle. If so, a feedback command is generated; otherwise, a normal signal is generated. When a feedback command is generated, the distance between the autonomous vehicle and the dynamic target is obtained and set as the following distance. The system then determines whether the following distance exceeds the preset following distance. If it does, a driving signal is generated; otherwise, a following control signal is generated. S52 includes S522: When a non-co-directional signal is generated, the driving routes of the dynamic target and the autonomous vehicle are obtained, the minimum straight-line distance between the dynamic target and the autonomous vehicle is obtained based on the driving routes, and the minimum straight-line distance is judged to see if it is greater than the preset safety distance. If it is, a pass signal is generated, and the autonomous vehicle drives normally through the dynamic target according to the driving route. If not, an avoidance signal is generated. The S6 process is as follows: Obtain the historical recognition and positioning information of the 3D target recognition and positioning model. The historical recognition and positioning information includes the total number of recognitions and the recognition response time. The response duration feature curve is constructed based on time series. The maximum peak value and minimum trough value are obtained from the response duration feature curve. The response interval is constructed based on the maximum peak value and minimum trough value. The ratio between the non-overlapping length of the response interval and the preset response interval and the length of the response interval is obtained and set as the response performance index. Based on historical identification and positioning information, the percentage of the total number of identifications in which the identified target came into contact with the autonomous vehicle or the number of times the autonomous vehicle normally performed preset operations after the target was identified is obtained, and this percentage is set as the execution plan fit. The preset weight factor coefficients of the response performance index and the execution plan fit are obtained, and the sum of the response performance index and the execution plan fit multiplied by the corresponding preset weight factor coefficients is set as the target recognition and positioning score. The target recognition and positioning score is then processed to obtain the autonomous update signal or data collection signal.

Images