Collision warning methods, devices and media based on fusion sensors

Abstract

This invention discloses a collision warning method, device, and medium based on fused sensors, relating to the field of data processing technology. The method includes: acquiring radar point cloud information in front of the vehicle using millimeter-wave radar and acquiring image information in front of the vehicle using a camera device; fusing the radar point cloud information and the image information in front of the vehicle using a three-dimensional environment analysis model to obtain a three-dimensional environment in front of the vehicle, wherein the three-dimensional environment in front of the vehicle includes obstacle distribution coordinates; traversing the obstacle distribution coordinates to perform a collision risk assessment and obtain a collision risk level; when the collision risk level is greater than or equal to a first risk level threshold, performing emergency braking; when the collision risk level is greater than a second risk level threshold, generating a collision warning signal to issue a warning, thereby achieving the technical effect of improving the timeliness of warnings by initially identifying the target contour through images and performing targeted analysis and modeling of part of the radar point cloud data, reducing the amount of computation.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a collision warning method, device and medium based on fusion sensors. Background Technology

[0002] Collision warning, as an important component of active safety technology, helps drivers perceive potential dangers in advance and take corresponding measures to avoid collisions. Sensor fusion technology plays a key role in this process.

[0003] Existing collision warning methods mainly employ two approaches when fusing and analyzing time-series data from multimodal sensors for early warning. One approach is to merge obstacle recognition results from radar point clouds and images to improve the comprehensiveness of recognition. The other approach is to model the three-dimensional static environment from the images and the dynamic objects from the radar. The drawback of the latter approach is that radar point cloud modeling involves a large amount of computation, which can easily affect the timeliness of early warning. Summary of the Invention

[0004] This invention provides a collision warning method, device, and medium based on fusion sensors, which enables the initial identification of target contours through images, and the targeted analysis and modeling of partial point cloud data by radar point cloud, thereby reducing the amount of computation and improving the timeliness of warnings.

[0005] In a first aspect, the present invention provides a collision warning method based on fused sensors, wherein the method includes:

[0006] The system collects radar point cloud information in front of the vehicle using millimeter-wave radar and image information in front of the vehicle using a camera device.

[0007] A three-dimensional environment analysis model is obtained, wherein the three-dimensional environment analysis model includes an image processing channel and a radar point cloud processing channel.

[0008] The image processing channel processes the image information in front of the vehicle to obtain obstacle contour recognition results. The radar point cloud processing channel then performs sparse analysis on the radar point cloud information in front of the vehicle based on the obstacle contour recognition results to obtain obstacle localization time sequence information.

[0009] Based on the obstacle contour recognition results and the obstacle positioning timing information, a three-dimensional environment in front of the vehicle is constructed, wherein the three-dimensional environment in front of the vehicle includes the obstacle distribution coordinates.

[0010] The collision risk assessment is performed by traversing the coordinates of the obstacle distribution to obtain the collision risk level.

[0011] Emergency braking is executed when the collision risk level is greater than or equal to the first risk level threshold.

[0012] When the collision risk level is greater than the second risk level threshold, a collision warning signal is generated to issue a warning.

[0013] Secondly, the present invention also provides a collision warning device based on fusion sensors, wherein the device includes:

[0014] The radar and camera data acquisition module is used to acquire radar point cloud information in front of the vehicle through millimeter-wave radar and to acquire image information in front of the vehicle through a camera device.

[0015] The analysis model acquisition module is used to obtain a three-dimensional environment analysis model, wherein the three-dimensional environment analysis model includes an image processing channel and a radar point cloud processing channel.

[0016] The analysis and processing module is used to process the image information in front of the vehicle through the image processing channel to obtain obstacle contour recognition results, and to perform sparse analysis on the radar point cloud information in front of the vehicle based on the obstacle contour recognition results through the radar point cloud processing channel to obtain obstacle localization time sequence information. The 3D environment fusion module is used to construct a 3D environment in front of the vehicle based on the obstacle contour recognition results and the obstacle localization time sequence information, wherein the 3D environment in front of the vehicle includes obstacle distribution coordinates.

[0017] The collision risk assessment module is used to traverse the coordinates of the obstacle distribution to assess the collision risk and obtain the collision risk level.

[0018] An emergency braking module is used to perform emergency braking when the collision risk level is greater than or equal to a first risk level threshold.

[0019] The collision warning module is used to generate a collision warning signal to issue a warning when the collision risk level is greater than the second risk level threshold.

[0020] Thirdly, the present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the collision warning method based on fused sensors provided by the present invention.

[0021] This invention discloses a collision warning method, device, and medium based on fusion sensors, comprising: acquiring point cloud information in front of the vehicle using millimeter-wave radar, and simultaneously acquiring image data in front of the vehicle using a camera device; fusing radar point cloud information and image information using a three-dimensional environment analysis model to generate a three-dimensional environment in front of the vehicle containing obstacle distribution coordinates; iterating through the obstacle distribution coordinates in the three-dimensional environment to perform collision risk assessment and determine the risk level; if the collision risk level reaches or exceeds a set first risk level threshold, triggering emergency braking; if the collision risk level is higher than a second risk level threshold, generating a collision warning signal and issuing a warning to the driver. The collision warning method, device, and medium based on fusion sensors disclosed in this invention solve the defects of large computational load in radar point cloud modeling, which easily affects the timeliness of warnings. It realizes the technical effect of improving the timeliness of warnings by initially identifying the target outline through images and then performing targeted analysis and modeling of part of the point cloud data by radar point cloud, thereby reducing the computational load. Attached Figure Description

[0022] Figure 1 This is a schematic flowchart of the collision warning method based on fusion sensors of the present invention;

[0023] Figure 2 This is a schematic diagram of the collision warning device based on fusion sensors according to the present invention.

[0024] Figure labeling: Radar and camera data acquisition module 11, analysis model acquisition module 12, analysis and processing module 13, 3D environment fusion module 14, collision risk assessment module 15, emergency braking module 16, collision warning module 17. Detailed Implementation

[0025] The technical solution provided in the embodiments of the present invention addresses the shortcomings of existing technologies, such as the large computational load of radar point cloud modeling, which easily affects the timeliness of early warning. The overall approach adopted is as follows:

[0026] First, radar point cloud information in front of the vehicle is acquired using millimeter-wave radar, while image information in front of the vehicle is acquired using a camera device. Then, the radar point cloud information and image information are fused using a three-dimensional environment analysis model to generate a three-dimensional environment in front of the vehicle, including the coordinates of obstacle distribution. Next, these obstacle distribution coordinates are traversed and a collision risk assessment is performed to obtain the collision risk level. If the collision risk level is greater than or equal to a preset first risk level threshold, emergency braking is triggered. If the collision risk level is greater than a second risk level threshold, a collision warning signal is generated and sent to provide a warning.

[0027] The above technical solutions will now be described in detail with reference to the accompanying drawings and specific embodiments to provide a better understanding of them. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be understood that the present invention is not limited to the exemplary embodiments used only to explain the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. Furthermore, it should be noted that, for ease of description, only the parts related to the present invention are shown in the drawings, not all of them.

[0028] Example 1

[0029] Figure 1 This is a flowchart illustrating the collision warning method based on fused sensors according to the present invention, wherein the method includes:

[0030] The system collects radar point cloud information in front of the vehicle using millimeter-wave radar and image information in front of the vehicle using a camera device.

[0031] Specifically, the data is first collected by sensors deployed in front of or to the front side of the target vehicle. The data collection results include radar point cloud information of the front of the vehicle obtained by millimeter-wave radar and image information of the front of the vehicle obtained by camera device.

[0032] Optionally, the millimeter-wave radar is installed in front of or to the front side of the target vehicle, and its field of view should cover the road area in front of the vehicle to ensure comprehensive perception of the target area. By emitting high-frequency electromagnetic waves, receiving reflected signals and processing data, the generated radar point cloud information in front of the vehicle includes the three-dimensional coordinates of each reflection point, reflection intensity information, timestamps, etc.

[0033] Optionally, the camera device is also installed in front of or to the front side of the vehicle, and overlaps with the field of view of the millimeter-wave radar to facilitate subsequent data fusion; the camera device includes a monocular camera, a binocular camera, or a wide-angle camera, etc.; the camera device acquires image information in front of the vehicle at a preset frame rate and resolution, including color information, scene texture information, etc.

[0034] Optionally, time synchronization can be achieved through timestamp marking to ensure that radar point cloud data and image data are aligned within the same time frame.

[0035] The above-described method involves synchronous coverage acquisition using millimeter-wave radar and camera devices to obtain multi-dimensional data of the same target area in front of the vehicle, which serves as the basis for subsequent data fusion.

[0036] A three-dimensional environment analysis model is obtained, wherein the three-dimensional environment analysis model includes an image processing channel and a radar point cloud processing channel.

[0037] Specifically, the 3D environment analysis model includes an image processing channel for processing image data and a radar point cloud processing channel for processing radar point cloud data. The image processing channel is used to obtain detailed information about obstacles, such as contours and textures, based on the image data. The radar point cloud processing channel analyzes the radar point cloud data to obtain high-sampling-rate obstacle location information. Specifically, the radar point cloud information and the image information in front of the vehicle provide distance information (point cloud) and texture information within a predetermined viewing angle range in front of the target vehicle, respectively. By fusing the precise depth information of the radar point cloud with the rich visual features of the image through the 3D environment analysis model, more comprehensive and reliable data is provided for obstacle recognition and analysis, thereby improving recognition accuracy.

[0038] The image processing channel processes the image information in front of the vehicle to obtain obstacle contour recognition results. The radar point cloud processing channel then performs sparse analysis on the radar point cloud information in front of the vehicle based on the obstacle contour recognition results to obtain obstacle localization time sequence information.

[0039] In some embodiments, the image processing channel processes the image information in front of the vehicle to obtain an obstacle contour recognition result, and the radar point cloud processing channel performs sparse analysis on the radar point cloud information in front of the vehicle based on the obstacle contour recognition result to obtain obstacle localization time sequence information, including:

[0040] Based on the first step length, the radar point cloud information in front of the vehicle is segmented to obtain the temporal information of the radar point cloud in front of the vehicle. Based on the second step length, the image information in front of the vehicle is segmented to obtain the temporal information of the image in front of the vehicle, wherein the second step length is greater than the first step length. The image information in front of the vehicle is input into the image processing channel to obtain the obstacle contour recognition result. Based on the obstacle contour recognition result, the radar point cloud processing channel performs sparse analysis on the temporal information of the radar point cloud in front of the vehicle to obtain the obstacle localization temporal information.

[0041] Specifically, firstly, the first step length is configured according to the preset time interval, and the point cloud data is sliced ​​based on the first step length. The continuous point cloud is divided into multiple time segments, generating multiple point cloud slices. Each point cloud slice contains obstacle spatial information within a specific time period, forming a point cloud time sequence.

[0042] Specifically, a second step length greater than the first step length is set, and consecutive frame images are grouped and sliced ​​at time intervals of the second step length to generate image time series segments. In other words, compared with point cloud segmentation, image segmentation has a longer time span to reduce the frequency of image processing and reduce computational costs. Each segment of the generated image contains scene images within a specific time period, forming an image time series.

[0043] Specifically, the three-dimensional environment analysis model includes parallel image processing channels and radar point cloud processing channels, which are used to process the aforementioned vehicle front image timing information and vehicle front radar point cloud timing information, respectively.

[0044] For example, the image processing channel is used to perform image preprocessing, feature extraction and target inspection, and temporal analysis on the temporal information of the image in front of the vehicle, thereby obtaining feature data such as the category, texture, lighting status, and motion status of the obstacles in the image information in front of the vehicle.

[0045] For example, the radar point cloud processing channel is used to perform image preprocessing, feature extraction and target inspection, and time-series analysis on the temporal information of the image in front of the vehicle, thereby identifying the corresponding obstacles from the image information in front of the vehicle and obtaining feature data such as the two-dimensional coordinates, size, and motion state of the obstacles.

[0046] By combining the first and second step lengths to process point cloud and image data respectively, the accuracy of high-frequency information is preserved while the computational efficiency of low-frequency information is taken into account, which helps to achieve accurate modeling and perception of the three-dimensional environment in front of the vehicle.

[0047] Specifically, firstly, the temporal information of the image in front of the vehicle is input to the image processing channel, and enhancement operations such as noise reduction and deblurring are performed on the input image to improve the visibility of key features. Then, a convolutional neural network (such as YOLO or Fast R-CNN) is used to perform object detection and instance segmentation on each frame of the image, extracting the two-dimensional contour (such as bounding box, contour polygon), surface texture and object category (such as vehicle, pedestrian, etc.) of each obstacle in front of the vehicle, and generating obstacle contour recognition results.

[0048] For example, the timing information of the radar point cloud in front of the vehicle is input to the radar point cloud processing channel for sparse analysis. First, the obstacle contour recognition result output by the image processing channel is mapped to the spatial range of the radar point cloud. Based on the mapping result, a subset of the point cloud falling within the obstacle area is extracted and sparse analysis is performed to generate multiple obstacle point cloud subsets. Then, the geometric center of each obstacle point cloud subset is calculated to determine the three-dimensional position and size of the obstacle. At the same time, based on the determined temporal relationship of the obstacle point cloud subsets, the dynamic trajectory, speed, and direction of the obstacle are calculated. The acquired three-dimensional position, size, dynamic trajectory, speed, and direction are output as the aforementioned obstacle localization timing information.

[0049] Specifically, obstacle contour recognition results provide information such as obstacle category, texture status (e.g., lighting status), 2D contour, and 2D motion trajectory. Obstacle localization timing information provides dynamic information such as obstacle 3D position, size, and motion trajectory. This helps to match the obstacle category and contour in the image with the 3D coordinates in the point cloud by fusing the image and point cloud processing results, generating a 3D environment in front of the vehicle that includes obstacle distribution information (category, 3D position, size, and dynamic status).

[0050] The above method fully leverages the advantages of both sensors through the synergistic effect of the image processing channel and the radar point cloud processing channel, which helps to achieve high-precision 3D reconstruction of the environment in front of the vehicle and provides reliable environmental perception data for the vehicle's path planning and control decisions.

[0051] Furthermore, based on the obstacle contour recognition result, the radar point cloud processing channel performs sparse analysis on the temporal information of the radar point cloud in front of the vehicle to obtain the obstacle localization temporal information, including:

[0052] Based on the obstacle contour recognition result, a first obstacle contour is extracted; based on the time-series information of the radar point cloud in front of the vehicle, the radar point cloud distribution information at a first moment is extracted; based on the first obstacle contour, sparse analysis is performed on the radar point cloud distribution information at the first moment to obtain the obstacle contour matching point cloud at the first moment; until based on the first obstacle contour, sparse analysis is performed on the radar point cloud distribution information at the Nth moment of the time-series information of the radar point cloud in front of the vehicle to obtain the obstacle contour matching point cloud at the Nth moment; based on the obstacle contour matching point cloud at the first moment up to the obstacle contour matching point cloud at the Nth moment, the obstacle localization time-series information is constructed.

[0053] Specifically, firstly, a first obstacle contour is determined by randomly selecting from the obstacle contour recognition results obtained from the image processing channel. This first obstacle contour is the boundary information in the two-dimensional image, such as a rectangular box or a polygonal contour. Then, based on the temporal characteristics (such as timestamps) of the first obstacle contour, the radar point cloud distribution information with consistent temporal characteristics at the first moment is extracted from the radar point cloud temporal information in front of the vehicle.

[0054] Preferably, the radar point cloud distribution information at the first moment is extracted from the radar point cloud time series information in front of the vehicle, with temporal features as the first extraction constraint and the coarse registration area obtained by mapping the first obstacle contour to the point cloud coordinate system using a projection algorithm or geometric transformation as the second extraction constraint.

[0055] Specifically, based on the obtained first obstacle contour and the radar point cloud distribution information at the first moment, sparse analysis is performed. That is, the first obstacle contour is used as a constraint condition, and only the point cloud falling within the spatial range corresponding to the contour is retained. At the same time, noise points are removed, and a stable and high-confidence subset of the point cloud is extracted and denoted as the obstacle contour matching point cloud at the first moment. The obstacle contour matching point cloud at the first moment can accurately obtain the three-dimensional coordinates and density distribution information of the obstacle.

[0056] Specifically, using the same method and process as described above for obtaining the obstacle contour matching point cloud at the first moment, iterative sparse analysis is performed on the radar point cloud distribution information at N moments of the vehicle's front radar point cloud time sequence information to extract the obstacle contour matching point cloud at each moment, thus obtaining all N moments of obstacle contour matching point clouds.

[0057] Specifically, based on the temporal relationship of the obstacle contour matching point cloud from the first moment to the Nth moment, the three-dimensional motion trajectory of the obstacle is calculated and obtained to reflect the relative position change, relative speed information, and rotation or other behavioral characteristics of the obstacle and the target vehicle.

[0058] The above process, by combining image recognition results with radar point cloud time-series data, achieves high-precision positioning and dynamic tracking of obstacles. The sparse analysis of the point cloud helps to effectively reduce noise interference and improve the efficiency of point cloud computing.

[0059] Furthermore, based on the first obstacle contour, sparse analysis is performed on the radar point cloud distribution information at the first moment to obtain the obstacle contour matching point cloud at the first moment, including:

[0060] Based on the first obstacle contour, the radar point cloud distribution information at the first moment is randomly distributed to obtain a first point cloud distribution region; the in-contour neighborhood density and out-of-contour neighborhood density of selected point clouds belonging to the first obstacle contour and whose distance from the first obstacle contour is less than or equal to a distance threshold are calculated; the ratio of the in-contour neighborhood density to the out-of-contour neighborhood density is calculated and set as the point cloud matching degree; the proportion of point clouds whose point cloud matching degree is greater than or equal to the point cloud matching degree threshold is counted and set as the first point cloud distribution region matching degree, and added to the point cloud distribution region matching degree set; the first point cloud distribution region is updated until the historical area of ​​the first obstacle contour distribution exceeds the preset area proportion of the radar point cloud distribution information at the first moment, and the point cloud distribution region with the maximum value in the point cloud distribution region matching degree set is extracted and set as the obstacle contour matching point cloud at the first moment.

[0061] Specifically, sparsity analysis is performed. First, based on the range of the first obstacle contour, a random distribution is generated from the point cloud distribution information at the first moment. That is, the first obstacle contour is randomly mapped to the radar point cloud distribution information at the first moment, and the obtained random distribution area is the first point cloud distribution area.

[0062] Optionally, the target vehicle is used as the pole of the polar coordinate system, and the first obstacle contour is mapped to the radar point cloud distribution information at the first moment in polar coordinate form. The polar radius and polar angle of the polar coordinates are generated by a random number generator to achieve random selection of the mapping point. In other words, the position determined by the polar coordinates is the coordinate of the center point of the first obstacle contour after mapping.

[0063] Specifically, points in the point cloud whose distance from the first obstacle contour is less than or equal to a threshold are selected, and their distribution density (the number of points per unit volume) is calculated to obtain the neighborhood density within the contour; points in the point cloud whose distance from the first obstacle contour is greater than a threshold are selected, and their distribution density is calculated as the neighborhood density outside the contour; then, the ratio of the neighborhood density within the contour to the neighborhood density outside the contour is calculated and recorded as the matching degree of the first point cloud distribution region.

[0064] Specifically, it is determined whether the matching degree of the selected point cloud is greater than or equal to the point cloud matching degree threshold, and the matching degree value of the cloud distribution area that meets the threshold is stored in the point cloud distribution area matching degree set. Then, a new point cloud distribution area is randomly generated, and the above steps are repeated to calculate the new point cloud distribution area matching degree value. The historical area of ​​the analyzed obstacle contour distribution area is accumulated until its area of ​​the radar point cloud distribution information at the first moment reaches or exceeds a preset proportion. That is, the radar point cloud distribution information at the first moment is randomly distributed to search for a preset proportion of point clouds. Then, from the point cloud distribution area matching degree set, the point cloud distribution area corresponding to the maximum matching degree is found as the obstacle contour matching point cloud at the first moment, which is used for subsequent obstacle localization time sequence information construction.

[0065] By performing sparse analysis and dynamic region updates through the above steps, point clouds that highly match the obstacle contours can be accurately extracted, ensuring high-precision identification of obstacle contours and distribution.

[0066] The collision risk assessment is performed by traversing the coordinates of the obstacle distribution to obtain the collision risk level.

[0067] In some embodiments, a collision risk assessment is performed by traversing the obstacle distribution coordinates to obtain a collision risk level, including:

[0068] Based on the obstacle distribution coordinates, extract the time series information of the first obstacle distribution coordinates; obtain the vehicle's moving speed, real-time vehicle position, and preset road trajectory; obtain the future time zone coordinate time series information of the first obstacle by performing movement prediction on the time series information of the first obstacle distribution coordinates; when the time series information of the first obstacle's future time zone coordinates intersects with the preset road trajectory, obtain the intersection prediction time and intersection prediction position; based on the vehicle's moving speed and real-time vehicle position, predict the vehicle position at the intersection prediction time to obtain the predicted vehicle position at the intersection time; calculate the difference between the predicted vehicle position at the intersection time and the predicted intersection position, and set it as the first risk assessment coefficient; calculate the difference between the predicted intersection time and the current time, and set it as the second risk assessment coefficient; perform a collision risk assessment based on the first risk assessment coefficient and the second risk assessment coefficient to obtain the collision risk level.

[0069] Specifically, firstly, based on the obstacle distribution coordinates in the three-dimensional environment in front of the vehicle, the first obstacle distribution coordinate time sequence information is obtained. Preferably, the first obstacle distribution coordinate time sequence information corresponds to the obstacle closest to the target vehicle. Then, the current vehicle speed of the target vehicle is obtained through a vehicle speed sensor or navigation system, and the vehicle's coordinate position is obtained in real time through a GPS positioning system or inertial navigation system as the vehicle's real-time position. Based on the geographical information of the target road and the vehicle's planned path, the vehicle's preset road trajectory is obtained.

[0070] Specifically, an object tracking algorithm is used to predict the future time zone of the coordinates of the first obstacle, thereby obtaining the time series information of the obstacle's future time zone coordinates. The prediction methods include kinematic model-based prediction, trajectory fitting-based prediction (such as polynomial fitting and spline interpolation), machine learning-based prediction, and probability model-based prediction. The future time zone refers to a period of time starting from the current moment.

[0071] Specifically, the system compares the future time zone coordinates of the first obstacle with the preset road trajectory to determine if there is an intersection. If an intersection exists, the predicted time and position of the intersection are recorded. Next, based on the current vehicle speed and position, the vehicle's position at the predicted intersection time is predicted using a linear motion model. Then, the difference between the predicted vehicle position and the predicted intersection position is calculated as the first risk assessment coefficient, which reflects the potential collision risk between the vehicle and the obstacle. Next, the time difference between the predicted intersection time and the current time is calculated as the second risk assessment coefficient, which represents the time margin for predicting a collision; a smaller coefficient indicates a higher urgency. Finally, a comprehensive assessment is performed based on the first and second risk assessment coefficients to determine the collision risk level, which can be categorized as: low risk (low distance, long time), medium risk (medium distance, short time), and high risk (small distance, short time).

[0072] Optionally, a weighted average, linear regression, or other algorithm can be used to calculate the collision risk level based on the first risk assessment coefficient and the second risk assessment coefficient.

[0073] The above-described method combines vehicle position, obstacle coordinates, and motion information to predict potential collision risks and conduct quantitative assessments, providing a basis for decision-making to take appropriate actions.

[0074] In some implementations, a collision risk assessment is performed based on the first risk assessment coefficient and the second risk assessment coefficient to obtain the collision risk level, including:

[0075] Obtain a collision risk level identification table, wherein the collision risk level identification table is a pre-set data table that identifies risk levels; input the first risk assessment coefficient and the second risk assessment coefficient into the collision risk level identification table to obtain the collision risk level.

[0076] Specifically, the collision risk level identification table is a pre-defined mapping table used to identify the risk level of a collision based on a given risk assessment coefficient (first risk assessment coefficient and second risk assessment coefficient).

[0077] For example, the collision risk level identification table includes multiple sets of associated first risk assessment coefficient intervals and second risk assessment coefficient intervals, and each set of associated first risk assessment coefficient intervals and second risk assessment coefficient intervals corresponds to a risk level. Among them, multiple first risk assessment coefficient intervals in multiple sets define different intervals of the first risk assessment coefficient (e.g., 2~5 meters, 5~10 meters, etc.); multiple second risk assessment coefficient intervals in multiple sets define different intervals of the second risk assessment coefficient (e.g., 2~5 seconds, 5~10 seconds, etc.).

[0078] Specifically, based on the first risk assessment coefficient and the second risk assessment coefficient, the corresponding interval in the collision risk level label table is searched. For example, if the first risk assessment coefficient is 3 meters and the second risk assessment coefficient is 4 seconds, the collision risk level label table is searched to determine the rows where the first risk assessment coefficient is in the range of 2 to 5 meters and the second risk assessment coefficient is in the range of 2 to 5 seconds, and the corresponding medium risk label is extracted as the collision risk level.

[0079] By inputting the first risk assessment coefficient and the second risk assessment coefficient and matching them with a pre-set risk level label table, the collision risk level is finally output. This can quantify and classify the collision risks faced by the vehicle, so that further avoidance measures can be taken (such as adjusting speed, activating automatic braking, etc.).

[0080] Emergency braking is executed when the collision risk level is greater than or equal to the first risk level threshold.

[0081] Specifically, if the collision risk level is greater than or equal to the first risk level threshold, it can be considered that the current collision prediction result indicates that the target vehicle has a high probability of collision. In this case, emergency braking will be automatically initiated to slow down the vehicle speed or bring it to a complete stop before the collision occurs, so as to reduce the possibility of a collision or mitigate the consequences of a collision.

[0082] When the collision risk level is greater than the second risk level threshold, a collision warning signal is generated to issue a warning.

[0083] Specifically, if the collision risk level is greater than the second risk level threshold, it means that although the target vehicle faces a certain risk, it is still within a controllable range. At this time, a warning signal is generated and sent to the driver through the vehicle's sound and vision (such as the in-vehicle screen or external lights), reminding the driver to pay attention to the obstacle in front and that evasive measures may be necessary, thereby improving reaction time and reducing the probability of accidents.

[0084] Optionally, the relationship between the first risk level threshold and the second risk level threshold is determined based on the collision warning requirements of the target vehicle, and the difference between the first risk level threshold and the second risk level threshold can also be adjusted according to requirements. For example, if the first risk level threshold is greater than the second risk level threshold, a warning signal is triggered simultaneously when emergency braking is performed when the collision risk level is greater than or equal to the first risk level threshold. If the first risk level threshold is equal to the second risk level threshold, emergency braking and a warning signal are only executed directly when the collision risk is high. Through the above-mentioned diversified configuration methods of the first risk level threshold and the second risk level threshold, a flexible adjustment space is provided for the collision warning of the target vehicle, which helps to achieve adaptive adjustment of the collision warning function and the collision warning sensitivity.

[0085] In summary, the collision warning method based on fusion sensors provided by this invention has the following technical effects:

[0086] This invention utilizes millimeter-wave radar to collect point cloud information in front of the vehicle, while simultaneously acquiring image data of the area in front of the vehicle via camera equipment. A three-dimensional environment analysis model is used to fuse the radar point cloud information and image data, generating a three-dimensional environment in front of the vehicle that includes obstacle distribution coordinates. The obstacle distribution coordinates in the three-dimensional environment are iterated one by one to assess collision risk and determine the risk level. If the collision risk level reaches or exceeds a set first risk level threshold, emergency braking is triggered. If the collision risk level is lower than a second risk level threshold, a collision warning signal is generated and a warning is issued to the driver. This invention, based on a fusion sensor-based collision warning method, device, and medium, solves the problem of high computational load in radar point cloud modeling, which easily affects the timeliness of warnings. It achieves the technical effect of improving the timeliness of warnings by initially identifying the target outline through images and then selectively performing partial point cloud data analysis and modeling from the radar point cloud, reducing computational load.

[0087] Example 2

[0088] Figure 2 This is a schematic diagram of the collision warning device based on fusion sensors according to the present invention. For example, Figure 1 The flowchart of the collision warning method based on fusion sensors of the present invention can be illustrated as follows: Figure 2 The structure shown is implemented.

[0089] Based on the same concept as the collision warning method based on fusion sensors in the above embodiments, the present invention also provides a collision warning device based on fusion sensors, comprising:

[0090] The radar and camera data acquisition module 11 is used to acquire radar point cloud information in front of the vehicle through millimeter-wave radar and to acquire image information in front of the vehicle through a camera device.

[0091] The analysis model acquisition module 12 is used to obtain a three-dimensional environment analysis model, wherein the three-dimensional environment analysis model includes an image processing channel and a radar point cloud processing channel.

[0092] The analysis and processing module 13 is used to process the image information in front of the vehicle through the image processing channel to obtain the obstacle contour recognition result, and to perform sparse analysis on the radar point cloud information in front of the vehicle based on the obstacle contour recognition result through the radar point cloud processing channel to obtain the obstacle positioning time sequence information.

[0093] The three-dimensional environment fusion module 14 is used to construct a three-dimensional environment in front of the vehicle based on the obstacle contour recognition result and the obstacle positioning time sequence information, wherein the three-dimensional environment in front of the vehicle includes obstacle distribution coordinates.

[0094] The collision risk assessment module 15 is used to traverse the distribution coordinates of the obstacles to perform collision risk assessment and obtain the collision risk level.

[0095] Emergency braking module 16 is used to perform emergency braking when the collision risk level is greater than or equal to the first risk level threshold.

[0096] The collision warning module 17 is used to generate a collision warning signal to issue a warning when the collision risk level is greater than the second risk level threshold.

[0097] In some embodiments, the analysis and processing module 13 further includes:

[0098] The radar point cloud segmentation unit is used to segment the radar point cloud information in front of the vehicle according to the first step length to obtain the timing information of the radar point cloud in front of the vehicle.

[0099] The image information segmentation unit is used to segment the image information in front of the vehicle according to a second step length to obtain the temporal information of the image in front of the vehicle, wherein the second step length is greater than the first step length.

[0100] An obstacle contour recognition unit is used to input the image information of the front of the vehicle into the image processing channel to obtain the obstacle contour recognition result.

[0101] The sparse analysis unit is used to perform sparse analysis on the temporal information of the radar point cloud in front of the vehicle based on the obstacle contour recognition result through the radar point cloud processing channel to obtain the obstacle positioning temporal information.

[0102] Furthermore, the sparse analysis unit in the analysis and processing module 13 also includes:

[0103] The contour extraction unit is used to extract the first obstacle contour based on the obstacle contour recognition result.

[0104] The point cloud distribution information extraction unit is used to extract the radar point cloud distribution information at the first moment based on the time sequence information of the radar point cloud in front of the vehicle.

[0105] The matching point cloud analysis unit is used to perform sparse analysis on the radar point cloud distribution information at the first moment based on the first obstacle contour to obtain the obstacle contour matching point cloud at the first moment.

[0106] The temporal iterative analysis unit is used to perform sparse analysis on the radar point cloud distribution information at the Nth time based on the first obstacle contour, to obtain the obstacle contour matching point cloud at the Nth time.

[0107] The positioning timing information construction unit is used to construct the obstacle positioning timing information based on the obstacle contour matching point cloud from the first time moment to the obstacle contour matching point cloud at the Nth time moment.

[0108] Furthermore, the execution steps of the matching point cloud analysis unit in the analysis and processing module 13 include:

[0109] Based on the first obstacle contour, the radar point cloud distribution information at the first moment is randomly distributed to obtain a first point cloud distribution region; the in-contour neighborhood density and out-of-contour neighborhood density of selected point clouds belonging to the first obstacle contour and whose distance from the first obstacle contour is less than or equal to a distance threshold are calculated; the ratio of the in-contour neighborhood density to the out-of-contour neighborhood density is calculated and set as the point cloud matching degree; the proportion of point clouds whose point cloud matching degree is greater than or equal to the point cloud matching degree threshold is counted and set as the first point cloud distribution region matching degree, and added to the point cloud distribution region matching degree set; the first point cloud distribution region is updated until the historical area of ​​the first obstacle contour distribution exceeds the preset area proportion of the radar point cloud distribution information at the first moment, and the point cloud distribution region with the maximum value in the point cloud distribution region matching degree set is extracted and set as the obstacle contour matching point cloud at the first moment.

[0110] In some embodiments, the collision risk assessment module 15 includes:

[0111] The obstacle distribution coordinate temporal information extraction unit is used to extract the first obstacle distribution coordinate temporal information based on the obstacle distribution coordinates.

[0112] The vehicle status acquisition unit is used to obtain the vehicle's moving speed, real-time vehicle location, and preset road trajectory.

[0113] The obstacle movement prediction unit is used to predict the movement of the first obstacle by performing movement prediction on the time series information of the first obstacle's distribution coordinates, thereby obtaining the time series information of the future time zone coordinates of the first obstacle.

[0114] The intersection prediction analysis unit is used to obtain the intersection prediction time and intersection prediction position when the future time zone coordinate time information of the first obstacle intersects with the preset road trajectory.

[0115] The vehicle position prediction unit is used to predict the vehicle position at the intersection prediction time by combining the vehicle's moving speed and the vehicle's real-time position, and obtain the predicted vehicle position at the intersection time.

[0116] The first risk assessment unit is used to calculate the difference between the predicted vehicle position at the intersection time and the predicted intersection position, and is set as the first risk assessment coefficient.

[0117] The second risk assessment unit is used to calculate the difference between the intersection prediction time and the current time, and is set as the second risk assessment coefficient.

[0118] The risk level assessment unit is used to assess the collision risk based on the first risk assessment coefficient and the second risk assessment coefficient to obtain the collision risk level.

[0119] In some implementations, the execution steps of the risk level assessment unit in the collision risk assessment module 15 also include:

[0120] Obtain a collision risk level identification table, wherein the collision risk level identification table is a pre-set data table that identifies risk levels; input the first risk assessment coefficient and the second risk assessment coefficient into the collision risk level identification table to obtain the collision risk level.

[0121] Example 3

[0122] This application also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the collision warning method based on fusion sensors in Embodiment 1.

[0123] It should be understood that the focus of the embodiments mentioned in this specification is their difference from other embodiments. The specific embodiments in the aforementioned Embodiment 1 are also applicable to the collision warning device based on fusion sensors described in Embodiment 2. For the sake of brevity, they will not be further elaborated here.

[0124] It should be understood that the embodiments disclosed in this invention and the above description enable those skilled in the art to implement this invention. However, this invention is not limited to the embodiments mentioned above. It should be understood that those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this invention, and should all be included within the protection scope of this invention.

Claims

1. A collision warning method based on fusion sensors, characterized in that, include: The vehicle's front radar point cloud information is collected using millimeter-wave radar, and the vehicle's front image information is collected using a camera device. A three-dimensional environment analysis model is obtained, wherein the three-dimensional environment analysis model includes an image processing channel and a radar point cloud processing channel; The image processing channel processes the image information in front of the vehicle to obtain obstacle contour recognition results. The radar point cloud processing channel performs sparse analysis on the radar point cloud information in front of the vehicle based on the obstacle contour recognition results to obtain obstacle localization time sequence information. Based on the obstacle contour recognition results and the obstacle positioning timing information, a three-dimensional environment in front of the vehicle is constructed, wherein the three-dimensional environment in front of the vehicle includes obstacle distribution coordinates; The collision risk assessment is performed by traversing the coordinates of the obstacle distribution to obtain the collision risk level; When the collision risk level is greater than or equal to the first risk level threshold, emergency braking is performed. When the collision risk level is greater than the second risk level threshold, a collision warning signal is generated to issue a warning. Specifically, the image processing channel processes the image information in front of the vehicle to obtain obstacle contour recognition results, and the radar point cloud processing channel performs sparse analysis on the radar point cloud information in front of the vehicle based on the obstacle contour recognition results to obtain obstacle localization time sequence information, including: Based on the length of the first step, the radar point cloud information in front of the vehicle is segmented to obtain the timing information of the radar point cloud in front of the vehicle. Based on the second step length, the image information in front of the vehicle is segmented to obtain the temporal information of the image in front of the vehicle, wherein the second step length is greater than the first step length; The image information of the front of the vehicle is input into the image processing channel to obtain the obstacle contour recognition result; The obstacle localization timing information is obtained by performing sparse analysis on the timing information of the radar point cloud in front of the vehicle based on the obstacle contour recognition result through the radar point cloud processing channel.

2. The method as described in claim 1, characterized in that, Based on the obstacle contour recognition result, the radar point cloud processing channel performs sparse analysis on the temporal information of the radar point cloud in front of the vehicle to obtain the obstacle localization temporal information, including: Based on the obstacle contour recognition results, the first obstacle contour is extracted; Based on the time sequence information of the radar point cloud in front of the vehicle, extract the radar point cloud distribution information at the first moment; Based on the first obstacle contour, sparse analysis is performed on the radar point cloud distribution information at the first moment to obtain the obstacle contour matching point cloud at the first moment. Based on the first obstacle contour, sparse analysis is performed on the radar point cloud distribution information at the Nth time of the radar point cloud time sequence information in front of the vehicle to obtain the obstacle contour matching point cloud at the Nth time. The obstacle localization time sequence information is constructed based on the obstacle contour matching point cloud from the first time step to the obstacle contour matching point cloud at the Nth time step.

3. The method as described in claim 2, characterized in that, Based on the first obstacle contour, sparse analysis is performed on the radar point cloud distribution information at the first moment to obtain the obstacle contour matching point cloud at the first moment, including: Based on the first obstacle contour, the radar point cloud distribution information at the first moment is randomly distributed to obtain the first point cloud distribution area. Calculate the in-contour neighborhood density and out-of-contour neighborhood density of selected point clouds that belong to the first obstacle contour and are at a distance less than or equal to a distance threshold from the first obstacle contour. Calculate the ratio of the density of the neighborhood within the contour to the density of the neighborhood outside the contour, and set it as the point cloud matching degree; The percentage of point clouds whose point cloud matching degree is greater than or equal to the point cloud matching degree threshold is counted and set as the first point cloud distribution area matching degree, and added to the point cloud distribution area matching degree set. Update the first point cloud distribution area until the historical area of ​​the first obstacle contour distribution exceeds the preset area ratio of the radar point cloud distribution information at the first moment. Extract the point cloud distribution area with the maximum value in the point cloud distribution area matching degree set and set it as the obstacle contour matching point cloud at the first moment.

4. The method as described in claim 1, characterized in that, The collision risk assessment is performed by traversing the coordinates of the obstacle distribution to obtain the collision risk level, including: Based on the obstacle distribution coordinates, extract the temporal information of the first obstacle distribution coordinates; Obtain vehicle speed, real-time vehicle location, and preset road trajectory; By performing motion prediction on the temporal information of the distribution coordinates of the first obstacle, the future time zone coordinate temporal information of the first obstacle can be obtained; When the future time zone coordinates of the first obstacle intersect with the preset road trajectory, the intersection prediction time and intersection prediction location are obtained; The vehicle position at the intersection prediction time is obtained by combining the vehicle's moving speed and the vehicle's real-time position. Calculate the difference between the predicted vehicle position at the intersection time and the predicted position at the intersection, and set it as the first risk assessment coefficient; Calculate the difference between the intersection prediction time and the current time, and set it as the second risk assessment coefficient; The collision risk level is obtained by conducting a collision risk assessment based on the first risk assessment coefficient and the second risk assessment coefficient.

5. The method as described in claim 4, characterized in that, The collision risk level is obtained by performing a collision risk assessment based on the first risk assessment coefficient and the second risk assessment coefficient, including: Obtain a collision risk level identification table, wherein the collision risk level identification table is a data table that is pre-set to identify risk levels; The first risk assessment coefficient and the second risk assessment coefficient are input into the collision risk level identification table to obtain the collision risk level.

6. A collision warning device based on fusion sensors, characterized in that, The device is used to execute the collision warning method based on fused sensors as described in any one of claims 1-5, the device comprising: The radar and camera data acquisition module is used to acquire radar point cloud information in front of the vehicle through millimeter-wave radar and to acquire image information in front of the vehicle through camera devices. The analysis model acquisition module is used to obtain a three-dimensional environment analysis model, wherein the three-dimensional environment analysis model includes an image processing channel and a radar point cloud processing channel; The analysis and processing module is used to process the image information in front of the vehicle through the image processing channel to obtain the obstacle contour recognition result, and to perform sparse analysis on the radar point cloud information in front of the vehicle based on the obstacle contour recognition result through the radar point cloud processing channel to obtain the obstacle positioning time sequence information. A three-dimensional environment fusion module is used to construct a three-dimensional environment in front of the vehicle based on the obstacle contour recognition results and the obstacle positioning time sequence information, wherein the three-dimensional environment in front of the vehicle includes obstacle distribution coordinates; The collision risk assessment module is used to traverse the coordinates of the obstacle distribution to assess the collision risk and obtain the collision risk level. An emergency braking module is used to perform emergency braking when the collision risk level is greater than or equal to a first risk level threshold. The collision warning module is used to generate a collision warning signal to issue a warning when the collision risk level is greater than the second risk level threshold.

7. The device as described in claim 6, characterized in that, The analysis and processing module further includes: The radar point cloud segmentation unit is used to segment the radar point cloud information in front of the vehicle according to the first step length to obtain the timing information of the radar point cloud in front of the vehicle. The image information segmentation unit is used to segment the image information in front of the vehicle according to the second step length to obtain the temporal information of the image in front of the vehicle, wherein the second step length is greater than the first step length; An obstacle contour recognition unit is used to input the image information of the front of the vehicle into the image processing channel to obtain the obstacle contour recognition result. The sparse analysis unit is used to perform sparse analysis on the temporal information of the radar point cloud in front of the vehicle based on the obstacle contour recognition result through the radar point cloud processing channel to obtain the obstacle positioning temporal information.

8. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the collision warning method based on fused sensors as described in any one of claims 1 to 5.

Images