Collision early warning method and device based on fusion sensor and medium

Abstract

The invention discloses a collision early warning method and device based on a fusion sensor and a medium, and relates to the technical field of data processing.The method comprises the steps that radar point cloud information in front of a vehicle is collected through a millimeter-wave radar, and image information in front of the vehicle is collected through a camera device; the radar point cloud information in front of the vehicle and the image information in front of the vehicle are fused through a three-dimensional environment analysis model to obtain a three-dimensional environment in front of the vehicle, and the three-dimensional environment in front of the vehicle comprises obstacle distribution coordinates; traversing the obstacle distribution coordinates to perform collision risk assessment to obtain a collision risk level; when the collision risk level is larger than or equal to a first risk level threshold value, emergency braking is executed; and when the collision risk level is greater than a second risk level threshold, generating a collision early warning signal for early warning, thereby achieving the technical effects of initially identifying a target contour through the image, executing analysis modeling of part of point cloud data in a targeted manner through the radar point cloud, reducing the calculation amount and further improving the early warning timeliness.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a collision warning method, device and medium based on a fusion sensor. Background Art

[0002] As an important part of active safety technology, collision warning can help drivers perceive potential dangers in advance and take corresponding measures to avoid collision accidents, and sensor fusion technology plays a key role in this.

[0003] There are currently two main ways for existing collision warning methods to perform fusion processing and analysis of time series data from multimodal sensors. One is to merge the obstacle recognition results of radar point clouds and images to improve the comprehensiveness of recognition. The other is to model the three-dimensional static environment with images and model dynamic objects with radar. The disadvantage is that the radar point cloud modeling requires a large amount of calculation, which can easily affect the timeliness of the warning. Summary of the invention

[0004] The present invention provides a collision warning method, device and medium based on fusion sensor, so as to realize the initial recognition of target contour through image, and the radar point cloud performs the analysis and modeling of part of the point cloud data in a targeted manner, thereby reducing the amount of calculation, and thus achieving the technical effect of improving the timeliness of warning.

[0005] In a first aspect, the present invention provides a collision warning method based on a fusion sensor, wherein the method comprises: The radar point cloud information in front of the vehicle is collected through the millimeter-wave radar, and the image information in front of the vehicle is collected through the camera device.

[0006] A three-dimensional environment analysis model is obtained, wherein the three-dimensional environment analysis model includes an image processing channel and a radar point cloud processing channel.

[0007] The image information in front of the vehicle is processed through the image processing channel to obtain an obstacle contour recognition result, and the radar point cloud information in front of the vehicle is sparsely analyzed based on the obstacle contour recognition result through the radar point cloud processing channel to obtain obstacle positioning timing information.

[0008] A three-dimensional environment in front of the vehicle is constructed according to the obstacle contour recognition result and the obstacle positioning time sequence information, wherein the three-dimensional environment in front of the vehicle includes obstacle distribution coordinates.

[0009] The obstacle distribution coordinates are traversed to perform collision risk assessment to obtain a collision risk level.

[0010] When the collision risk level is greater than or equal to a first risk level threshold, emergency braking is performed.

[0011] When the collision risk level is greater than a second risk level threshold, a collision warning signal is generated for warning.

[0012] In a second aspect, the present invention further provides a collision warning device based on a fusion sensor, wherein the device comprises: The radar and camera data acquisition module is used to collect radar point cloud information in front of the vehicle through the millimeter wave radar and collect image information in front of the vehicle through the camera device.

[0013] The analysis model acquisition module is used to obtain a three-dimensional environment analysis model, wherein the three-dimensional environment analysis model includes an image processing channel and a radar point cloud processing channel.

[0014] The analysis and processing module is used to process the image information in front of the vehicle through the image processing channel to obtain the obstacle contour recognition result, and to perform sparse analysis on the radar point cloud information in front of the vehicle based on the obstacle contour recognition result through the radar point cloud processing channel to obtain obstacle positioning time sequence information. The three-dimensional environment fusion module is used to construct a three-dimensional environment in front of the vehicle according to the obstacle contour recognition result and the obstacle positioning time sequence information, wherein the three-dimensional environment in front of the vehicle includes the obstacle distribution coordinates.

[0015] The collision risk assessment module is used to traverse the obstacle distribution coordinates to perform collision risk assessment and obtain a collision risk level.

[0016] The emergency braking module is configured to perform emergency braking when the collision risk level is greater than or equal to a first risk level threshold.

[0017] The collision warning module is used to generate a collision warning signal for warning when the collision risk level is greater than a second risk level threshold.

[0018] In a third aspect, the present invention further provides a storage medium storing a computer program, which, when executed by a processor, implements the collision warning method based on fusion sensor provided by the present invention.

[0019] The present invention discloses a collision warning method, device and medium based on a fusion sensor, including: collecting point cloud information in front of a vehicle through a millimeter-wave radar, and obtaining image data in front of the vehicle through a camera device; using a three-dimensional environment analysis model to fuse radar point cloud information with image information to generate a three-dimensional environment in front of the vehicle including obstacle distribution coordinates; traversing the obstacle distribution coordinates in the three-dimensional environment one by one, performing collision risk assessment and determining the risk level; if the collision risk level reaches or exceeds a set first risk level threshold, emergency braking is triggered; if the collision risk level is higher than a second risk level threshold, a collision warning signal is generated, and a warning prompt is issued to the driver. The collision warning method, device and medium based on a fusion sensor disclosed by the present invention solve the defects of large amount of calculation in radar point cloud modeling, which easily affects the timeliness of warning, and realizes the initial recognition of target contours through images, and the radar point cloud performs targeted analysis and modeling of part of the point cloud data, thereby reducing the amount of calculation, thereby achieving the technical effect of improving the timeliness of warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the collision warning method based on fusion sensor of the present invention; Figure 2 It is a structural schematic diagram of a collision warning device based on a fusion sensor of the present invention.

[0021] Explanation of the reference numerals: radar and camera data acquisition module 11 , analysis model acquisition module 12 , analysis and processing module 13 , three-dimensional environment fusion module 14 , collision risk assessment module 15 , emergency braking module 16 , collision warning module 17 . DETAILED DESCRIPTION

[0022] The technical solution provided in the embodiments of the present invention is to solve the defects of the prior art that radar point cloud modeling has a large amount of calculation and is easy to affect the timeliness of early warning. The overall idea adopted is as follows: First, the radar point cloud information in front of the vehicle is collected by the millimeter wave radar, and the image information in front of the vehicle is collected by the camera device; then, the radar point cloud information and the image information are fused using the three-dimensional environment analysis model to generate a three-dimensional environment in front of the vehicle, including obstacle distribution coordinates; then, these obstacle distribution coordinates are traversed and a collision risk assessment is performed to obtain a collision risk level; if the collision risk level is greater than or equal to a preset first risk level threshold, emergency braking is triggered; if the collision risk level is greater than a second risk level threshold, a collision warning signal is generated and sent for warning.

[0023] The above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods of the specification to better understand the above technical solution. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments of the present invention. It should be understood that the present invention is not limited to the example embodiments used only to explain the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, it should be noted that, for the convenience of description, only the parts related to the present invention are shown in the drawings, rather than all of them.

[0024] Embodiment 1 Figure 1 The figure is a flow chart of a collision warning method based on a fusion sensor of the present invention, wherein the method comprises: The radar point cloud information in front of the vehicle is collected through the millimeter-wave radar, and the image information in front of the vehicle is collected through the camera device.

[0025] Specifically, first, sensing is performed by using sensors arranged in front of or on the front side of the target vehicle. The sensing results include radar point cloud information in front of the vehicle obtained by the millimeter-wave radar and image information in front of the vehicle obtained by the camera device.

[0026] Optionally, the millimeter-wave radar is installed in front of or to the front side of the target vehicle, and its field of view should cover the road in front of the vehicle to ensure comprehensive perception of the target area; by emitting high-frequency electromagnetic waves, receiving reflected signals and performing data processing, the radar point cloud information in front of the vehicle is generated, including the three-dimensional coordinates of each reflection point, reflection intensity information, timestamp, etc.

[0027] Optionally, a camera device is also installed in front of or on the front side of the vehicle, and maintains a certain overlap with the field of view of the millimeter-wave radar to facilitate subsequent data fusion; wherein the camera device includes a monocular camera, a binocular camera or a wide-angle camera, etc.; the camera device collects image information in front of the vehicle at a preset frame rate and resolution, including color information, scene texture information, etc.

[0028] Optionally, time synchronization is performed through timestamp marking to ensure that the radar point cloud data and the image data are aligned in the same time frame.

[0029] The above method steps, through the synchronous coverage collection of the millimeter wave radar and the camera device, can obtain multi-dimensional data of the same target area in front of the vehicle as the basis for subsequent data fusion.

[0030] A three-dimensional environment analysis model is obtained, wherein the three-dimensional environment analysis model includes an image processing channel and a radar point cloud processing channel.

[0031] Specifically, the three-dimensional environment analysis model includes an image processing channel for processing image data and a radar point cloud processing channel for processing radar point cloud data, wherein the image processing channel is used to obtain detailed information of obstacles, such as contours, textures, etc., based on image data, and the radar point cloud processing channel is used to analyze radar point cloud data to obtain obstacle orientation information with a high sampling rate. Specifically, the radar point cloud information in front of the vehicle and the image information in front of the vehicle respectively provide distance information (point cloud) and texture information within a predetermined viewing angle in front of the target vehicle. Through the three-dimensional environment analysis model, the precise depth information of the radar point cloud is integrated with the rich visual features of the image, which helps to provide more comprehensive and reliable data for obstacle recognition and analysis, thereby improving recognition accuracy.

[0032] The image information in front of the vehicle is processed through the image processing channel to obtain an obstacle contour recognition result, and the radar point cloud information in front of the vehicle is sparsely analyzed based on the obstacle contour recognition result through the radar point cloud processing channel to obtain obstacle positioning timing information.

[0033] In some embodiments, the image information in front of the vehicle is processed by the image processing channel to obtain an obstacle contour recognition result, and the radar point cloud information in front of the vehicle is sparsely analyzed based on the obstacle contour recognition result by the radar point cloud processing channel to obtain obstacle positioning time sequence information, including: According to the first step length, the radar point cloud information in front of the vehicle is segmented to obtain the time series information of the radar point cloud in front of the vehicle; according to the second step length, the image information in front of the vehicle is segmented to obtain the time series information of the image in front of the vehicle, wherein the second step length is greater than the first step length; the image information in front of the vehicle is input into the image processing channel to obtain the obstacle contour recognition result; the radar point cloud processing channel performs sparse analysis on the time series information of the radar point cloud in front of the vehicle based on the obstacle contour recognition result to obtain the obstacle positioning time series information. .

[0034] Specifically, first, the first step length is configured according to a preset time interval, and the point cloud data is sliced ​​based on the first step length to divide the continuous point cloud into multiple time segments to generate multiple point cloud slices. Each point cloud slice contains the spatial information of obstacles within a specific time period, forming a point cloud time sequence.

[0035] Specifically, a second step size is set that is larger than the first step size, and continuous frame images are grouped and sliced ​​with the second step size as the time interval to generate image time series segments. That is to say, compared with point cloud segmentation, the time span of image segmentation is longer to reduce the frequency of image processing and reduce the computational cost. Each image slice generated by segmentation contains the scene image within a specific time period, forming an image time series sequence.

[0036] Specifically, the three-dimensional environment analysis model includes an image processing channel and a radar point cloud processing channel that are arranged in parallel, which are used to process the above-mentioned vehicle front image time series information and vehicle front radar point cloud time series information respectively.

[0037] Exemplarily, the image processing channel is used to perform image preprocessing, feature extraction and target inspection, and timing analysis of the timing information of the image in front of the vehicle, so as to obtain characteristic data such as the category, texture, lighting status, motion status, etc. of the corresponding obstacle in the image information in front of the vehicle.

[0038] Exemplarily, the radar point cloud processing channel is used to perform image preprocessing, feature extraction and target inspection, and timing analysis of the time series information of the image in front of the vehicle, so as to identify the corresponding obstacle from the image information in front of the vehicle and obtain the obstacle's two-dimensional coordinates, size, motion state and other characteristic data.

[0039] Through the above steps, combined with the first step and the second step to process the point cloud and image data respectively, the accuracy of high-frequency information is retained while taking into account the calculation efficiency of low-frequency information, which helps to achieve accurate modeling and perception of the three-dimensional environment in front of the vehicle.

[0040] Specifically, first, the time series information of the image in front of the vehicle is input into the image processing channel, and the input image is enhanced by denoising, deblurring and other enhancement operations to improve the visibility of key features; then, a convolutional neural network (such as YOLO, Fast R-CNN) is used to perform target detection and instance segmentation on each frame of the image, extract the two-dimensional outline (such as bounding box, outline polygon), surface texture and object category (such as vehicle, pedestrian, etc.) of each obstacle in front of the vehicle, and generate the obstacle outline recognition result.

[0041] Exemplarily, the time series information of the radar point cloud in front of the vehicle is input into the radar point cloud processing channel for sparse analysis. First, the obstacle contour recognition result output by the image processing channel is mapped to the spatial range of the radar point cloud, and based on the mapping result, the point cloud subset falling within the obstacle area is extracted, and sparse analysis is performed to generate multiple obstacle point cloud subsets; then the geometric center of each obstacle point cloud subset is calculated to determine the three-dimensional position and size of the obstacle; at the same time, according to the determined time series relationship of the obstacle point cloud subsets, the dynamic trajectory, speed and direction of the obstacle are calculated; the obtained three-dimensional position, size, dynamic trajectory, speed and direction are output as the above-mentioned obstacle positioning time series information.

[0042] Specifically, the obstacle contour recognition results are used to provide information such as the obstacle category, texture status (such as lighting status), two-dimensional contour and two-dimensional motion trajectory, and the obstacle positioning timing information is used to provide dynamic information such as the obstacle's three-dimensional position, size, motion trajectory, etc., which helps to subsequently match the obstacle category and contour in the image with the three-dimensional coordinates in the point cloud through the fusion of image and point cloud processing results, and generate a three-dimensional environment in front of the vehicle that contains obstacle distribution information (category, three-dimensional position, size, dynamic status).

[0043] The above method fully utilizes the advantages of the two sensors through the synergy of the image processing channel and the radar point cloud processing channel, which helps to reconstruct the high-precision three-dimensional environment in front of the vehicle and provide reliable environmental perception data for the vehicle's path planning and control decisions.

[0044] Furthermore, the radar point cloud processing channel performs sparse analysis on the time series information of the radar point cloud in front of the vehicle based on the obstacle contour recognition result to obtain the obstacle positioning time series information, including: According to the obstacle contour recognition result, a first obstacle contour is extracted; according to the timing information of the radar point cloud in front of the vehicle, the radar point cloud distribution information at the first moment is extracted; based on the first obstacle contour, the radar point cloud distribution information at the first moment is sparsely analyzed to obtain the obstacle contour matching point cloud at the first moment; until based on the first obstacle contour, the radar point cloud distribution information at the Nth moment of the radar point cloud timing information in front of the vehicle is sparsely analyzed to obtain the obstacle contour matching point cloud at the Nth moment; according to the obstacle contour matching point cloud at the first moment until the obstacle contour matching point cloud at the Nth moment, the obstacle positioning timing information is constructed.

[0045] Specifically, first, a first obstacle contour is randomly selected from the obstacle contour recognition results obtained from the image processing channel to determine the first obstacle contour, which is the boundary information in the two-dimensional image, such as a rectangular frame or a polygonal contour; then, according to the temporal characteristics (such as timestamp) of the first obstacle contour, the radar point cloud distribution information at the first moment with consistent temporal characteristics is extracted from the radar point cloud temporal information in front of the vehicle.

[0046] Preferably, the time series feature is used as the first extraction constraint, and the coarse registration area obtained by mapping the first obstacle contour to the point cloud coordinate system using a projection algorithm or a geometric transformation is used as the second extraction constraint, and the radar point cloud distribution information at the first moment is extracted from the radar point cloud time series information in front of the vehicle.

[0047] Specifically, a sparse analysis is performed based on the acquired first obstacle contour and the radar point cloud distribution information at the first moment. That is, the first obstacle contour is used as a constraint condition, and only the point cloud falling within the spatial range corresponding to the contour is retained. At the same time, noise points are eliminated, and a stable and high-confidence point cloud subset is extracted and recorded as the obstacle contour matching point cloud at the first moment. The obstacle contour matching point cloud at the first moment can accurately obtain the three-dimensional coordinates and density distribution information of the obstacle.

[0048] Specifically, the same method and process as that for obtaining the obstacle contour matching point cloud at the first moment is adopted to perform iterative sparse analysis on the radar point cloud distribution information at N moments of the radar point cloud time series information in front of the vehicle, extract the obstacle contour matching point cloud at each moment, and obtain the obstacle contour matching point clouds at all N moments.

[0049] Specifically, according to the temporal relationship of the obstacle contour matching point cloud from the first moment to the Nth moment, the three-dimensional motion trajectory of the obstacle is calculated to reflect the relative position change between the obstacle and the target vehicle, the relative speed information, the rotation of the obstacle itself or other behavioral characteristics.

[0050] The above process achieves high-precision positioning and dynamic tracking of obstacles by combining image recognition results with radar point cloud time series data. The sparse analysis of point clouds helps to effectively reduce noise interference and improve the efficiency of point cloud computing.

[0051] Further, based on the first obstacle contour, sparse analysis is performed on the radar point cloud distribution information at the first moment to obtain the obstacle contour matching point cloud at the first moment, including: According to the first obstacle contour, the radar point cloud distribution information is randomly distributed at the first moment to obtain a first point cloud distribution area; the inner contour neighborhood density and the outer contour neighborhood density of the selected point cloud that is inside the first obstacle contour and has a distance less than or equal to a distance threshold from the first obstacle contour are calculated; the ratio of the inner contour neighborhood density to the outer contour neighborhood density is calculated and set as the point cloud matching degree; the proportion of the number of point clouds whose point cloud matching degrees of all selected point clouds are greater than or equal to the point cloud matching degree threshold is counted and set as the first point cloud distribution area matching degree, and added to the point cloud distribution area matching degree set; the first point cloud distribution area is updated until the first obstacle contour distribution history area exceeds the preset area proportion of the radar point cloud distribution information at the first moment, and the maximum value point cloud distribution area in the point cloud distribution area matching degree set is extracted and set as the obstacle contour matching point cloud at the first moment.

[0052] Specifically, a sparse analysis is performed. First, in the point cloud distribution information at the first moment, a random distribution is generated based on the range of the first obstacle contour, that is, the first obstacle contour is randomly mapped to the radar point cloud distribution information at the first moment, and the randomly distributed area obtained is the first point cloud distribution area.

[0053] Optionally, the target vehicle is taken as the pole of the polar coordinate system, and the first obstacle contour is mapped to the radar point cloud distribution information at the first moment in the form of polar coordinates, wherein the polar diameter and polar angle of the polar coordinates are generated by a random number generator to achieve random selection of mapping points. In other words, the position determined by the polar coordinates is the coordinate of the center point of the first obstacle contour after mapping.

[0054] Specifically, the points in the point cloud whose distance to the first obstacle contour is less than or equal to the threshold are screened out, and their distribution density (the number of points in a unit volume) is counted to obtain the neighborhood density inside the contour; the points in the point cloud whose distance to the first obstacle contour is greater than the threshold are screened out, and their distribution density is counted as the neighborhood density outside the contour; then, the ratio of the neighborhood density inside the contour to the neighborhood density outside the contour is calculated, which is recorded as the first point cloud distribution area matching degree of the first point cloud distribution area.

[0055] Specifically, it is determined whether the matching degree of the selected point cloud is greater than or equal to the point cloud matching degree threshold, and the matching degree value of the cloud distribution area that meets the requirements is stored in the point cloud distribution area matching degree set; then, a new point cloud distribution area is randomly generated, and the above steps are repeated to calculate the matching degree value of the new point cloud distribution area, and the historical area of ​​the obstacle contour distribution area that has been analyzed is accumulated until the area of ​​the radar point cloud distribution information at the first moment reaches or exceeds the preset proportion, that is, the radar point cloud distribution information at the first moment is randomly distributed to search for point clouds with a preset proportion; and then, from the point cloud distribution area matching degree set, the point cloud distribution area corresponding to the maximum matching degree is found as the obstacle contour matching point cloud at the first moment, which is used for the subsequent construction of obstacle positioning time series information.

[0056] Through the above steps of sparse analysis and dynamic area update, it is possible to accurately extract a point cloud that highly matches the obstacle contour, ensuring high-precision recognition of the obstacle contour and distribution.

[0057] The obstacle distribution coordinates are traversed to perform collision risk assessment to obtain a collision risk level.

[0058] In some embodiments, traversing the obstacle distribution coordinates to perform collision risk assessment and obtain a collision risk level includes: According to the obstacle distribution coordinates, extract the time series information of the first obstacle distribution coordinates; obtain the vehicle moving speed, the vehicle real-time position and the preset road trajectory; obtain the time series information of the first obstacle future time zone coordinates by performing movement prediction on the first obstacle distribution coordinates time series information; when the first obstacle future time zone coordinates time series information intersects with the preset road trajectory, obtain the intersection prediction time and the intersection prediction position; predict the vehicle position at the intersection prediction time based on the vehicle moving speed and the vehicle real-time position, and obtain the vehicle prediction position at the intersection time; calculate the difference between the vehicle prediction position at the intersection time and the intersection prediction position, and set it as a first risk assessment coefficient; calculate the difference between the intersection prediction time and the current time, and set it as a second risk assessment coefficient; perform collision risk assessment according to the first risk assessment coefficient and the second risk assessment coefficient to obtain the collision risk level.

[0059] Specifically, firstly, based on the obstacle distribution coordinates in the three-dimensional environment in front of the vehicle, first obstacle distribution coordinate time series information is obtained. Preferably, the first obstacle distribution coordinate time series information corresponds to the obstacle closest to the target vehicle. Then, the current vehicle moving speed of the target vehicle is obtained through a vehicle speed sensor or a navigation system, and the coordinate position of the vehicle is obtained in real time through a GPS positioning system or an inertial navigation system as the real-time position of the vehicle. According to the geographic information of the target road and the planned path of the vehicle, the preset road trajectory of the vehicle is obtained.

[0060] Specifically, an object tracking algorithm is used to predict the future time zone of the coordinates of the first obstacle to obtain the coordinate time series information of the obstacle in the future time zone, wherein the prediction methods include prediction based on kinematic models, prediction based on trajectory fitting (such as polynomial fitting, spline interpolation), prediction based on machine learning, prediction based on probability models, etc.; the future time zone refers to a period of time starting from the current moment.

[0061] Specifically, compare the future time zone coordinate time series information of the first obstacle with the preset road trajectory to determine whether there is an intersection. If there is an intersection, record the predicted intersection time and predicted intersection position; then, predict the position of the vehicle at the predicted intersection time through a linear motion model based on the current vehicle speed and position; then, calculate the difference between the predicted position of the vehicle at the intersection time and the predicted position of the intersection as a first risk assessment coefficient, which reflects the potential collision risk between the vehicle and the obstacle; then, calculate the time difference between the predicted intersection time and the current time as a second risk assessment coefficient, which represents the time margin for predicting the collision, the smaller the time, the more urgent it is; finally, based on the first risk assessment coefficient and the second risk assessment coefficient, conduct a comprehensive assessment and give a collision risk level, where the collision risk level can be divided into: low risk (low distance, long time), medium risk (medium distance, short time), and high risk (small distance, short time).

[0062] Optionally, weighted average, linear regression or other forms of algorithms are used to calculate the collision risk level based on the first risk assessment coefficient and the second risk assessment coefficient.

[0063] The above method process predicts potential collision risks by combining vehicle position, obstacle coordinates and motion information, and performs quantitative evaluation on them, providing a decision-making basis for subsequent corresponding actions.

[0064] In some implementations, performing collision risk assessment according to the first risk assessment coefficient and the second risk assessment coefficient to obtain the collision risk level includes: A collision risk level identification table is obtained, wherein the collision risk level identification table is a data table for pre-setting identification risk levels; the first risk assessment coefficient and the second risk assessment coefficient are input into the collision risk level identification table to obtain the collision risk level.

[0065] Specifically, the collision risk level identification table is a pre-set mapping table used to identify the collision risk level according to a given risk assessment coefficient (a first risk assessment coefficient and a second risk assessment coefficient).

[0066] Exemplarily, the collision risk level identification table includes multiple groups of associated first risk assessment coefficient intervals and second risk assessment coefficient intervals, and each group of associated first risk assessment coefficient intervals and second risk assessment coefficient intervals corresponds to a risk level, wherein the multiple first risk assessment coefficient intervals in the multiple groups define different intervals of the first risk assessment coefficient (such as: 2~5 meters, 5~10 meters, etc.); the multiple second risk assessment coefficient intervals in the multiple groups define different intervals of the second risk assessment coefficient (such as: 2~5 seconds, 5~10 seconds, etc.).

[0067] Specifically, according to the first risk assessment coefficient and the second risk assessment coefficient, the corresponding interval in the collision risk level identification table is searched. For example, if the first risk assessment coefficient is 3 meters and the second risk assessment coefficient is 4 seconds, the collision risk level identification table is searched to determine the rows where the first risk assessment coefficient is in the range of 2 to 5 meters and the second risk assessment coefficient is in the range of 2 to 5 seconds, and the corresponding medium risk identification is extracted as the collision risk level.

[0068] By inputting the first risk assessment coefficient and the second risk assessment coefficient to match the pre-set risk level identification table, and finally outputting the collision risk level, the collision risk faced by the vehicle can be quantified and classified so that further risk avoidance measures can be taken (such as adjusting the speed, enabling automatic braking, etc.).

[0069] When the collision risk level is greater than or equal to a first risk level threshold, emergency braking is performed.

[0070] Specifically, if the collision risk level is greater than or equal to the first risk level threshold, it can be considered that the current collision prediction result indicates that the target vehicle has a high probability of collision, and emergency braking is automatically initiated to slow down the vehicle or stop it completely before the collision occurs, so as to reduce the possibility of a collision or mitigate the consequences of a collision.

[0071] When the collision risk level is greater than a second risk level threshold, a collision warning signal is generated for warning.

[0072] Specifically, if the collision risk level is greater than the second risk level threshold, it means that although the target vehicle faces certain risks, it is still within the controllable range. At this time, a warning signal is generated and sent to the driver through the vehicle's sound and vision (such as the in-car screen or external lights), reminding the driver to pay attention to obstacles ahead and may need to take avoidance measures, thereby achieving the effect of improving reaction time and reducing the probability of accidents.

[0073] Optionally, the size relationship between the first risk level threshold and the second risk level threshold is determined based on the collision warning requirements of the target vehicle, and the difference between the first risk level threshold and the second risk level threshold can also be adjusted according to the requirements; illustratively, if the first risk level threshold is greater than the second risk level threshold, then when emergency braking is performed when the collision risk level is greater than or equal to the first risk level threshold, a warning signal is synchronously triggered for warning; if the first risk level threshold is equal to the second risk level threshold, emergency braking and warning signal are directly executed only when the collision risk is high; through the above-mentioned diversified configuration method of the first risk level threshold and the second risk level threshold, a flexible adjustment space is provided for the collision warning of the target vehicle, which helps to achieve adaptive adjustment of the collision warning function and the collision warning sensitivity.

[0074] In summary, the collision warning method based on fusion sensor provided by the present invention has the following technical effects: The point cloud information in front of the vehicle is collected by millimeter-wave radar, and the image data in front of the vehicle is obtained by a camera device; the radar point cloud information and the image information are fused by using a three-dimensional environment analysis model to generate a three-dimensional environment in front of the vehicle including the obstacle distribution coordinates; the obstacle distribution coordinates in the three-dimensional environment are traversed one by one, the collision risk is evaluated and the risk level is determined; if the collision risk level reaches or exceeds the set first risk level threshold, emergency braking is triggered; if the collision risk level is lower than the second risk level threshold, a collision warning signal is generated and a warning prompt is issued to the driver. The collision warning method, device and medium based on fusion sensors disclosed in the present invention solve the defects of large amount of calculation of radar point cloud modeling, which is easy to affect the timeliness of warning, thereby realizing the initial recognition of target contour through image, and the radar point cloud performs targeted analysis and modeling of part of the point cloud data, reducing the amount of calculation, and thus achieving the technical effect of improving the timeliness of warning.

[0075] Embodiment 2 Figure 2 : is a schematic diagram of the structure of the collision warning device based on the fusion sensor of the present invention. For example, Figure 1 The flow chart of the collision warning method based on the fusion sensor of the present invention can be shown as follows: Figure 2 The structure shown is implemented.

[0076] Based on the same concept as the collision warning method based on the fusion sensor in the above embodiment, the present invention also provides a collision warning device based on the fusion sensor, including: The radar and camera data acquisition module 11 is used to collect radar point cloud information in front of the vehicle through the millimeter wave radar and collect image information in front of the vehicle through the camera device.

[0077] The analysis model acquisition module 12 is used to obtain a three-dimensional environment analysis model, wherein the three-dimensional environment analysis model includes an image processing channel and a radar point cloud processing channel.

[0078] The analysis and processing module 13 is used to process the image information in front of the vehicle through the image processing channel to obtain an obstacle contour recognition result, and to perform sparse analysis on the radar point cloud information in front of the vehicle based on the obstacle contour recognition result through the radar point cloud processing channel to obtain obstacle positioning timing information.

[0079] The three-dimensional environment fusion module 14 is used to construct a three-dimensional environment in front of the vehicle according to the obstacle contour recognition result and the obstacle positioning time sequence information, wherein the three-dimensional environment in front of the vehicle includes obstacle distribution coordinates.

[0080] The collision risk assessment module 15 is used to traverse the obstacle distribution coordinates to perform collision risk assessment and obtain a collision risk level.

[0081] The emergency braking module 16 is configured to perform emergency braking when the collision risk level is greater than or equal to a first risk level threshold.

[0082] The collision warning module 17 is used to generate a collision warning signal for warning when the collision risk level is greater than a second risk level threshold.

[0083] In some embodiments, the analysis and processing module 13 further includes: The radar point cloud segmentation unit is used to segment the radar point cloud information in front of the vehicle according to the first step to obtain the radar point cloud time series information in front of the vehicle.

[0084] The image information segmentation unit is used to segment the image information in front of the vehicle according to a second step length to obtain time sequence information of the image in front of the vehicle, wherein the second step length is greater than the first step length.

[0085] The obstacle contour recognition unit is used to input the image information in front of the vehicle into the image processing channel to obtain the obstacle contour recognition result.

[0086] A sparse analysis unit is used to perform sparse analysis on the radar point cloud time series information in front of the vehicle based on the obstacle contour recognition result through the radar point cloud processing channel to obtain the obstacle positioning time series information.

[0087] Furthermore, the sparse analysis unit in the analysis and processing module 13 also includes: The contour extraction unit is used to extract a first obstacle contour according to the obstacle contour recognition result.

[0088] The point cloud distribution information extraction unit is used to extract the radar point cloud distribution information at the first moment according to the time series information of the radar point cloud in front of the vehicle.

[0089] The matching point cloud analysis unit is used to perform sparse analysis on the radar point cloud distribution information at the first moment based on the first obstacle contour to obtain a matching point cloud of the obstacle contour at the first moment.

[0090] The timing iteration analysis unit is used to perform sparse analysis on the radar point cloud distribution information at the Nth moment of the radar point cloud timing information in front of the vehicle based on the first obstacle contour, so as to obtain the obstacle contour matching point cloud at the Nth moment.

[0091] The positioning timing information construction unit is used to construct the obstacle positioning timing information according to the obstacle contour matching point cloud at the first moment to the obstacle contour matching point cloud at the Nth moment.

[0092] Furthermore, the execution steps of the matching point cloud analysis unit in the analysis and processing module 13 include: According to the first obstacle contour, the radar point cloud distribution information is randomly distributed at the first moment to obtain a first point cloud distribution area; the inner contour neighborhood density and the outer contour neighborhood density of the selected point cloud that is inside the first obstacle contour and has a distance less than or equal to a distance threshold from the first obstacle contour are calculated; the ratio of the inner contour neighborhood density to the outer contour neighborhood density is calculated and set as the point cloud matching degree; the proportion of the number of point clouds whose point cloud matching degrees of all selected point clouds are greater than or equal to the point cloud matching degree threshold is counted and set as the first point cloud distribution area matching degree, and added to the point cloud distribution area matching degree set; the first point cloud distribution area is updated until the first obstacle contour distribution history area exceeds the preset area proportion of the radar point cloud distribution information at the first moment, and the maximum value point cloud distribution area in the point cloud distribution area matching degree set is extracted and set as the obstacle contour matching point cloud at the first moment.

[0093] In some embodiments, the collision risk assessment module 15 includes: The obstacle distribution coordinate time series information extraction unit is used to extract the first obstacle distribution coordinate time series information according to the obstacle distribution coordinates.

[0094] The vehicle status acquisition unit is used to obtain the vehicle moving speed, the vehicle real-time position and the preset road trajectory.

[0095] The obstacle movement prediction unit is used to obtain the future time zone coordinate time series information of the first obstacle by performing movement prediction on the first obstacle distribution coordinate time series information.

[0096] The intersection prediction analysis unit is used to obtain the intersection prediction time and intersection prediction position when the future time zone coordinate time series information of the first obstacle intersects with the preset road trajectory.

[0097] The vehicle position prediction unit is used to predict the vehicle position at the intersection prediction moment based on the vehicle moving speed and the vehicle real-time position, so as to obtain the vehicle predicted position at the intersection moment.

[0098] The first risk assessment unit is used to calculate the difference between the predicted position of the vehicle at the intersection moment and the predicted position of the intersection, and set it as a first risk assessment coefficient.

[0099] The second risk assessment unit is used to calculate the difference between the intersection prediction time and the current time, and set it as a second risk assessment coefficient.

[0100] A risk level assessment unit is used to perform a collision risk assessment according to the first risk assessment coefficient and the second risk assessment coefficient to obtain the collision risk level.

[0101] In some implementations, the execution step of the risk level assessment unit in the collision risk assessment module 15 further includes: A collision risk level identification table is obtained, wherein the collision risk level identification table is a data table for pre-setting identification risk levels; the first risk assessment coefficient and the second risk assessment coefficient are input into the collision risk level identification table to obtain the collision risk level.

[0102] Embodiment 3 The present application also provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the collision warning method based on fusion sensor in embodiment 1 are implemented.

[0103] It should be understood that the embodiments mentioned in this specification focus on their differences from other embodiments. The specific embodiments in the aforementioned embodiment one are also applicable to the collision warning device based on fusion sensor described in embodiment two. For the sake of brevity of the specification, they will not be further elaborated here.

[0104] It should be understood that the embodiments disclosed in the present invention and the above description can enable those skilled in the art to use the present invention to implement the present invention. At the same time, the present invention is not limited to the above-mentioned embodiments. It should be understood that those skilled in the art can still modify the technical solutions recorded in the above-mentioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A collision warning method based on fusion sensors, characterized in that: include: The radar point cloud information in front of the vehicle is collected by the millimeter wave radar, and the image information in front of the vehicle is collected by the camera device; Obtaining a three-dimensional environment analysis model, wherein the three-dimensional environment analysis model includes an image processing channel and a radar point cloud processing channel; The image information in front of the vehicle is processed through the image processing channel to obtain an obstacle contour recognition result, and the radar point cloud information in front of the vehicle is sparsely analyzed based on the obstacle contour recognition result through the radar point cloud processing channel to obtain obstacle positioning time sequence information; Constructing a three-dimensional environment in front of the vehicle according to the obstacle contour recognition result and the obstacle positioning time sequence information, wherein the three-dimensional environment in front of the vehicle includes obstacle distribution coordinates; Traversing the obstacle distribution coordinates to perform collision risk assessment and obtain a collision risk level; When the collision risk level is greater than or equal to a first risk level threshold, performing emergency braking; When the collision risk level is greater than a second risk level threshold, a collision warning signal is generated for warning.

2. The method according to claim 1, characterized in that The image information in front of the vehicle is processed through the image processing channel to obtain an obstacle contour recognition result, and the radar point cloud information in front of the vehicle is sparsely analyzed based on the obstacle contour recognition result through the radar point cloud processing channel to obtain obstacle positioning time sequence information, including: According to the first step, the radar point cloud information in front of the vehicle is segmented to obtain the time series information of the radar point cloud in front of the vehicle; Segmenting the image information in front of the vehicle according to a second step length to obtain time series information of the image in front of the vehicle, wherein the second step length is greater than the first step length; Inputting the image information in front of the vehicle into the image processing channel to obtain the obstacle contour recognition result; The obstacle positioning time series information is obtained by performing sparse analysis on the radar point cloud time series information in front of the vehicle based on the obstacle contour recognition result through the radar point cloud processing channel.

3. The method according to claim 2, characterized in that The obstacle positioning time series information is obtained by performing sparse analysis on the radar point cloud time series information in front of the vehicle based on the obstacle contour recognition result through the radar point cloud processing channel, including: Extracting a first obstacle contour according to the obstacle contour recognition result; Extracting radar point cloud distribution information at the first moment according to the time series information of the radar point cloud in front of the vehicle; Based on the first obstacle contour, performing sparse analysis on the radar point cloud distribution information at the first moment to obtain a matching point cloud of the obstacle contour at the first moment; Until, based on the first obstacle contour, sparse analysis is performed on the radar point cloud distribution information at the Nth moment of the radar point cloud time series information in front of the vehicle to obtain the obstacle contour matching point cloud at the Nth moment; The obstacle positioning timing information is constructed according to the obstacle contour matching point cloud at the first moment to the obstacle contour matching point cloud at the Nth moment.

4. The method according to claim 3, characterized in that Based on the first obstacle contour, sparse analysis is performed on the radar point cloud distribution information at the first moment to obtain the obstacle contour matching point cloud at the first moment, including: According to the first obstacle contour, at the first moment, radar point cloud distribution information is randomly distributed to obtain a first point cloud distribution area; Calculating the inner-contour neighborhood density and the outer-contour neighborhood density of the selected point cloud that is inside the first obstacle contour and has a distance from the first obstacle contour that is less than or equal to a distance threshold; Calculate the ratio of the neighborhood density inside the contour to the neighborhood density outside the contour, and set it as the point cloud matching degree; Count the percentage of point clouds whose point cloud matching degree of all selected point clouds is greater than or equal to the point cloud matching degree threshold, set it as the first point cloud distribution area matching degree, and add it to the point cloud distribution area matching degree set; The first point cloud distribution area is updated until the historical area of ​​the first obstacle contour distribution exceeds the preset area ratio of the radar point cloud distribution information at the first moment, and the point cloud distribution area with the maximum value in the point cloud distribution area matching degree set is extracted and set as the obstacle contour matching point cloud at the first moment.

5. The method according to claim 1, characterized in that Traversing the obstacle distribution coordinates to perform collision risk assessment and obtain a collision risk level, including: Extracting first obstacle distribution coordinate time series information according to the obstacle distribution coordinates; Obtain vehicle moving speed, vehicle real-time location and preset road trajectory; Obtaining the future time zone coordinate time series information of the first obstacle by performing movement prediction on the first obstacle distribution coordinate time series information; When the future time zone coordinate time series information of the first obstacle intersects with the preset road trajectory, obtaining the predicted intersection time and the predicted intersection position; The vehicle position prediction at the intersection prediction time is performed based on the vehicle moving speed and the vehicle real-time position, to obtain the vehicle predicted position at the intersection time; Calculate the difference between the predicted position of the vehicle at the intersection moment and the predicted position of the intersection, and set it as a first risk assessment coefficient; Calculate the difference between the intersection prediction time and the current time, and set it as the second risk assessment coefficient; A collision risk assessment is performed according to the first risk assessment coefficient and the second risk assessment coefficient to obtain the collision risk level.

6. The method according to claim 5, characterized in that Performing a collision risk assessment according to the first risk assessment coefficient and the second risk assessment coefficient to obtain the collision risk level includes: Obtaining a collision risk level identification table, wherein the collision risk level identification table is a data table of pre-set identification risk levels; The first risk assessment coefficient and the second risk assessment coefficient are input into the collision risk level identification table to obtain the collision risk level.

7. A collision warning device based on fusion sensors, characterized in that: The device is used to execute the collision warning method based on fusion sensor according to any one of claims 1 to 6, and the device includes: Radar and camera data acquisition module, used to collect radar point cloud information in front of the vehicle through millimeter wave radar, and collect image information in front of the vehicle through camera device; An analysis model acquisition module, used to obtain a three-dimensional environment analysis model, wherein the three-dimensional environment analysis model includes an image processing channel and a radar point cloud processing channel; an analysis and processing module, configured to process the image information in front of the vehicle through the image processing channel to obtain an obstacle contour recognition result, and to perform sparse analysis on the radar point cloud information in front of the vehicle based on the obstacle contour recognition result through the radar point cloud processing channel to obtain obstacle positioning time sequence information; A three-dimensional environment fusion module, used to construct a three-dimensional environment in front of the vehicle according to the obstacle contour recognition result and the obstacle positioning time sequence information, wherein the three-dimensional environment in front of the vehicle includes obstacle distribution coordinates; A collision risk assessment module, used to traverse the obstacle distribution coordinates to perform collision risk assessment and obtain a collision risk level; an emergency braking module, configured to perform emergency braking when the collision risk level is greater than or equal to a first risk level threshold; The collision warning module is used to generate a collision warning signal for warning when the collision risk level is greater than a second risk level threshold.

8. The device according to claim 7, characterized in that The analysis and processing module also includes: A radar point cloud segmentation unit, configured to segment the radar point cloud information in front of the vehicle according to the first step length, and obtain time series information of the radar point cloud in front of the vehicle; an image information segmentation unit, configured to segment the image information in front of the vehicle according to a second step length to obtain time sequence information of the image in front of the vehicle, wherein the second step length is greater than the first step length; An obstacle contour recognition unit, used for inputting the image information in front of the vehicle into the image processing channel to obtain the obstacle contour recognition result; A sparse analysis unit is used to perform sparse analysis on the radar point cloud time series information in front of the vehicle based on the obstacle contour recognition result through the radar point cloud processing channel to obtain the obstacle positioning time series information.

9. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the collision warning method based on fusion sensor as described in any one of claims 1 to 6 is implemented.

Images