Multi-sensor data fusion method, device, equipment and medium

By screening and fusion of multi-sensor data in vehicle assisted driving technology, the problem of low fusion efficiency of multi-sensor data is solved, and the stability and reliability of assisted driving are improved.

CN119027916APending Publication Date: 2024-11-26FAW JIEFANG AUTOMOTIVE CO
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411222462.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The multi-sensor data fusion efficiency in existing vehicle assisted driving technologies is low, resulting in large amounts of calculations, slow calculation speed, and difficult to issue execution instructions in a timely manner, affecting the stability and reliability of assisted driving.

Method used

The image sensor obtains lane line data and image initial obstacle data. The radar sensor obtains radar initial obstacle data, filters the data based on preset filter conditions, obtains image target obstacle data and radar target obstacle data, and performs data fusion Processing, fusion of obstacle data is used to assist driving. The filtering conditions include the ratio of the lateral offset distance to the obstacle lane width is less than the preset value, paying attention to obstacles near the vehicle's current lane to reduce the amount of data fusion calculation.

Benefits of technology

It improves the efficiency of multi-sensor data fusion, reduces the amount of calculation, enables assisted driving application software to issue execution instructions in a timely manner, and improves the stability and reliability of assisted driving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027916A_ABST
    Figure CN119027916A_ABST
Patent Text Reader

Abstract

The invention relates to a multi-sensor data fusion method and device, equipment and a medium. The method comprises the following steps: acquiring lane line data and image initial obstacle data currently corresponding to a vehicle through an image sensor, and acquiring radar initial obstacle data corresponding to the vehicle through a radar sensor, performing screening processing on the image initial obstacle data and the radar initial obstacle data based on the lane line data and a preset screening condition to obtain image target obstacle data and radar target obstacle data, and performing data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data; the fused obstacle data is used for auxiliary driving of the vehicle. By adopting the method, the data fusion efficiency can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of vehicle assisted driving technology, and in particular to a multi-sensor data fusion method, device, equipment and medium. Background Art

[0002] In the field of commercial vehicle assisted driving technology, Advanced Driver Assistance System (ADAS) is used to improve driving safety, enhance driving comfort and reduce the burden on drivers. ADAS senses changes in the vehicle's surroundings and provides functions such as lane departure warning, collision warning, and automatic emergency braking to help drivers better control the vehicle and reduce the occurrence of traffic accidents. Among them, the advanced driver assistance system needs to accurately determine the relevant information of obstacles during the vehicle's driving process.

[0003] In related technologies, data from multiple sensors such as cameras and millimeter-wave radars are generally fused to determine information such as the location and running speed of obstacles.

[0004] However, the efficiency of the multi-sensor data fusion method in the related art is low. Summary of the invention

[0005] Based on this, it is necessary to provide a multi-sensor data fusion method, device, equipment and medium that can improve efficiency in response to the above technical problems.

[0006] In a first aspect, the present application provides a multi-sensor data fusion method, comprising:

[0007] The lane line data and initial image obstacle data corresponding to the vehicle are obtained through the image sensor, and the initial radar obstacle data corresponding to the vehicle are obtained through the radar sensor;

[0008] The image initial obstacle data and the radar initial obstacle data are screened and processed based on the lane line data and the preset screening conditions to obtain the image target obstacle data and the radar target obstacle data. The preset screening conditions include that the ratio between the lateral offset distance and the obstacle lane width is less than a preset value, the lateral offset distance is the shortest distance between the obstacle and the center trajectory line corresponding to the vehicle, and the obstacle lane width is the lane width corresponding to the position of the obstacle;

[0009] Data fusion processing is performed based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data; the fused obstacle data is used for assisted driving of the vehicle.

[0010] In one of the embodiments, the image initial obstacle data includes a longitudinal distance and a lateral distance of the obstacle relative to the vehicle, the longitudinal distance represents the distance between the obstacle and the vehicle in a first direction, and the lateral distance represents the distance between the obstacle and the vehicle in a second direction, the first direction corresponds to the forward direction of the vehicle, and the second direction is perpendicular to the first direction;

[0011] The process of filtering and processing the initial obstacle data of the image based on the lane line data and the preset filtering conditions to obtain the target obstacle data of the image includes:

[0012] Obstacle lane width is obtained by calculating and processing based on lane line data and longitudinal distance;

[0013] Calculate and process based on lane line data, longitudinal distance and lateral distance to obtain lateral offset distance;

[0014] If the ratio between the lateral offset distance and the width of the obstacle lane is greater than or equal to the preset value, the initial obstacle data of the image is deleted;

[0015] If the ratio between the lateral offset distance and the obstacle lane width is less than a preset value, the image initial obstacle data is used as the image target obstacle data.

[0016] In one of the embodiments, the number of image target obstacle data is at least one, and the number of radar target obstacle data is at least one;

[0017] Data fusion processing is performed based on image target obstacle data and radar target obstacle data to obtain fused obstacle data, including:

[0018] For each image target obstacle data, in the order of the longitudinal distance in the radar target obstacle data from small to large, it is judged in turn whether each radar target obstacle data and the image target obstacle data meet the preset matching conditions, and the preset matching conditions include that the longitudinal distance difference, the lateral distance difference and the longitudinal speed difference between the image target obstacle data and the radar target obstacle data are all less than the corresponding preset difference thresholds;

[0019] When the radar target obstacle data and the image target obstacle data meet a preset matching condition, a matching cost value of the radar target obstacle data relative to the image target obstacle data is obtained, and the radar target obstacle data with the smallest matching cost value is determined as the data to be fused;

[0020] The data to be fused is fused with the image target obstacle data to obtain fused obstacle data corresponding to the image target obstacle data, wherein the data to be fused does not participate in the data fusion processing process of other image target obstacle data.

[0021] In one embodiment, the data to be fused is fused with the image target obstacle data to obtain fused obstacle data corresponding to the image target obstacle data, including:

[0022] The longitudinal distance, longitudinal velocity and longitudinal acceleration in the data to be fused are determined as the longitudinal distance, longitudinal velocity and longitudinal acceleration in the fused obstacle data, and the lateral distance, lateral velocity and lateral acceleration in the image target obstacle data are determined as the lateral distance, lateral velocity and lateral acceleration in the fused obstacle data, so as to obtain the fused obstacle data.

[0023] In one embodiment, the process of performing data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain the fused obstacle data further includes:

[0024] When all radar target obstacle data and image target obstacle data do not satisfy the preset matching condition, the image target obstacle data is used as the fused obstacle data corresponding to the image target obstacle data.

[0025] In one embodiment, the amount of image target obstacle data is 0, the amount of radar target obstacle data is not 0, and data fusion processing is performed based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data, including:

[0026] The radar target obstacle data with the smallest lateral distance among the radar target obstacle data are determined as the fused obstacle data.

[0027] In one embodiment, the amount of image target obstacle data is not 0, the amount of radar target obstacle data is 0, and data fusion processing is performed based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data, including:

[0028] The image target obstacle data is determined as fused obstacle data.

[0029] In a second aspect, the present application also provides a multi-sensor data fusion device, comprising:

[0030] An acquisition module is used to acquire lane line data and image initial obstacle data corresponding to the vehicle through an image sensor, and acquire radar initial obstacle data corresponding to the vehicle through a radar sensor;

[0031] A screening module is used to screen and process the image initial obstacle data and the radar initial obstacle data based on the lane line data and preset screening conditions to obtain image target obstacle data and radar target obstacle data. The preset screening conditions include that the ratio between the lateral offset distance and the obstacle lane width is less than a preset value, the lateral offset distance is the shortest distance between the obstacle and the center trajectory line corresponding to the vehicle, and the obstacle lane width is the lane width corresponding to the position of the obstacle;

[0032] The fusion module is used to perform data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data; the fused obstacle data is used for auxiliary driving of the vehicle.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] The lane line data and initial image obstacle data corresponding to the vehicle are obtained through the image sensor, and the initial radar obstacle data corresponding to the vehicle are obtained through the radar sensor;

[0035] The image initial obstacle data and the radar initial obstacle data are screened and processed based on the lane line data and the preset screening conditions to obtain the image target obstacle data and the radar target obstacle data. The preset screening conditions include that the ratio between the lateral offset distance and the obstacle lane width is less than a preset value, the lateral offset distance is the shortest distance between the obstacle and the center trajectory line corresponding to the vehicle, and the obstacle lane width is the lane width corresponding to the position of the obstacle;

[0036] Data fusion processing is performed based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data; the fused obstacle data is used for assisted driving of the vehicle.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0038] The lane line data and initial image obstacle data corresponding to the vehicle are obtained through the image sensor, and the initial radar obstacle data corresponding to the vehicle are obtained through the radar sensor;

[0039] The image initial obstacle data and the radar initial obstacle data are screened and processed based on the lane line data and the preset screening conditions to obtain the image target obstacle data and the radar target obstacle data. The preset screening conditions include that the ratio between the lateral offset distance and the obstacle lane width is less than a preset value, the lateral offset distance is the shortest distance between the obstacle and the center trajectory line corresponding to the vehicle, and the obstacle lane width is the lane width corresponding to the position of the obstacle;

[0040] Data fusion processing is performed based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data; the fused obstacle data is used for assisted driving of the vehicle.

[0041] The multi-sensor data fusion method, device, equipment and medium obtain lane line data and image initial obstacle data corresponding to the vehicle through an image sensor, obtain radar initial obstacle data corresponding to the vehicle through a radar sensor, filter and process the image initial obstacle data and radar initial obstacle data based on the lane line data and preset filtering conditions to obtain image target obstacle data and radar target obstacle data, the preset filtering conditions include that the ratio between the lateral offset distance and the obstacle lane width is less than a preset value, the lateral offset distance is the shortest distance between the obstacle and the center trajectory line corresponding to the vehicle, the obstacle lane width is the lane width corresponding to the obstacle position, and data fusion processing is performed based on the image target obstacle data and the radar target obstacle data to obtain the fused obstacle data. Obstacle data; fused obstacle data is used for vehicle assisted driving; in this way, the problem of low data fusion efficiency caused by a large number of original target-level obstacles obtained by image sensors and radar sensors and a large amount of data fusion calculation is avoided. In this embodiment, the original target-level obstacles are screened according to the relationship between the shortest distance between the obstacle and the center trajectory line corresponding to the vehicle and the lane width corresponding to the obstacle position, and obstacles near the vehicle's current driving lane are paid attention to. The screened image target obstacle data and radar target obstacle data are used for subsequent data fusion processing, which reduces the amount of data fusion calculation and improves the efficiency of data fusion, so that the assisted driving application software can issue corresponding execution instructions according to the fusion results in time, thereby improving the stability and reliability of assisted driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] FIG1 is an application environment diagram of a multi-sensor data fusion method according to an embodiment;

[0044] Figure 2 A schematic diagram of a multi-sensor data fusion method according to an embodiment;

[0045] Figure 3 is an exemplary schematic diagram of the relative position relationship between a vehicle and an obstacle in one embodiment;

[0046] Figure 4 is an exemplary schematic diagram of the relative position relationship between a vehicle and an obstacle in another embodiment;

[0047] Figure 5 A schematic diagram of a flow chart of steps for obtaining image target obstacle data in one embodiment;

[0048] Figure 6 is a flow chart of the steps of obtaining fused obstacle data in one embodiment;

[0049] Figure 7 is a structural block diagram of a multi-sensor data fusion device in one embodiment;

[0050] FIG. 8 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] The multi-sensor data fusion method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. In some embodiments, the structures shown in Figure 1 can be used as part of a perception device in a driving assistance system. The image sensor 102 may be a monocular camera or a binocular camera; illustratively, the image sensor 102 is installed in the middle of the top of the front windshield of the vehicle to obtain image data directly in front of the vehicle; the system on a chip (SoC) 104 runs various parts of the entire computing device corresponding to the image sensor, including an operating system and an image processing program, etc. The image processing program in the SoC 104 processes the image obtained by the image sensor 102 to obtain information corresponding to multiple target-level obstacles and lane line data of the vehicle currently traveling. The information corresponding to each target-level obstacle includes but is not limited to obstacle ID (identification), life cycle, obstacle size, position, speed and acceleration, etc. The lane line data includes but is not limited to left lane line parameters and right lane line parameters of the current lane; the radar sensor 106 may be a millimeter wave radar, or a laser radar, an infrared radar, an ultrasonic radar or a corner radar; the radar sensor 106 may output information corresponding to multiple target-level obstacles, and the information corresponding to each target-level obstacle includes but is not limited to obstacle ID, life cycle, obstacle size, position, speed and acceleration, etc.; the microcontroller (Micro Control The microcontroller 108 is used to receive target-level obstacle information, lane line data, target-level obstacle information provided by the radar sensor 106, and vehicle driving information provided by the vehicle controller. The microcontroller 108 runs auxiliary driving application software, including but not limited to perception fusion application software, ACC (Adaptive Cruise Control) application software, LCK (Lane Centering Keep) application software, and AEB (Autonomous Emergency Braking) application software, and sends the execution results of each auxiliary driving application software to the vehicle controller 110 through the CAN bus. The vehicle controller 110 is used to provide the image sensor 102 and the radar sensor 106 with information such as the vehicle speed and gear position. The image sensor 102 and the radar sensor 106 need to detect that the vehicle meets certain conditions before sending effective information corresponding to the target-level obstacle; the vehicle controller 110 is also used to receive execution instructions sent by the driving assistance application software, control the vehicle's throttle opening, brake pedal opening, steering wheel angle, etc., so that the vehicle can stably follow the vehicle in the current lane.

[0053] In an exemplary embodiment, Figure 2As shown, a multi-sensor data fusion method is provided, which is described by taking the method applied to the microcontroller 108 in FIG. 1 as an example, and includes the following steps 202 to 206. Among them:

[0054] Step 202: Acquire lane line data and image initial obstacle data corresponding to the vehicle through an image sensor, and acquire radar initial obstacle data corresponding to the vehicle through a radar sensor.

[0055] The image initial obstacle data refers to the information corresponding to the target-level obstacles obtained by the image sensor, and the radar initial obstacle data refers to the information corresponding to the target-level obstacles obtained by the radar sensor. The target-level obstacle refers to an independent individual that can be described by attributes such as position, speed, acceleration, and category; for example, the target-level obstacle can be other vehicles running in front of the vehicle.

[0056] The data types included in the image initial obstacle data and the radar initial obstacle data may be the same, wherein “image” and “radar” are used to define the source of the data.

[0057] Among them, a quadratic or cubic polynomial is generally used as the mathematical model of the lane line, and the polynomial coefficients are the lane line parameters; in this embodiment, the lane line data includes the left lane line parameters and the right lane line parameters corresponding to the lane in which the vehicle is currently traveling.

[0058] Exemplarily, the process of acquiring lane line data may include: acquiring a front image through an image sensor; detecting left lane lines and right lane lines based on the front image; performing polynomial fitting based on the detected left lane lines and right lane lines to obtain left lane line parameters and right lane line parameters, that is, each coefficient of the polynomial corresponding to the left lane line, and each coefficient of the polynomial corresponding to the right lane line.

[0059] Step 204 , the image initial obstacle data and the radar initial obstacle data are screened based on the lane line data and the preset screening conditions to obtain image target obstacle data and radar target obstacle data.

[0060] Among them, the preset screening conditions include that the ratio of the lateral offset distance to the obstacle lane width is less than a preset value, the lateral offset distance is the shortest distance between the obstacle and the center trajectory line corresponding to the vehicle, and the obstacle lane width is the lane width corresponding to the position of the obstacle.

[0061] Please refer to Figure 3 , which is an example of a schematic diagram for calculating the relative position between a vehicle and an obstacle when the vehicle is traveling on a curve with a certain curvature; Figure 3As shown in the figure, assuming that the vehicle is only driving in the lane (that is, the lane corresponding to the two blue dotted lines) and driving in the middle of the lane, the center trajectory line corresponding to the vehicle is as follows Figure 3 As shown by the red dotted line in the figure; point O is the origin of the coordinate system corresponding to the image sensor; point P is the point closest to the vehicle, point Q is the point closest to the center track line, and line segment PQ is the lateral offset distance of the obstacle relative to the center track line corresponding to the vehicle. In this embodiment, S ltr The lateral offset distance is used to represent the distance of the obstacle relative to the lateral distance of the vehicle.

[0062] Exemplarily, the mathematical expression of the center trajectory is shown in Formula 1:

[0063] , formula 1

[0064] in, , , and is the lane line parameter corresponding to point O, which can be determined by the average value of the left lane line parameter and the right lane line parameter corresponding to point O, that is, ; represents the ordinate at a distance X from point O. When the vehicle is traveling along the center line of the lane, =0.

[0065] Please refer to Figure 4 , which is an example of a schematic diagram for calculating the relative position between a vehicle and an obstacle when the vehicle is traveling on a straight road; Figure 4 , the shortest distance PQ between the obstacle and the center trajectory line corresponding to the vehicle is the ordinate of point P in the first obstacle data given by the image sensor. On a straight road, the center trajectory line is a horizontal straight line passing through the origin of the image sensor coordinate system (i.e. Figure 3 The x-axis in Eq. 1 is 0.

[0066] Exemplarily, the width of the obstacle lane is the lane width corresponding to the position of the obstacle, which refers to the difference between the ordinate corresponding to the left lane line and the ordinate corresponding to the right lane line at the position of point P. Exemplarily, the width of the obstacle lane remains unchanged when the vehicle is traveling on a straight lane.

[0067] Among them, screening the image initial obstacle data and the radar initial obstacle data means deleting the image initial obstacle data that does not meet the preset screening conditions in the image initial obstacle data, deleting the radar initial obstacle data that does not meet the preset screening conditions in the radar initial obstacle data, and only retaining the image initial obstacle data and the radar initial obstacle data that meet the preset screening conditions for subsequent fusion processing, wherein the number of image target obstacle data is less than or equal to the number of image initial obstacle data, and the number of radar target obstacle data is less than or equal to the number of image initial obstacle data. This avoids the problem of using all image initial obstacle data for data fusion in the related art, which results in large amount of calculation, slow calculation speed, and low data fusion efficiency.

[0068] Among them, the ratio between the lateral offset distance and the obstacle lane width is less than the preset value to indicate the distance relationship between the obstacle and the current lane of the vehicle; if the ratio between the lateral offset distance and the obstacle lane width is greater than or equal to the preset value, this embodiment considers that the distance between the obstacle and the current lane of the vehicle is far, and does not serve as the source data for data fusion; if the ratio between the lateral offset distance and the obstacle lane width is less than the preset value, this embodiment considers that the distance between the obstacle and the current lane of the vehicle is close or is located in the current lane, and retains the image initial obstacle data or radar initial obstacle data corresponding to the obstacle as image target obstacle data or radar target obstacle data for subsequent data fusion processing.

[0069] In a possible implementation, the preset ratio ranges from 1.5 to 2.5; optionally, the preset ratio is 2, that is, the image initial obstacle data and radar initial obstacle data with a lateral offset distance outside 2 times the obstacle lane width are filtered out, and only the image initial obstacle data and radar initial obstacle data with a lateral offset distance within 2 times the obstacle lane width are retained.

[0070] Step 206: Perform data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data; the fused obstacle data is used for assisting the vehicle in driving.

[0071] The multi-sensor data fusion method provided in the above embodiment obtains lane line data and image initial obstacle data corresponding to the vehicle through an image sensor, obtains radar initial obstacle data corresponding to the vehicle through a radar sensor, and screens the image initial obstacle data and radar initial obstacle data based on the lane line data and preset screening conditions to obtain image target obstacle data and radar target obstacle data. The preset screening conditions include that the ratio between the lateral offset distance and the obstacle lane width is less than a preset value, the lateral offset distance is the shortest distance between the obstacle and the center trajectory line corresponding to the vehicle, and the obstacle lane width is the lane width corresponding to the position of the obstacle. Data fusion processing is performed based on the image target obstacle data and the radar target obstacle data to obtain a fused obstacle. Data; fused obstacle data is used for assisted driving of the vehicle; in this way, the problem of low data fusion efficiency caused by a large number of original target-level obstacles obtained by image sensors and radar sensors and a large amount of data fusion calculation is avoided. In this embodiment, the original target-level obstacles are screened according to the relationship between the shortest distance between the obstacle and the center trajectory line corresponding to the vehicle and the lane width corresponding to the obstacle position, and obstacles near the current lane of the vehicle are paid attention to. The screened image target obstacle data and radar target obstacle data are used for subsequent data fusion processing, which reduces the amount of data fusion calculation and improves the efficiency of data fusion, so that the assisted driving application software can issue corresponding execution instructions according to the fusion results in time, thereby improving the stability and reliability of assisted driving.

[0072] In an exemplary embodiment, based on Figure 2 In the embodiment shown, in this embodiment, the initial obstacle data of the image includes the longitudinal distance and the lateral distance of the obstacle relative to the vehicle, the longitudinal distance represents the distance between the obstacle and the vehicle in a first direction, and the lateral distance represents the distance between the obstacle and the vehicle in a second direction, the first direction corresponds to the forward direction of the vehicle, and the second direction is perpendicular to the first direction.

[0073] In one possible implementation, please refer to Figure 3 , the first direction is the X direction in the image sensor coordinate system, the second direction is the Y direction in the image sensor coordinate system, the longitudinal distance may be the X coordinate of point P in the image sensor coordinate system; the lateral distance may be the Y coordinate of point P in the image sensor coordinate system.

[0074] In this embodiment, please refer to Figure 5 The process of filtering and processing the initial obstacle data of the image based on the lane line data and the preset filtering conditions to obtain the target obstacle data of the image includes steps 502 to 508, wherein:

[0075] Step 502: Calculate and process based on the lane line data and the longitudinal distance to obtain the obstacle lane width.

[0076] For example, please refer to Figure 3 , the calculation process of the obstacle lane width Width corresponding to point P is shown in Formula 2:

[0077] , formula 2

[0078] Where X is the X coordinate of point P in the image sensor coordinate system, , , and is the left lane line parameter of the vehicle’s current lane, , , and It is the right lane line parameter of the vehicle's current lane.

[0079] Step 504 , performing calculations based on the lane line data, the longitudinal distance and the lateral distance to obtain a lateral offset distance.

[0080] In one possible implementation, please refer to Figure 3 , point R is the intersection of the tangent line of point P and point Q on the centerline trajectory and the horizontal coordinate, point A is the horizontal coordinate of point P in the image sensor coordinate system, point B is the vertical coordinate of point P in the image sensor coordinate system, point N is the intersection of the tangent line at point Q on the center trajectory and the horizontal axis in the image sensor coordinate system, length L is the longitudinal distance of the obstacle relative to the vehicle, angle α is the azimuth of the obstacle relative to the vehicle, angle β is the angle between the tangent line QN at point Q and the horizontal axis in the image sensor coordinate system, and angle γ is the angle between coil PQ and coil RP. The lateral offset distance of point P relative to the center trajectory can be shown as formula 3:

[0081] , formula 3

[0082] in, , Formula 3 can be converted into the form of Formula 4:

[0083] , Formula 4

[0084] Among them, angle β is the intersection of the tangent line of the center trajectory line at point Q and the X-axis, but the X coordinate of point Q is unknown at this time. According to Figure 3 From the content shown, it can be seen that point Q is close to point R. In this embodiment, the tangent angle of point R is used to replace the tangent angle of point Q. The X coordinate of point R is the longitudinal distance L of the obstacle. According to the mathematical expression of the center trajectory (Formula 1), it can be obtained ; That is, the calculation formula of the lateral distance offset distance can be converted into Formula 5:

[0085] , Formula 5

[0086] in, is the lateral distance in the image's initial obstacle data. For example, It can be the Y-axis coordinate of point P. In other embodiments, the image initial obstacle data does not include the lateral distance of the obstacle relative to the vehicle, but includes the azimuth angle α and the longitudinal distance L of the obstacle relative to the vehicle, which can be directly substituted into Formula 5 for calculation.

[0087] In general, when the azimuth angle α of the obstacle relative to the vehicle is small, , Formula 5 can be approximately expressed as Formula 6:

[0088] , Formula 6

[0089] Step 506: If the ratio between the lateral offset distance and the obstacle lane width is greater than or equal to a preset value, the initial obstacle data of the image is deleted.

[0090] Step 508: If the ratio between the lateral offset distance and the obstacle lane width is less than a preset value, the image initial obstacle data is used as the image target obstacle data.

[0091] Correspondingly, in this embodiment, the radar initial obstacle data includes the longitudinal distance and the lateral distance of the obstacle relative to the vehicle. Based on a similar method to steps 502 to 506, the radar initial obstacle data is processed to obtain radar target obstacle data.

[0092] In an exemplary embodiment, based on Figure 2 In the embodiment shown, the number of image target obstacle data is at least one, and the number of radar target obstacle data is at least one. In this embodiment, the number of fused obstacle data is consistent with the number of image target obstacle data.

[0093] Please refer to Figure 6 , a process of performing data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data, and sequentially performing the matching and fusion processing shown in steps 602 to 606 on each image target obstacle. In a possible implementation, all image target obstacle data are sorted in ascending order of longitudinal distance, and data fusion processing is first performed on the image target obstacle data with the smallest longitudinal distance, that is, data fusion processing is first performed on the obstacle closest to the vehicle in the vehicle's forward direction.

[0094] Step 602: In the order of the longitudinal distance in the radar target obstacle data from small to large, it is determined in turn whether each radar target obstacle data and the image target obstacle data meet the preset matching condition. The preset matching condition includes that the longitudinal distance difference, the lateral distance difference and the longitudinal speed difference between the image target obstacle data and the radar target obstacle data are all less than the corresponding preset difference thresholds.

[0095] Exemplarily, the longitudinal distance difference between the radar target obstacle data and the image target obstacle data is , the horizontal distance difference is , the longitudinal velocity difference is , when formula 7 is satisfied, it is considered that the radar target obstacle data and the image target obstacle data are matched successfully.

[0096] ;Formula 7

[0097] in, They are the longitudinal distance difference threshold, the lateral distance difference threshold and the longitudinal speed difference threshold respectively.

[0098] Step 604: When the radar target obstacle data and the image target obstacle data satisfy a preset matching condition, a matching cost value of the radar target obstacle data relative to the image target obstacle data is obtained, and the radar target obstacle data with the smallest matching cost value is determined as the data to be fused.

[0099] After the radar target obstacle data and the image target obstacle data are successfully matched, the matching cost of the radar target obstacle data for the image target obstacle data is recorded. Exemplarily, the calculation process of the matching cost cost is shown in Formula 8:

[0100] , Formula 8

[0101] There may be multiple radar target obstacle data that successfully match the image target obstacle data. At this time, the radar target obstacle data that is finally fused with the image target obstacle data is selected based on the matching cost value.

[0102] Step 606: Fusing the data to be fused with the image target obstacle data to obtain fused obstacle data corresponding to the image target obstacle data.

[0103] The data to be fused does not participate in the data fusion processing of other image target obstacle data.

[0104] In a possible implementation, a plurality of image target obstacle data form a data set, and a plurality of radar target obstacle data form a data set. Each data set is sorted in ascending order of longitudinal distance. After a data fusion process is completed for a piece of image target obstacle data and a piece of radar target obstacle data, the image target obstacle data and the radar target obstacle data are removed from the corresponding data sets.

[0105] The data to be fused is the radar target obstacle data that is successfully matched with the image target obstacle data and has the smallest matching cost.

[0106] In a possible implementation, fusing the data to be fused with the image target obstacle data to obtain fused obstacle data corresponding to the image target obstacle data further includes: determining the longitudinal distance, longitudinal velocity, and longitudinal acceleration in the data to be fused as the longitudinal distance, longitudinal velocity, and longitudinal acceleration in the fused obstacle data, and determining the lateral distance, lateral velocity, and lateral acceleration in the image target obstacle data as the lateral distance, lateral velocity, and lateral acceleration in the fused obstacle data to obtain the fused obstacle data.

[0107] In one possible implementation, please refer to Figure 6 The process of performing data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data also includes step 608, wherein:

[0108] Step 608: When all radar target obstacle data and image target obstacle data do not satisfy the preset matching condition, the image target obstacle data is used as the fused obstacle data corresponding to the image target obstacle data.

[0109] In this embodiment, the lateral distance, lateral speed, lateral acceleration, longitudinal distance, longitudinal speed and longitudinal acceleration in the fused obstacle data are consistent with those in the image target obstacle data.

[0110] In an exemplary embodiment, based on Figure 2 In the embodiment shown, the amount of image target obstacle data is 0, and the amount of radar target obstacle data is not 0. The process of performing data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data involved in this embodiment includes: determining the radar target obstacle data with the smallest lateral distance among the radar target obstacle data as the fused obstacle data.

[0111] Exemplarily, before performing data fusion processing, it is first determined whether to perform data matching based on the quantity of image target obstacle data and radar target obstacle data.

[0112] Among them, in the absence of valid image target obstacle data, the radar target obstacle data with the smallest lateral distance among the radar target obstacle data is determined as the fused obstacle data, that is, the radar target obstacle data corresponding to the obstacle closest to the current lane of the vehicle is retained as the fusion result. Exemplarily, the obstacle is used as the following target.

[0113] In an exemplary embodiment, based on Figure 2 In the illustrated embodiment, the amount of image target obstacle data is not 0, the amount of radar target obstacle data is 0, and the process of performing data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data includes: determining the image target obstacle data as fused obstacle data.

[0114] In the case where there is no valid radar target obstacle data, each image target obstacle data is directly used as each fused obstacle data. At this time, the number of fused obstacle data is the same as the number of image target obstacle data.

[0115] In a possible implementation manner, when the amount of radar target obstacle data is greater than the amount of image target obstacle data, there may be radar target obstacle data that does not participate in the subsequent data fusion processing process.

[0116] It can be understood that when the number of image target obstacle data is not 0, the number of fused obstacle data is the same as the number of image target obstacle data. When the number of image target obstacle data is 0 and the number of fused obstacle data is not 0, the number of fused obstacle data is 1.

[0117] In a possible implementation, the amount of radar target obstacle data is 0, the amount of image target obstacle data is also 0, and the amount of obtained fused obstacle data is also 0, that is, the current fusion result is no obstacle.

[0118] In an exemplary embodiment, a multi-sensor data fusion method is provided, which is described by taking the method applied to the microcontroller 108 in FIG. 1 as an example, and includes the following steps S2 to S16, wherein:

[0119] Step S2, obtaining lane line data and image initial obstacle data corresponding to the vehicle through the image sensor, and obtaining radar initial obstacle data corresponding to the vehicle through the radar sensor.

[0120] The image initial obstacle data includes the longitudinal distance and lateral distance of the obstacle relative to the vehicle, and the radar initial obstacle data includes the longitudinal distance and lateral distance of the obstacle relative to the vehicle. The longitudinal distance represents the distance between the obstacle and the vehicle in the first direction, and the lateral distance represents the distance between the obstacle and the vehicle in the second direction. The first direction corresponds to the forward direction of the vehicle, and the second direction is perpendicular to the first direction.

[0121] Step S4, performing calculation based on the lane line data and the longitudinal distance to obtain the obstacle lane width;

[0122] Step S6, performing calculation based on the lane line data, the longitudinal distance and the lateral distance to obtain a lateral offset distance.

[0123] Step S8: if the ratio between the lateral offset distance and the obstacle lane width is greater than or equal to a preset value, deleting the image initial obstacle data or the radar initial obstacle data;

[0124] Step S10: If the ratio between the lateral offset distance and the obstacle lane width is less than a preset value, the image initial obstacle data is used as the image target obstacle data, or the radar initial obstacle data is used as the radar target obstacle data.

[0125] Step S12: When the number of image target obstacle data is at least one and the number of radar target obstacle data is at least one, execute steps S122 to S1210, wherein:

[0126] Step S122: for each image target obstacle data, in the order of the longitudinal distance in the radar target obstacle data from small to large, it is determined in turn whether each radar target obstacle data and the image target obstacle data meet a preset matching condition, wherein the preset matching condition includes that the longitudinal distance difference, the lateral distance difference and the longitudinal speed difference between the image target obstacle data and the radar target obstacle data are all less than the corresponding preset difference thresholds;

[0127] Step S124, when the radar target obstacle data and the image target obstacle data meet a preset matching condition, a matching cost value of the radar target obstacle data relative to the image target obstacle data is obtained, and the radar target obstacle data with the smallest matching cost value is determined as the data to be fused;

[0128] Step S126, fusing the data to be fused with the image target obstacle data to obtain fused obstacle data corresponding to the image target obstacle data, wherein the data to be fused does not participate in the data fusion process of other image target obstacle data.

[0129] Optionally, the process of fusing the data to be fused with the image target obstacle data to obtain fused obstacle data corresponding to the image target obstacle data includes: determining the longitudinal distance, longitudinal velocity and longitudinal acceleration in the data to be fused as the longitudinal distance, longitudinal velocity and longitudinal acceleration in the fused obstacle data, and determining the lateral distance, lateral velocity and lateral acceleration in the image target obstacle data as the lateral distance, lateral velocity and lateral acceleration in the fused obstacle data to obtain the fused obstacle data.

[0130] Step S128: When all radar target obstacle data and image target obstacle data do not satisfy the preset matching condition, the image target obstacle data is used as the fused obstacle data corresponding to the image target obstacle data.

[0131] Step S14: when the number of image target obstacle data is 0 and the number of radar target obstacle data is not 0, the radar target obstacle data with the smallest lateral distance among the radar target obstacle data is determined as the fused obstacle data.

[0132] Step S16: when the amount of image target obstacle data is not 0 and the amount of radar target obstacle data is 0, the image target obstacle data is determined as fused obstacle data.

[0133] Exemplarily, the lane line parameter data acquired by the image sensor is shown in Table 1.

[0134] Table 1. Lane line parameter data example

[0135]

[0136] The number of initial obstacle data of the image obtained by the image sensor is 1, as shown in Table 2.

[0137] Table 2. Example of initial obstacle data in an image

[0138]

[0139] The number of radar initial obstacle data obtained by the radar sensor is 3, as shown in Table 3.

[0140] Table 3, Example of initial radar obstacle data

[0141]

[0142] Based on Formula 5 and Formula 2, the lateral offset distance and obstacle lane width of the initial obstacle data of the image with ID 49 are determined respectively. Among them,

[0143] ;

[0144] ;

[0145] ;

[0146] ;

[0147] ;

[0148] ;

[0149] ;

[0150] ;

[0151] ;

[0152] =4.21;

[0153] in, 2*Width, the initial obstacle data of the image meets the screening conditions, and the initial obstacle data of the image is used as the target obstacle data of the image.

[0154] The calculation process of the lateral offset distance and obstacle lane width of the above three radar initial obstacle data is similar to that of the image initial obstacle data, which will not be repeated here. The specific values ​​obtained are shown in Table 4.

[0155] Table 4. Lateral offset distance and obstacle lane width corresponding to the initial obstacle data of three radars

[0156]

[0157] Combined with the data in Table 4, the two radar initial obstacle data with ID 22 and ID 18 are used as radar target obstacle data to participate in the subsequent data matching and fusion processing process.

[0158] Among them, for the image target obstacle data with ID 49, the longitudinal distance difference threshold, lateral distance difference threshold and longitudinal speed difference threshold are set as PosX_gate=15m, PosY_gate=1.5m, and VelX_gate=3m / s respectively;

[0159] For the radar target obstacle data with ID 22, |deltX| = 2.8, |deltY| = 0.3253, |deltVelX| = 0.6 are calculated, which meets the preset matching conditions. The matching cost of the radar target obstacle data to the image target obstacle data is calculated as cost = |deltX| 2 +|deltY|2 +|deltVelX| 2 =8.3058.

[0160] For the radar target obstacle data with ID 18, |deltX| = 123.5, |deltY| = 1.2145, |deltVelX| = -0.33 are calculated, which does not meet the preset matching conditions and does not need to calculate the cost value.

[0161] Therefore, the radar target obstacle data with ID 22 and the image target obstacle data with ID 49 are fused, and the obtained fused obstacle data is shown in Table 5.

[0162] Table 5. Example of fused obstacle data

[0163]

[0164] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0165] It is understood that the term "based on" used in this application is used to describe one or more factors that affect the determination, and does not exclude other factors that may affect the determination. For example, the phrase "determine A based on B" means that the determination of A can be based entirely or at least partially on factor B, that is, B is a factor that affects the determination of A, but does not exclude that the determination of A is also based on C.

[0166] Based on the same inventive concept, the embodiment of the present application also provides a multi-sensor data fusion device for implementing the multi-sensor data fusion method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more multi-sensor data fusion device embodiments provided below can refer to the limitations of the multi-sensor data fusion method above, and will not be repeated here.

[0167] In an exemplary embodiment, Figure 7As shown, a multi-sensor data fusion device is provided, including: an acquisition module 702, a screening module 704 and a fusion module 706, wherein:

[0168] An acquisition module 702 is used to acquire lane line data and image initial obstacle data corresponding to the vehicle through an image sensor, and acquire radar initial obstacle data corresponding to the vehicle through a radar sensor;

[0169] The screening module 704 is used to screen the image initial obstacle data and the radar initial obstacle data based on the lane line data and preset screening conditions to obtain image target obstacle data and radar target obstacle data. The preset screening conditions include that the ratio between the lateral offset distance and the obstacle lane width is less than a preset value, the lateral offset distance is the shortest distance between the obstacle and the center trajectory line corresponding to the vehicle, and the obstacle lane width is the lane width corresponding to the position of the obstacle;

[0170] The fusion module 706 is used to perform data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data; the fused obstacle data is used for the vehicle to perform assisted driving.

[0171] In an exemplary embodiment, the image initial obstacle data includes the longitudinal distance and lateral distance of the obstacle relative to the vehicle, the longitudinal distance represents the distance between the obstacle and the vehicle in a first direction, and the lateral distance represents the distance between the obstacle and the vehicle in a second direction, the first direction corresponds to the forward direction of the vehicle, and the second direction is perpendicular to the first direction; the screening module 704 is used to perform calculation processing based on the lane line data and the longitudinal distance to obtain the obstacle lane width, and perform calculation processing based on the lane line data, the longitudinal distance and the lateral distance to obtain the lateral offset distance. If the ratio between the lateral offset distance and the obstacle lane width is greater than or equal to a preset value, the image initial obstacle data is deleted; if the ratio between the lateral offset distance and the obstacle lane width is less than the preset value, the image initial obstacle data is used as the image target obstacle data.

[0172] In an exemplary embodiment, the number of image target obstacle data is at least one, and the number of radar target obstacle data is at least one; the fusion module 706 is used to determine, for each image target obstacle data, in the order of the longitudinal distance in the radar target obstacle data from small to large, whether each radar target obstacle data and the image target obstacle data meet a preset matching condition, wherein the preset matching condition includes that the longitudinal distance difference, the lateral distance difference and the longitudinal speed difference between the image target obstacle data and the radar target obstacle data are all less than the corresponding preset difference threshold; when the preset matching condition is met between the radar target obstacle data and the image target obstacle data, a matching cost value of the radar target obstacle data relative to the image target obstacle data is obtained, and the radar target obstacle data with the smallest matching cost value is determined as the data to be fused; the data to be fused is fused with the image target obstacle data to obtain fused obstacle data corresponding to the image target obstacle data, wherein the data to be fused does not participate in the data fusion processing process of other image target obstacle data.

[0173] In an exemplary embodiment, the fusion module 706 is used to determine the longitudinal distance, longitudinal velocity and longitudinal acceleration in the data to be fused as the longitudinal distance, longitudinal velocity and longitudinal acceleration in the fused obstacle data, and determine the lateral distance, lateral velocity and lateral acceleration in the image target obstacle data as the lateral distance, lateral velocity and lateral acceleration in the fused obstacle data, so as to obtain the fused obstacle data.

[0174] In an exemplary embodiment, the fusion module 706 is configured to use the image target obstacle data as the fused obstacle data corresponding to the image target obstacle data when all radar target obstacle data and the image target obstacle data do not satisfy a preset matching condition.

[0175] In an exemplary embodiment, the number of image target obstacle data is 0, and the number of radar target obstacle data is not 0, and the fusion module 706 is used to determine the radar target obstacle data with the smallest lateral distance among the radar target obstacle data as the fused obstacle data.

[0176] In an exemplary embodiment, the amount of image target obstacle data is not 0, the amount of radar target obstacle data is 0, and the fusion module 706 is configured to determine the image target obstacle data as fused obstacle data.

[0177] Each module in the multi-sensor data fusion device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0178] In an exemplary embodiment, a computer device is provided, which may be a vehicle-mounted device, and its internal structure diagram may be shown in FIG8. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data that needs to be stored and called when executing a multi-sensor data fusion method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-sensor data fusion method is implemented.

[0179] Those skilled in the art will understand that the structure shown in FIG. 8 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0180] In one embodiment, a vehicle-mounted device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0181] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0182] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0184] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0185] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0186] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-sensor data fusion method, characterized in that: The method comprises: The lane line data and initial image obstacle data corresponding to the vehicle are obtained through the image sensor, and the initial radar obstacle data corresponding to the vehicle are obtained through the radar sensor; The image initial obstacle data and the radar initial obstacle data are screened and processed based on the lane line data and preset screening conditions to obtain image target obstacle data and radar target obstacle data, wherein the preset screening conditions include that a ratio between a lateral offset distance and an obstacle lane width is less than a preset value, the lateral offset distance is the shortest distance between an obstacle and a center trajectory line corresponding to the vehicle, and the obstacle lane width is a lane width corresponding to a position where the obstacle is located; Data fusion processing is performed based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data; the fused obstacle data is used for auxiliary driving of the vehicle.

2. The method according to claim 1, characterized in that The image initial obstacle data includes a longitudinal distance and a lateral distance of the obstacle relative to the vehicle, wherein the longitudinal distance represents a distance between the obstacle and the vehicle in a first direction, and the lateral distance represents a distance between the obstacle and the vehicle in a second direction, wherein the first direction corresponds to a forward direction of the vehicle, and the second direction is perpendicular to the first direction; The process of filtering and processing the initial obstacle data of the image based on the lane line data and the preset filtering conditions to obtain the target obstacle data of the image includes: Calculating and processing based on the lane line data and the longitudinal distance to obtain the obstacle lane width; Performing calculation based on the lane line data, the longitudinal distance and the lateral distance to obtain the lateral offset distance; If the ratio between the lateral offset distance and the obstacle lane width is greater than or equal to the preset value, deleting the initial obstacle data of the image; If the ratio of the lateral offset distance to the obstacle lane width is less than the preset value, the image initial obstacle data is used as the image target obstacle data.

3. The method according to claim 1, characterized in that The number of the image target obstacle data is at least one, and the number of the radar target obstacle data is at least one; The performing data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data includes: For each of the image target obstacle data, in the order of the longitudinal distance in the radar target obstacle data from small to large, it is judged in turn whether each of the radar target obstacle data and the image target obstacle data meets a preset matching condition, wherein the preset matching condition includes that the longitudinal distance difference, the lateral distance difference, and the longitudinal speed difference between the image target obstacle data and the radar target obstacle data are all less than the corresponding preset difference thresholds; When the preset matching condition is satisfied between the radar target obstacle data and the image target obstacle data, a matching cost value of the radar target obstacle data relative to the image target obstacle data is obtained, and the radar target obstacle data with the smallest matching cost value is determined as the data to be fused; The data to be fused is fused with the image target obstacle data to obtain fused obstacle data corresponding to the image target obstacle data, wherein the data to be fused does not participate in the data fusion processing process of other image target obstacle data.

4. The method according to claim 3, characterized in that The fusing the data to be fused with the image target obstacle data to obtain fused obstacle data corresponding to the image target obstacle data includes: The longitudinal distance, longitudinal velocity and longitudinal acceleration in the data to be fused are determined as the longitudinal distance, longitudinal velocity and longitudinal acceleration in the fused obstacle data, and the lateral distance, lateral velocity and lateral acceleration in the image target obstacle data are determined as the lateral distance, lateral velocity and lateral acceleration in the fused obstacle data, so as to obtain the fused obstacle data.

5. The method according to claim 3, characterized in that: The process of performing data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data also includes: In a case where all of the radar target obstacle data and the image target obstacle data do not satisfy the preset matching condition, the image target obstacle data is used as fused obstacle data corresponding to the image target obstacle data.

6. The method according to claim 1, characterized in that The amount of the image target obstacle data is 0, the amount of the radar target obstacle data is not 0, and the performing of data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data includes: The radar target obstacle data with the smallest lateral distance among the radar target obstacle data are determined as the fused obstacle data.

7. The method according to claim 1, characterized in that The amount of the image target obstacle data is not 0, the amount of the radar target obstacle data is 0, and the performing of data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data includes: The image target obstacle data is determined as the fused obstacle data.

8. A multi-sensor data fusion device, characterized in that: The device comprises: An acquisition module is used to acquire lane line data and image initial obstacle data corresponding to the vehicle through an image sensor, and acquire radar initial obstacle data corresponding to the vehicle through a radar sensor; a screening module, configured to screen the image initial obstacle data and the radar initial obstacle data based on the lane line data and preset screening conditions to obtain image target obstacle data and radar target obstacle data, wherein the preset screening conditions include that a ratio between a lateral offset distance and an obstacle lane width is less than a preset value, the lateral offset distance being the shortest distance between an obstacle and a center trajectory line corresponding to the vehicle, and the obstacle lane width being a lane width corresponding to a position where the obstacle is located; A fusion module is used to perform data fusion processing based on the image target obstacle data and the radar target obstacle data to obtain fused obstacle data; the fused obstacle data is used for the vehicle to perform assisted driving.

9. A vehicle-mounted device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Vehicle environment sensing data screening method, device and equipment and storage medium

    CN120541261A