A method and device for hierarchical fusion of vehicle-road cooperative perception data

Abstract

The present invention proposes a hierarchical fusion method for vehicle-road cooperative perception data, including: obtaining the original perception data on the vehicle side collected by the first sensor on the vehicle, and preprocessing it into the first data in a preset format, obtaining the vehicle-side perception trajectory parameters according to the first data, fusing them to obtain a first-level fusion result, obtaining the original perception data on the road side collected by the second sensor deployed on the road side, and preprocessing it into the second data in a preset format, obtaining the road side perception trajectory parameters according to the second data, fusing them to obtain a road side fusion result, and fusing the first-level fusion result and the road side fusion result to obtain a second-level fusion result, obtaining multiple second-level fusion results for fusion to obtain a third-level fusion result, sending the third-level fusion result to the vehicle, and the vehicle uses the third fusion result as the final perception result. This method solves the problem of asynchronous fusion of multi-source data on the vehicle and road, expands the vehicle's perception range, and outputs high-precision perception results.

Description

Technical Field

[0001] The present invention relates to the field of vehicle-road collaboration and automobile automatic driving technology, and in particular to a method and device for hierarchical fusion of vehicle-road collaboration perception data. Background Art

[0002] In recent years, the autonomous driving technology that has received widespread attention uses sensors installed on vehicles to perceive the surrounding environment of vehicles. However, relying solely on the limited sensors installed on autonomous vehicles to detect the surrounding environment of vehicles is prone to blind spots, occlusions and other problems. The perception range of a single vehicle is limited and cannot meet the adaptability requirements of emergencies. It is necessary to use roadside perception equipment and rely on the communication between the vehicle side and the network side to build a vehicle-road cooperative network for perception fusion. In the vehicle-road cooperative scenario, the roadside perception equipment can perceive the surrounding environment of the vehicle, thereby providing some blind spot environmental information for the autonomous driving vehicle, expanding the vehicle's perception range and improving perception performance. Due to the characteristics of the vehicle-road cooperative scenario with a large number of nodes, high-speed vehicle movement, and highly dynamic changes in the environment, it is difficult for the autonomous driving vehicle and the roadside perception equipment to keep the perception frequency and update time consistent, and it is easy for a data source to fail. The problem of asynchronous output of multi-source information is very serious. The existing fusion method is only for a single vehicle system, and it is difficult to handle newly added or newly lost data source information. The data source information is not fully utilized and there are large errors. Summary of the invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0004] To this end, the first purpose of the present invention is to propose a method and device for hierarchical fusion of vehicle-road collaborative perception data to solve the problem of asynchronous fusion of multi-source vehicle-road data in vehicle-road collaborative scenarios, expand the vehicle perception range, output high-precision perception results, and improve perception performance.

[0005] The second object of the present invention is to provide a device.

[0006] A third object of the present invention is to provide a non-transitory computer-readable storage medium.

[0007] A fourth object of the present invention is to provide a computer program product.

[0008] To achieve the above object, a first embodiment of the present invention provides a method, comprising:

[0009] Acquire vehicle-side raw perception data collected by a first sensor installed on the vehicle, preprocess the vehicle-side raw perception data into first data in a preset format, and obtain vehicle-side perception trajectory parameters according to the first data in the preset format;

[0010] Performing fusion processing on the vehicle side perception trajectory parameters to obtain a first-level fusion result;

[0011] Acquire roadside raw perception data collected by a second sensor deployed on the roadside, preprocess the roadside raw perception data into second data in a preset format, and obtain roadside perception trajectory parameters according to the second data in the preset format;

[0012] Performing fusion processing on the roadside perception trajectory parameters to obtain a roadside fusion result, and fusing the primary fusion result with the roadside fusion result to obtain a secondary fusion result;

[0013] Acquire secondary fusion results from multiple roadside units, fuse the secondary fusion results from the multiple roadside units to obtain a third-level fusion result, and send the third-level fusion result to the autonomous driving vehicle, wherein the autonomous driving vehicle uses the third fusion result as the final perception result.

[0014] Optionally, in an embodiment of the present invention, the first data in the preset format includes data in a point cloud format and / or data in a pixel map format. When the first data in the preset format is data in the point cloud format and / or pixel map format, obtaining the vehicle side perception trajectory parameter according to the first data in the preset format includes:

[0015] Extracting obstacle 3D bounding boxes from the point cloud and / or image using point cloud detection and segmentation technology and / or image detection and segmentation technology;

[0016] Obtain the corresponding vehicle side perception trajectory parameters according to the 3D bounding box, where the vehicle perception trajectory parameters include: the center coordinates of the obstacle 3D bounding box The speed estimated from the center coordinates of the obstacle bounding box in two consecutive frames and

[0017] Optionally, in an embodiment of the present invention, the fusing process of the vehicle-side sensing trajectory parameters to obtain a first-level fusion result includes:

[0018] fusing the vehicle side perception trajectory parameters according to the first sensor;

[0019] Obtaining a nominal state based on a time interval period according to a kinematic equation, wherein the nominal state includes a predicted trajectory parameter at a next moment;

[0020] The state estimation covariance and Kalman gain are calculated to obtain the first-level fusion result.

[0021] Optionally, in one embodiment of the present invention, the roadside perception trajectory parameters are fused to obtain a roadside fusion result, and the primary fusion result and the roadside fusion result are fused to obtain a secondary fusion result, including:

[0022] Taking the time T0 when the first-level fusion result and the roadside fusion result are first received as the initial fusion time, the first-level fusion result and the roadside fusion result are fused;

[0023] And T c <T f <2T c , T0+nT f is the current fusion moment, where T f is the fusion period, T C is the upload period of the first-level fusion result to the roadside unit, n is a positive integer;

[0024] T0+T f The moment is only integrated at T0+(n-1)T f To T0+nT f The first-level fusion results and roadside fusion results received during the period.

[0025] Optionally, in one embodiment of the present invention, multi-source data fusion is performed on the primary fusion result and the roadside fusion result to obtain a secondary fusion result, including:

[0026] The fusion process is used as a function node and multiple data sources are used as variable nodes. If the fusion function contains the variable, the corresponding nodes of the two are connected with directed edges to build a fusion model.

[0027] If the message transmission between the function node and the variable node satisfies the Gaussian distribution, all messages are calculated according to the message transmission rule and fusion order, and the final fusion result at the x coordinate at time t can be obtained:

[0028]

[0029] Among them, taking the x-axis as an example, x t-1 and x t are the fusion output values ​​at time t-1 and time t respectively, is the noise variance of multi-source data, The x-axis coordinates of the multi-source data input. and They are The noise variance and measurement value of ; and, the y coordinate y at time t can be obtained t , speed in the x-axis direction Speed ​​in the y-axis direction The final fusion result is:

[0030]

[0031]

[0032]

[0033] If only one first-level fusion result is received, the first-level fusion result is used as the second-level fusion result;

[0034] If multiple first-level fusion results are received, the first-level fusion results are fused and the fused result is used as the second-level fusion result.

[0035] Optionally, in an embodiment of the present invention, fusing the secondary fusion results from the multiple roadside units to obtain a third-level fusion result includes:

[0036] Taking the time T3 when the secondary fusion result is first received as the initial fusion time, fusing the secondary fusion results from the multiple roadside units;

[0037] The fusion period is T d (T d >T c ), and T3+nT d is the current fusion time, the multi-site data fusion includes: T3+nT d The moment is only integrated at T3+(n-1)T d To T3+nT d The secondary fusion results received during the period;

[0038] The secondary fusion results are fused to obtain the third-level fusion results, where the third-level fusion results include: the center coordinates of the obstacle 3D bounding box after multi-site data fusion And the speed of the obstacle in the x-axis and y-axis directions after fusion Its range is the area covered by one or more roadside units.

[0039] To achieve the above-mentioned purpose, a second embodiment of the present invention proposes a vehicle-road cooperative perception data hierarchical fusion device, comprising:

[0040] A first acquisition module is used to acquire vehicle-side raw perception data collected by a first sensor installed on the vehicle, preprocess the vehicle-side raw perception data into first data in a preset format, and obtain vehicle-side perception trajectory parameters according to the first data in the preset format;

[0041] A first processing module is used to perform fusion processing on the vehicle side perception trajectory parameters to obtain a first-level fusion result;

[0042] A second acquisition module is used to acquire roadside raw perception data collected by a second sensor deployed on the roadside, preprocess the roadside raw perception data into second data in a preset format, and obtain roadside perception trajectory parameters according to the second data in the preset format;

[0043] A second processing module is used to perform fusion processing on the roadside perception trajectory parameters to obtain a roadside fusion result, and to fuse the primary fusion result with the roadside fusion result to obtain a secondary fusion result;

[0044] The third processing module is used to obtain the second-level fusion results from multiple roadside units, and fuse the second-level fusion results from the multiple roadside units to obtain a third-level fusion result, and send the third-level fusion result to the autonomous driving vehicle, wherein the autonomous driving vehicle uses the third fusion result as the final perception result.

[0045] Optionally, in one embodiment of the present invention, the first acquisition module is further used to:

[0046] Extracting obstacle 3D bounding boxes from the point cloud and / or image using point cloud detection and segmentation technology and / or image detection and segmentation technology;

[0047] Obtain the corresponding vehicle side perception trajectory parameters according to the 3D bounding box, where the vehicle perception trajectory parameters include: the center coordinates of the obstacle 3D bounding box The speed estimated from the center coordinates of the obstacle bounding box in two consecutive frames and

[0048] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the small sample image incremental classification method based on embedded enhancement and adaptation described in the first aspect of the present invention is implemented.

[0049] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the small sample image incremental classification method based on embedded enhancement and adaptation described in the first aspect of the present invention.

[0050] In summary, the present invention proposes a method and device for hierarchical fusion of vehicle-road collaborative perception data, so that autonomous driving vehicles can obtain more environmental information of blind spots. In the vehicle-road collaborative scenario, it can solve the problem of asynchronous output of multi-source data, expand the vehicle's perception range, output high-precision perception results, and improve perception performance.

[0051] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0053] Figure 1 A schematic diagram of a process flow of a method for hierarchical fusion of vehicle-road cooperative perception data provided by an embodiment of the present invention;

[0054] Figure 2 A multi-source data fusion model diagram provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of the structure of a vehicle-road cooperative perception data hierarchical fusion device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0057] A method and device for hierarchical fusion of vehicle-road cooperative perception data according to an embodiment of the present invention will be described below with reference to the accompanying drawings.

[0058] Figure 1 A flowchart of a method for hierarchical fusion of vehicle-road cooperative perception data provided by an embodiment of the present invention.

[0059] like Figure 1 As shown, the vehicle-road cooperative perception data hierarchical fusion method includes the following steps:

[0060] Step S10: obtaining vehicle-side raw perception data collected by a first sensor installed on the vehicle, preprocessing the vehicle-side raw perception data into first data in a preset format, and obtaining vehicle-side perception trajectory parameters according to the first data in the preset format.

[0061] Among them, in one embodiment of the present invention, obtaining the vehicle-side raw perception data collected by a first sensor installed on the vehicle includes:

[0062] Extracting obstacle 3D bounding boxes from the point cloud and / or image using point cloud detection and segmentation technology and / or image detection and segmentation technology;

[0063] Obtain the corresponding vehicle side perception trajectory parameters according to the 3D bounding box, where the vehicle side perception trajectory parameters include: the center coordinates of the obstacle 3D bounding box The speed estimated from the center coordinates of the obstacle bounding box in two consecutive frames and

[0064] Specifically, in one embodiment of the present invention, original perception data on the vehicle side collected by multiple first sensors (such as millimeter wave radars, laser radars, and cameras) installed on the autonomous driving vehicle are obtained, and a perception data preprocessing module deployed on the autonomous driving vehicle performs data preprocessing on the original perception data to obtain data in a point cloud or pixel map format. The time required for the data preprocessing process is assumed to be T1, wherein the data preprocessing is: converting the original perception data on the vehicle side into a point cloud format, including:

[0065] If the original perception data on the vehicle side is in bin format or txt format, it is converted to ply point cloud format;

[0066] If the original perception data on the vehicle side is in ply point cloud format, this format is retained;

[0067] If the original perception data on the vehicle side is in a two-dimensional image format, it is converted into pixel image format data with depth information.

[0068] Among them, the original perception data of the vehicle side is converted into a two-dimensional image format into a pixel image format with depth information using binocular vision technology, including:

[0069] According to the conversion relationship from the world coordinate system to the pixel coordinate system, the camera internal and external parameters are obtained by using the image-based single-target positioning technology. The conversion relationship from the world coordinate system to the pixel coordinate system is as follows:

[0070]

[0071] Among them, u and v are the coordinates in the camera pixel coordinate system, and f x 、f y , u0, v0 are the camera internal parameters, R, T are the camera external parameters, that is, the rotation matrix and translation matrix, X w , Y w , Z w is the coordinate in the world coordinate system, Z c is the scale factor.

[0072] Based on the single-target positioning technology, the intrinsic and extrinsic parameters are obtained. The dual-target positioning technology is used for the camera, and the relative position relationship of the left and right cameras is obtained through the following formula:

[0073]

[0074] Among them, R is the rotation matrix of the right camera relative to the left camera, T is the translation matrix of the right camera relative to the left camera, and R r and R l are the rotation matrices of the right camera and the left camera respectively, T r and T l are the translation matrices of the right camera and the left camera respectively, R r , R l 、T r 、T l All can be obtained by camera single-target positioning technology;

[0075] According to the parameter data obtained after calibration and the relative position of the binoculars, the left and right views are dedistorted and aligned respectively, so that the imaging origin coordinates of the left and right views are consistent.

[0076] Match the corresponding image points on the left and right views to obtain disparity data, and calculate the depth information using the following formula:

[0077]

[0078] Where f is the focal length, B is the camera center distance, X R -X T For parallax.

[0079] Furthermore, in an embodiment of the present invention, obtaining the vehicle side perception trajectory parameter according to the first data in a preset format includes:

[0080] The perception data preprocessing module can extract obstacle information (such as autonomous driving vehicles, pedestrians, trees, etc.) from the preprocessed point cloud image and pixel image, obtain the obstacle 3D bounding box and its corresponding vehicle-side perception trajectory parameters, and transmit the perception trajectory parameters to the multi-sensor fusion module through the physical interface. The perception trajectory parameters include: the center coordinates of the obstacle 3D bounding box The speed of the obstacle in the x-axis and y-axis directions

[0081] Step S20: performing fusion processing on the vehicle side perception trajectory parameters to obtain a first-level fusion result.

[0082] In one embodiment of the present invention, the fusion process may include:

[0083] When the first sensor is only equipped with a camera on the vehicle side, the input trajectory parameters are fused with the time interval Δt1 between two consecutive times (such as t-Δt1 and t) to predict the trajectory parameters at time t using the trajectory parameters perceived by the pixel map at time t-Δt1 according to the kinematic equation. and take it as a nominal state;

[0084] Compute the state estimate covariance and Kalman gain Among them, F is the state transfer matrix, Q is the system noise covariance, is the state covariance at time t-Δt1, H is the observation matrix, R camera is the camera noise variance;

[0085] The camera measurement status at time t Correction of nominal state as error state Get the trajectory fusion result at time t, and use it as the first-level fusion result:

[0086]

[0087] When the first sensor is equipped with a laser radar and / or a millimeter-wave radar on the vehicle side, the input trajectory parameters are fused with the time interval Δt2 between two consecutive times (such as t-Δt2, t) when the point cloud image is received to perceive the trajectory parameters: the trajectory parameters at time t are predicted using the trajectory parameters obtained from the point cloud image at time t-Δt2 according to the kinematic equation. And use it as the nominal state. Calculate the state estimate covariance and Kalman gain Among them, F is the state transfer matrix, Q is the system noise covariance, is the state covariance at time t-Δt2, H is the observation matrix, R lidar is the noise variance of the laser radar or millimeter wave radar, and the point cloud measurement state at time t Correction of nominal state as error state Get the trajectory fusion result at time t, and use it as the first-level fusion result:

[0088]

[0089] When the first sensor is equipped with a camera, a laser radar and / or a millimeter-wave radar on the vehicle side, the input trajectory parameters are fused with the time interval Δt1 between two consecutive times (such as t-Δt1, t) when the pixel map is received to perceive the trajectory parameters: the trajectory parameters at time t are predicted using the trajectory parameters obtained from the pixel map at time t-Δt1 according to the kinematic equation. and take it as a nominal state;

[0090] Compute the state estimate covariance and Kalman gain Among them, F is the state transfer matrix, Q is the system noise covariance, is the state covariance at time t-Δt2, H is the observation matrix, R lidaris the noise variance of the laser radar or millimeter wave radar. The point cloud image measurement state at time t

[0091]

[0092] Correction of nominal state as error state Get the trajectory fusion result at time t, and use it as the first-level fusion result:

[0093]

[0094] It should be explained that in the embodiment of the present invention, based on the above calculation formula, the trajectory parameters obtained from the point cloud image at time t-Δt1 are used to calculate the predicted trajectory parameters at time t, and the uncertainty (such as covariance) and gain coefficient of the prediction process are calculated. If the trajectory parameters perceived by the point cloud image at time t have been received, the predicted trajectory parameters at time t are corrected using the trajectory parameters perceived by the point cloud image at time t, and the corrected trajectory parameters at time t are the first-level fusion results; if the trajectory parameters perceived by the point cloud image at time t have not been received, the predicted trajectory parameters at time t are corrected using the trajectory parameters perceived by the pixel image at time t, and the corrected trajectory parameters at time t are the first-level fusion results, the uncertainty is updated, and the first-level fusion results are transmitted to the roadside unit through the physical interface.

[0095] Step S30: obtaining original roadside perception data collected by a second sensor deployed on the roadside, preprocessing the original roadside perception data into second data in a preset format, and obtaining roadside perception trajectory parameters according to the second data in the preset format.

[0096] In one embodiment of the present invention, this step is applied to a roadside unit.

[0097] And, in one embodiment of the present invention, a second sensor on the road side (such as a roadside camera) is used every T r Time (T c =nT r ) Periodically transmit the roadside raw perception data and the three-dimensional coordinates of the roadside perception equipment to the roadside unit via wire.

[0098] The first-level fusion result is obtained and transmitted to the roadside unit through the physical interface, and the original roadside perception data and the three-dimensional coordinates of the roadside perception device are transmitted to the roadside unit through the physical interface.

[0099] Regarding the relevant instructions and explanations on obtaining the roadside perception trajectory parameters according to the second data in a preset format, please refer to the above-mentioned relevant instructions and explanations on obtaining the vehicle-side perception trajectory parameters according to the first data in a preset format. To avoid repetition, they will not be repeated here.

[0100] Step S40: fusing the roadside perception trajectory parameters to obtain a roadside fusion result, and fusing the primary fusion result and the roadside fusion result to obtain a secondary fusion result.

[0101] In one embodiment of the present invention, the data preprocessing includes: converting the original roadside sensing data of the laser radar and the millimeter wave radar into a point cloud format:

[0102] If the original roadside perception data is in bin format or txt format, it is converted into ply point cloud format;

[0103] If the original roadside perception data is in ply point cloud format, this format is retained;

[0104] If the original roadside perception data is in a two-dimensional image format, it is converted into pixel image format data with depth information.

[0105] And, in one embodiment of the present invention, the method of converting the original roadside perception data in a two-dimensional picture format into pixel image format data with depth information may include:

[0106] The camera's single-target and binocular-target positioning technologies are used to obtain the camera's intrinsic parameters, extrinsic parameters, and relative position relationship. According to the parameter data obtained after calibration and the binocular relative position, the left and right views are dedistorted and aligned respectively to make the imaging origin coordinates of the left and right views consistent;

[0107] Match the corresponding image points on the left and right views to obtain disparity data, and calculate the depth information using the depth information calculation formula;

[0108] The obstacle depth information and obstacle 3D bounding box are extracted from the point cloud image and pixel image. The obstacle depth information is the distance d between the obstacle and the roadside sensing device. Then, the horizontal distance l between the obstacle and the roadside sensing device is obtained according to the three-dimensional coordinates of the roadside sensing device. The calculation formula is as follows:

[0109]

[0110] Among them, z road is the height of the roadside sensing device from the ground, and then the roadside sensing trajectory parameters corresponding to the obstacle are obtained based on the distance conversion method between the coordinate points, and the roadside sensing trajectory parameters are transmitted to the roadside unit through the physical interface. Among them, the road test sensing trajectory parameters include: the center coordinates of the obstacle 3D bounding box The speed of the obstacle in the x-axis and y-axis directions

[0111] The roadside multi-sensor fusion module performs roadside multi-sensor fusion on the received roadside perception trajectory parameters to obtain a roadside fusion result, wherein the roadside fusion result includes: the center coordinates of the 3D boundary box of the obstacle after the roadside multi-sensor fusion The speed of the obstacle in the x-axis and y-axis directions after the fusion of multiple roadside sensors

[0112] The steps of roadside multi-sensor fusion can refer to the description and explanation of the above step S20, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0113] Specifically, in one embodiment of the present invention, the time T0 when the primary fusion result and the roadside fusion result are first received is taken as the initial fusion time, and the primary fusion result and the roadside fusion result are fused, and the fusion period is set as T f , and T c <T f <2T c , let T0+nT f is the current fusion moment, which includes:

[0114] T0+T f The moment is only integrated at T0+(n-1)T f To T0+nT f The first-level fusion results and roadside fusion results received during the period, where T f is the fusion period, T C is the upload period of the first-level fusion result to the roadside unit, and n is a positive integer.

[0115] Furthermore, in one embodiment of the present invention, a method for performing multi-source data fusion on the primary fusion result and the roadside fusion result to obtain a secondary fusion result may include:

[0116] The fusion process is used as a function node and multiple data sources are used as variable nodes. If the fusion function contains the variable, the corresponding nodes of the two are connected with directed edges to build a multivariate data fusion model.

[0117] And, in one embodiment of the present disclosure, Figure 2 A multi-source data fusion model diagram provided by an embodiment of the present invention; Figure 2 As shown in the figure, if the message transmission between the function node and the variable node satisfies the Gaussian distribution, all messages are calculated according to the message transmission rule and the fusion order, and the final fusion result at the x coordinate at time t can be obtained:

[0118]

[0119] Among them, taking the x-axis as an example, x t-1 and xt are the fusion output values ​​at time t-1 and time t, respectively, w and f are fusion function nodes, and the data from the multiple data sources are fused into the function node as variable nodes. is the noise variance of multi-source data, The x-axis coordinates of the multi-source data input. and They are The noise variance and measurement value of

[0120] And, we can get the y coordinate y at time t t , speed in the x-axis direction Speed ​​in the y-axis direction The final fusion result is:

[0121]

[0122]

[0123]

[0124] If only one first-level fusion result is received, the first-level fusion result is used as the second-level fusion result;

[0125] If multiple first-level fusion results are received, the first-level fusion results are fused and the fused result is used as the second-level fusion result.

[0126] It should be explained that in one embodiment of the present invention, if the roadside unit receives multiple roadside fusion results, the latest roadside fusion result is selected for fusion; if the roadside unit only receives the first-level fusion result from the vehicle; if only the first-level fusion result of one vehicle is received, the first-level fusion result is transmitted to the cloud server through the physical interface as the second-level fusion result; if the first-level fusion results of more than one vehicle are received, the first-level fusion results are fused, and the fused result is transmitted to the cloud server through the physical interface as the second-level fusion result; if at the next fusion time (i.e., T0+(n+1)T f When a fusion result from a new data source is received at the time (time), a new variable node is added to the model; if a data source is unavailable (for example, the data source is damaged or is no longer covered by the roadside unit) or the first-level fusion result or roadside fusion result is not received from the data source, the relevant variable node is deleted.

[0127] Step S50: Obtain the secondary fusion results from multiple roadside units, and fuse the secondary fusion results from multiple roadside units to obtain a third-level fusion result, and send the third-level fusion result to the autonomous driving vehicle, wherein the autonomous driving vehicle uses the third fusion result as the final perception result.

[0128] In one embodiment of the present invention, this step is applied to the cloud server, which receives the secondary fusion result uploaded by the roadside unit and transmits the secondary fusion result to the multi-site data fusion module through the physical interface;

[0129] And, in one embodiment of the present invention, the time T3 when the secondary fusion result is first received is taken as the initial fusion time, and the received secondary fusion result is subjected to multi-site data fusion, and the multi-site data fusion period is set to T d (T d >T c ), let T3+nT d is the current fusion moment, which includes:

[0130] T3+nT d The moment is only integrated at T3+(n-1)T d To T3+nT d The second-level fusion results received during the period; the fusion process is taken as the function node, and the different roadside units are taken as the variable nodes. If the fusion function contains the variable, the corresponding nodes of the two are connected with directed edges to build a fusion model. The second-level fusion results are taken as the messages transmitted between the variable nodes and the function nodes. According to the message transmission rules and fusion order, all messages sent from the starting node to the ending node are calculated. The calculation result is the third-level fusion result, and the third-level fusion result is sent to the autonomous driving vehicle through the physical interface.

[0131] Also, in one embodiment of the present invention, if the cloud server receives m secondary fusion results from the same roadside unit, the mth secondary fusion result is selected for fusion, where m is a positive integer greater than 1; if the cloud server only receives the secondary fusion result from one roadside unit, the secondary fusion result is directly sent to the autonomous driving vehicle through the physical interface as a third-level fusion result, and the autonomous driving vehicle uses the third-level fusion result as the final perception result.

[0132] In summary, the present invention proposes a method for hierarchical fusion of vehicle-road collaborative perception data, so that autonomous driving vehicles can obtain more environmental information of blind spots, expand the vehicle's perception range in vehicle-road collaborative scenarios, output high-precision perception results, and improve perception performance.

[0133] Figure 3 A schematic diagram of the structure of a vehicle-road cooperative perception data hierarchical fusion device provided in an embodiment of the present invention.

[0134] like Figure 3 As shown, the vehicle-road cooperative perception data hierarchical fusion device includes the following modules:

[0135] A first acquisition module is used to acquire vehicle-side raw perception data collected by a first sensor installed on the vehicle, preprocess the vehicle-side raw perception data into first data in a preset format, and obtain vehicle-side perception trajectory parameters according to the first data in the preset format;

[0136] A first processing module is used to perform fusion processing on the vehicle side perception trajectory parameters to obtain a first-level fusion result;

[0137] A second acquisition module is used to acquire roadside raw perception data collected by a second sensor deployed on the roadside, preprocess the roadside raw perception data into second data in a preset format, and obtain roadside perception trajectory parameters according to the second data in the preset format;

[0138] A second processing module is used to perform fusion processing on the roadside perception trajectory parameters to obtain a roadside fusion result, and to fuse the primary fusion result with the roadside fusion result to obtain a secondary fusion result;

[0139] The third processing module is used to obtain the second-level fusion results from multiple roadside units, and fuse the second-level fusion results from the multiple roadside units to obtain a third-level fusion result, and send the third-level fusion result to the autonomous driving vehicle, wherein the autonomous driving vehicle uses the third fusion result as the final perception result.

[0140] In one embodiment of the present disclosure, the first acquisition module is further used to:

[0141] Extracting obstacle 3D bounding boxes from the point cloud and / or image using point cloud detection and segmentation technology and / or image detection and segmentation technology;

[0142] Obtain the corresponding vehicle side perception trajectory parameters according to the 3D bounding box, where the vehicle perception trajectory parameters include: the center coordinates of the obstacle 3D bounding box The speed estimated from the center coordinates of the obstacle bounding box in two consecutive frames and

[0143] To sum up, the present invention proposes a device for hierarchical fusion of vehicle-road collaborative perception data, so that autonomous driving vehicles can obtain more environmental information of blind spots. In the vehicle-road collaborative scenario, it can solve the problem of asynchronous output of multi-source data, expand the vehicle's perception range, output high-precision perception results, and improve perception performance.

[0144] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0145] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0146] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.

[0147] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0148] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0149] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0150] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0151] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for hierarchical fusion of vehicle-road cooperative perception data, characterized in that: The method comprises the following steps: Acquire vehicle-side raw perception data collected by a first sensor installed on the vehicle, preprocess the vehicle-side raw perception data into first data in a preset format, and obtain vehicle-side perception trajectory parameters according to the first data in the preset format; Performing fusion processing on the vehicle side perception trajectory parameters to obtain a first-level fusion result; Acquire roadside raw perception data collected by a second sensor deployed on the roadside, preprocess the roadside raw perception data into second data in a preset format, and obtain roadside perception trajectory parameters according to the second data in the preset format; Performing fusion processing on the roadside perception trajectory parameters to obtain a roadside fusion result, and fusing the primary fusion result with the roadside fusion result to obtain a secondary fusion result; Acquire secondary fusion results from multiple roadside units, fuse the secondary fusion results from the multiple roadside units to obtain a third-level fusion result, and send the third-level fusion result to the autonomous driving vehicle, wherein the autonomous driving vehicle uses the third-level fusion result as the final perception result; The fusing process of the vehicle-side sensing trajectory parameters to obtain a first-level fusion result includes: fusing the vehicle side perception trajectory parameters according to the first sensor; Obtaining a nominal state based on a time interval period according to a kinematic equation, wherein the nominal state includes a predicted trajectory parameter at a next moment; Calculate the state estimation covariance and Kalman gain to obtain the first-level fusion result; The roadside perception trajectory parameters are fused to obtain a roadside fusion result, and the first-level fusion result and the roadside fusion result are fused to obtain a second-level fusion result: Taking the time T0 when the first-level fusion result and the roadside fusion result are first received as the initial fusion time, multi-source data fusion is performed on the first-level fusion result and the roadside fusion result; And T c <T f <2T c , T0+nT f is the current fusion moment, where T f is the fusion period, T C is the upload period of the first-level fusion result to the roadside unit, n is a positive integer; T0+T f The moment is only integrated at T0+(n-1)T f To T0+nT f The first-level fusion results and roadside fusion results received during the period; The multi-source data fusion includes: The fusion process is used as a function node and multiple data sources are used as variable nodes. If the fusion function contains the variable, the corresponding nodes of the two are connected with directed edges to build a fusion model. If the message transmission between the function node and the variable node satisfies the Gaussian distribution, all messages are calculated according to the message transmission rule and fusion order, and the final fusion result at the x coordinate at time t can be obtained: Among them, x t-1 and x t are the fusion output values ​​at time t-1 and time t respectively, is the noise variance of multi-source data, The x-axis coordinates of the multi-source data input. and They are The noise variance and measurement value of And, the final fusion result at the y coordinate at time t can be obtained: If only one first-level fusion result is received, the first-level fusion result is used as the second-level fusion result; If multiple first-level fusion results are received, the first-level fusion results are fused and the fused result is used as the second-level fusion result; The step of fusing the secondary fusion results from the multiple roadside units to obtain a third-level fusion result includes: Taking the time T3 when the secondary fusion result is first received as the initial fusion time, fusing the secondary fusion results from the multiple roadside units; T3+nT d The moment is only integrated at T3+(n-1)T d To T3+nT d The secondary fusion results received during the period, where T d is the fusion cycle, and T3+nT d is the current fusion moment, T d >T c ; The secondary fusion results are fused to obtain the third-level fusion results, where the third-level fusion results include: the center coordinates of the obstacle 3D bounding box after multi-site data fusion And the speed of the obstacle in the x-axis and y-axis directions after fusion Its range is the area covered by multiple roadside units.

2. The method according to claim 1, characterized in that The first data in the preset format includes data in a point cloud format and / or data in a pixel map format. When the first data in the preset format is data in the point cloud format and / or pixel map format, obtaining the vehicle side perception trajectory parameter according to the first data in the preset format includes: Extracting obstacle 3D bounding boxes from the point cloud and / or image using point cloud detection and segmentation technology and / or image detection and segmentation technology; Obtain the corresponding vehicle side perception trajectory parameters according to the 3D bounding box, where the vehicle side perception trajectory parameters include: the center coordinates of the obstacle 3D bounding box The speed estimated from the center coordinates of the obstacle bounding box in two consecutive frames and 3. A vehicle-road cooperative perception data hierarchical fusion device, characterized in that: The device is used to implement the vehicle-road cooperative perception data hierarchical fusion method according to claim 1, and the device includes: A first acquisition module is used to acquire vehicle-side raw perception data collected by a first sensor installed on the vehicle, preprocess the vehicle-side raw perception data into first data in a preset format, and obtain vehicle-side perception trajectory parameters according to the first data in the preset format; A first processing module is used to perform fusion processing on the vehicle side perception trajectory parameters to obtain a first-level fusion result; A second acquisition module is used to acquire roadside raw perception data collected by a second sensor deployed on the roadside, preprocess the roadside raw perception data into second data in a preset format, and obtain roadside perception trajectory parameters according to the second data in the preset format; A second processing module is used to perform fusion processing on the roadside perception trajectory parameters to obtain a roadside fusion result, and to fuse the primary fusion result with the roadside fusion result to obtain a secondary fusion result; The third processing module is used to obtain the second-level fusion results from multiple roadside units, and fuse the second-level fusion results from the multiple roadside units to obtain a third-level fusion result, and send the third-level fusion result to the autonomous driving vehicle, wherein the autonomous driving vehicle uses the third-level fusion result as the final perception result.

4. The device according to claim 3, characterized in that The first acquisition module is further used for: Extracting obstacle 3D bounding boxes from the point cloud and / or image using point cloud detection and segmentation technology and / or image detection and segmentation technology; Obtain the corresponding vehicle side perception trajectory parameters according to the 3D bounding box, where the vehicle side perception trajectory parameters include: the center coordinates of the obstacle 3D bounding box The speed estimated from the center coordinates of the obstacle bounding box in two consecutive frames and 5. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 2 is implemented.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images