Visual information fusion processing method, device and storage medium based on vehicle-road collaboration

Abstract

The embodiment of the present application discloses a method, device and storage medium for visual information fusion processing based on vehicle-road collaboration, the method comprising: obtaining first image data corresponding to the onboard unit of the target vehicle; determining second image data obtained by multiple roadside units corresponding to the first image data according to time-synchronized matching rules, forming a target image data set according to the first image data and the corresponding multiple second image data; fusing the target image data set based on image features corresponding to overlapping perception areas to obtain beyond-visual-range visual information corresponding to the target vehicle. The fusion of visual perception information of different road sections can be achieved, providing comprehensive beyond-visual-range visual information for the autonomous driving decision system, and improving the vehicle's intelligent driving capabilities.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving control, and specifically to a visual information fusion processing method, device and storage medium based on vehicle-road collaboration. Background Art

[0002] In intelligent driving control, it is often necessary to provide the intelligent driving decision system with environmental information around the vehicle in order to improve the vehicle's intelligent driving capabilities. In related technologies, the surrounding environmental information is often collected through a variety of sensors installed on the vehicle (such as lidar, cameras, etc.). However, due to the limitation of the sensor collection range, the collected environmental information around the vehicle is limited and cannot meet the needs of intelligent driving. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a visual information fusion processing method, device and storage medium based on vehicle-road collaboration, aiming to provide the vehicle with more comprehensive environmental information and improve the vehicle's intelligent driving capabilities.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a visual information fusion processing method based on vehicle-road collaboration, including:

[0006] Acquire first image data corresponding to the onboard unit of the target vehicle;

[0007] According to a time synchronization matching rule, second image data acquired by each of a plurality of roadside units corresponding to the first image data is determined, and a target image data set is formed according to the first image data and the corresponding plurality of second image data; wherein the roadside units are arranged at intervals along a lane direction and the perception areas of adjacent roadside units overlap;

[0008] The target image data set is subjected to information fusion based on image features corresponding to overlapping perception areas to obtain beyond-visual-range visual information corresponding to the target vehicle.

[0009] In a second aspect, an embodiment of the present application provides a visual information fusion processing method based on vehicle-road collaboration, which is characterized in that it is applied to an on-board unit of a vehicle to implement the visual information fusion processing method described in the aforementioned embodiment.

[0010] In a third aspect, an embodiment of the present application provides a visual information fusion processing device, comprising:

[0011] An acquisition module, used for acquiring first image data corresponding to the on-board unit of the target vehicle;

[0012] A determination module, configured to determine, according to a time synchronization matching rule, second image data acquired by a roadside unit corresponding to the first image data, and form a target image data set according to the first image data and the corresponding second image data; wherein the roadside units are arranged at intervals along a lane direction and the perception areas of adjacent roadside units overlap;

[0013] The image fusion module is used to fuse the target image data set based on the image features corresponding to the overlapping perception areas to obtain the beyond-visual-range visual information corresponding to the target vehicle.

[0014] In a fourth aspect, an embodiment of the present application provides a visual information fusion processing device based on vehicle-road collaboration, wherein the visual information fusion processing device is one of the following: a vehicle-mounted unit, a roadside unit, and a server; the image information processing device includes:

[0015] A memory for storing executable programs;

[0016] The processor is used to implement the visual information fusion processing method based on vehicle-road collaboration described in the above embodiment when executing the executable program stored in the memory.

[0017] In a fifth aspect, an embodiment of the present application provides a vehicle-mounted unit, comprising: a memory for storing an executable program; and a processor for executing the executable program stored in the memory to implement the visual information fusion processing method based on vehicle-road collaboration as described in any of the aforementioned embodiments.

[0018] In a sixth aspect, an embodiment of the present application provides a computer storage medium storing an executable program. When the executable program is executed by a processor, the visual information fusion processing method based on vehicle-road collaboration described in the aforementioned embodiment is implemented.

[0019] In the technical solution provided in the embodiment of the present application, by acquiring the first image data corresponding to the on-board unit of the target vehicle, the second image data respectively acquired by multiple roadside units corresponding to the first image data are determined according to the time synchronization matching rules, a target image data set is formed according to the first image data and the corresponding multiple second image data, and the target image data set is subjected to information fusion based on the image features corresponding to the overlapping perception areas to obtain the beyond-visual-range visual information corresponding to the target vehicle. This can realize the fusion of visual perception information of different road sections, provide comprehensive beyond-visual-range visual information for the autonomous driving decision system, and improve the vehicle's intelligent driving capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of a visual information fusion processing method based on vehicle-road collaboration in one embodiment of the present application;

[0021] Figure 2 This is a schematic diagram of the spatial distribution of roadside units and vehicles in one embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of a process for performing information fusion on a target image data set in one embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of the left and right lane lines after perspective transformation in one embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of a process for performing information fusion on a target image data set in another embodiment of the present application;

[0025] Figure 6 This is a schematic diagram of the structure of a visual information fusion processing device in one embodiment of the present application;

[0026] Figure 7 It is a structural schematic diagram of a visual information fusion processing device in another embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solution of the present application is further described in detail below in conjunction with the drawings and specific embodiments of the specification. It should be understood that the embodiments provided herein are only used to explain the present application and are not intended to limit the present application. In addition, the embodiments provided below are partial embodiments for implementing the present application, rather than providing all embodiments for implementing the present application. In the absence of conflict, the technical solutions recorded in the embodiments of the present application can be implemented in any combination.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which the present application belongs. The terms used herein in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0029] The present application embodiment provides a visual information fusion processing method based on vehicle-road collaboration, which can be applied to a vehicle-mounted unit, a roadside unit or a server. Figure 1 , the method comprising:

[0030] Step 101, obtaining first image data corresponding to the on-board unit of the target vehicle;

[0031] In this embodiment, the vehicle's on-board unit is coupled to the first perception module and the first communication module. The on-board unit can obtain the corresponding first image data through the first perception module, and can communicate with the roadside unit and / or the server through the first communication module.

[0032] Step 102, determining second image data acquired by a plurality of roadside units corresponding to the first image data according to a time synchronization matching rule, and forming a target image data set according to the first image data and the corresponding plurality of second image data; wherein the roadside units are arranged at intervals along the lane direction and the perception areas of adjacent roadside units overlap;

[0033] In this embodiment, the roadside unit includes a second sensing module and a second communication module. The roadside units are arranged at intervals along the lane direction, and the sensing areas of adjacent roadside units overlap. Figure 2 As shown, the perception fields of the second perception modules of the adjacent roadside units 201A and 201B overlap, so that the second image data corresponding to the adjacent roadside units all have visual information of the overlapping area. Each roadside unit can be connected to the vehicle-mounted unit and / or the server via the second communication module.

[0034] Step 103: performing information fusion on the target image data set based on image features corresponding to overlapping perception areas to obtain beyond-visual-range visual information corresponding to the target vehicle.

[0035] Since the second image data in the target image data set are based on the perception fields of the second perception modules of the adjacent roadside units, the second image data corresponding to the adjacent roadside units all have visual information of the overlapping areas. Figure 2 The perception field of view of the on-board unit of the target vehicle 202 at least overlaps with the perception field of view of a roadside unit 201B. Therefore, the first image data corresponding to the target vehicle and the second image data corresponding to at least one roadside unit have visual information of the overlapping area. By fusing the image features corresponding to the overlapping perception areas of each image data, the beyond-visual-range visual information corresponding to the target vehicle can be obtained.

[0036] The embodiments of the present application can realize the fusion of visual information perceived by roadside units and vehicles on different road sections, provide comprehensive beyond-visual-range visual information for the autonomous driving decision-making system, and improve the vehicle's intelligent driving capabilities.

[0037] In the embodiment of the present application, the first and second perception modules may be monocular cameras or multi-cameras, and the first and second communication modules may be wireless communication modules based on wireless LAN or 4G, 5G, etc. When the first perception module and the second perception module are both monocular stand-alone, they can provide two-dimensional visual information beyond the visual range; when the first perception module and the second perception module are both multi-cameras, they can provide three-dimensional visual information beyond the visual range.

[0038] In some embodiments, according to the matching rules of time synchronization, the second image data acquired by multiple roadside units corresponding to the first image data are determined, including: based on the first timestamp of the first image data and the second timestamp of the second image data sent by each roadside unit within the communication range of the target vehicle, according to the time information of the first timestamp and the time information of the second timestamp, the second image data that meets the matching rules of time synchronization is determined to be multiple second image data corresponding to the first image data. Exemplarily, the on-board unit and / or roadside unit of each vehicle can generate a corresponding timestamp based on GPS (Global Positioning System) / differential GPS, or can generate a corresponding timestamp through a PTN (Packet Transport Network) network based on the 1588 protocol and other methods.

[0039] In some embodiments, when it is determined that each second image data acquired by a roadside unit corresponds to two-dimensional image information acquired by a monocular camera, that is, when the second perception module of each roadside unit adopts a monocular camera, the first perception module corresponding to the vehicle's onboard unit may be a monocular camera or a multi-camera, and the corresponding first image data is two-dimensional image data acquired by a monocular camera, or two-dimensional image data acquired by a multi-camera. Optionally, the first image data of the target vehicle may be selected according to the type of the second image data acquired by the roadside unit. For example, when it is determined that the second image data of each roadside unit is two-dimensional image data acquired by a monocular camera, the acquired first image data is two-dimensional image data. Please refer to Figure 3 , performing information fusion on the target image data set based on image features corresponding to the overlapping perception areas, including:

[0040] Step 301, determining a stitching order of two-dimensional images based on position data corresponding to the first image data and each of the second image data;

[0041] The position data corresponding to the first image data acquired by the vehicle-mounted unit, and the position data corresponding to the second image data acquired by the roadside unit. For example, the vehicle-mounted unit can acquire the position data corresponding to the first image data through the GPS positioning device on the vehicle, and the roadside unit can acquire the position data corresponding to the second image data through the configured GPS positioning device, or the position data of the roadside unit is pre-defined and configured to correspond to the identification information of the roadside unit, and the position data corresponding to its second image data can be determined according to the identification information of the roadside unit. The splicing order of the two-dimensional image can be determined based on the position data corresponding to the first image data and each of the second image data. Exemplarily, the image data are sorted along the direction of the lane extension according to the position data corresponding to each image data, so as to determine the splicing order of the two-dimensional image, which can greatly reduce the computational consumption of search and matching during image splicing.

[0042] Step 302, determining a perspective transformation matrix corresponding to the same plane viewing angle for each image data based on the lane line features in the first image data and each of the second image data;

[0043] Based on the lane line features in each image data, a perspective transformation matrix corresponding to the conversion of each image data to the same plane viewing angle is determined to convert each image data to the same plane viewing angle.

[0044] In one embodiment, a perspective transformation matrix corresponding to each image data converted to the same plane viewing angle is determined based on lane line features in the first image data and each of the second image data, including: obtaining a left lane line and a right lane line of a set lane in each image data; extracting intersection points of the left lane line and the right lane line at at least two different image heights, and determining a corresponding perspective transformation matrix based on the principle of a fixed width between lane lines.

[0045] Here, the quadratic fitting function corresponding to the lane line in each image data can be determined by a lane line extraction method. The lane line extraction method can adopt an existing lane line extraction method, which will not be described in detail here. Obtaining the left lane line and the right lane line of the set lane in each image data can be to obtain the quadratic fitting function corresponding to the left lane line and the right lane line corresponding to the current lane of the target vehicle. According to the quadratic fitting function corresponding to the left and right lane lines, the intersection points of the left and right lane lines at at least two different heights are extracted. For example, the intersection points P1 and P2 of the left and right lane lines with the bottom of the image and the intersection points P3 and P4 of the pixel position at a set distance L (L can be taken as 1 / 3 of the image height) from the bottom of the image are taken for perspective transformation. According to the principle of fixed width between lane lines, the lane width after each perspective transformation remains consistent, that is, P1 to P4 can form a fixed width rectangle after perspective transformation.

[0046] For example, let H be the perspective transformation matrix. For a pixel (u, v) at a certain point on the original image, the pixel (x, y) after transformation by H has the relationship of formula (1):

[0047]

[0048] Among them, h 33 =1, so the perspective transformation has 8 degrees of freedom, namely h 11 、h 12 、h 13 、h 21 、h 22 、h 23 、h 31 、h 32 , requires at least 4 pairs of points to calculate.

[0049] According to the quadratic fitting function corresponding to the left and right lane lines, the intersection points of the left and right lane lines at at least two different heights are extracted. For example, the intersection points P1 and P2 of the left and right lane lines with the bottom of the image and the intersection points P3 and P4 with a set distance L (L can be 1 / 3 of the image height) from the bottom of the image are taken, and the coordinates of the four points are respectively P1 = (u1, v1), P2 = (u2, v2), P3 = (u3, v3), and P4 = (u4, v4). Based on this, the corresponding other four points P1' = (u3, v1), P2' = (u4, v2), P3' = (u3, v3), and P4' = (u4, v4) are determined. The perspective transformation matrix H of formula (1) can be calculated with these four pairs of points.

[0050] Since the perspective transformation matrix determined by the above formula (1) may not guarantee that the lane lines after perspective transformation are parallel, in some embodiments, the perspective transformation matrix can also be optimized so that the lane lines after perspective transformation remain parallel as much as possible. Optionally, determining the perspective transformation matrix corresponding to the conversion of each image data to the same plane perspective based on the lane line features in the first image data and each of the second image data also includes: obtaining the first lane line and the second lane line after the left lane line and the right lane line of the set lane in each image data are converted by the perspective transformation matrix; based on the target point where the normal of any point on the first lane line intersects with the second lane line, optimizing the perspective transformation matrix according to the equal slope of the arbitrary point and the target point.

[0051] Here, a lane line is used as the reference lane line, and the reference lane line is not limited to the left lane line or the right lane line of any lane. The left lane line of the current lane is taken as an example. Figure 4 As shown, assuming that the quadratic equation corresponding to the left lane line L after perspective transformation is expressed by formula (2):

[0052] x=a1y2 +a2y+a3 (2)

[0053] The quadratic equation corresponding to the right lane line R after perspective transformation is expressed by formula (3):

[0054] x=b1y 2 +b2y+b3 (3)

[0055] The horizontal coordinate is x and the vertical coordinate is y. Then the slope at any y coordinate can be expressed by formula (4):

[0056] x′=2a1y+a2 (4)

[0057] Suppose the coordinates of a point on the left lane are (x i ,y i ), then the normal equation at this point can be expressed by formula (5):

[0058]

[0059] According to formula (5), the intersection point (x j ,y j ), similarly, according to formula (4), the slope x of the point on the right lane line can be calculated j ′. Assume that the left and right lane lines become parallel after perspective transformation, x′ i =x j ′.

[0060] In this way, according to the point (x i ,y i ) and the point (x j ,y j ) are equal, the perspective transformation matrix H can be optimized.

[0061] In some embodiments, the perspective transformation matrix H can be further optimized by a nonlinear optimization method. It should be noted that the nonlinear optimization method can be an optimization of the perspective transformation matrix H determined according to the above formula (1), or it can be a further optimization based on the above optimization of the perspective transformation matrix based on equal slopes.

[0062] Optionally, determining the perspective transformation matrix corresponding to the conversion of each image data to the same plane viewing angle based on the lane line features in the first image data and each of the second image data also includes: obtaining the first lane line and the second lane line of the left lane line and the right lane line of the set lane in each image data after being converted by the perspective transformation matrix; obtaining the first slopes of multiple pixel points on the first lane line and the second slopes of the intersections where the normals of the multiple pixel points intersect with the second lane line, determining a cost function based on the difference between each of the first slopes and the corresponding second slope; and optimizing the perspective transformation matrix using a nonlinear optimization method based on the cost function.

[0063] Here, the cost function can be determined using the slope of each point on the left lane line and the slope difference between the normal line of each point and the intersection of the right lane line. The cost function is shown in formula (6):

[0064]

[0065] Where N is the number of lane line pixels in the image, i is the pixel number of the lane line along the y direction (i.e., the image height direction), and x′ i is the slope of the left lane line at point i obtained by formula (4), j is the intersection of the normal line of the right lane line and the left lane line at point i, x′ j is the slope of the jth point on the right lane line obtained by formula (4). The cost function can be expressed by formula (7):

[0066]

[0067] Among them, I(u i ,v i ) is the coordinate point (x) on the lane line after perspective transformation i ,y i ) corresponds to the pixel point on the original image, I(u j ,v j ) is the coordinate point (x) on the lane line after perspective transformation j ,y j ) on the original image. G represents the function of extracting the slope of the lane line at the original image after the lane line pixel point is transformed through perspective, which can be expressed by formula (8):

[0068] G=g(a1,a2,a3,b1,b2,b3,H) (8)

[0069] Where a1, a2, a3 are coefficient parameters of the quadratic equation for the left lane line, b1, b2, b3 are coefficient parameters of the quadratic equation for the right lane line, and H is the perspective transformation matrix required to transform the original image into a target image with parallel lane lines. Optionally, in the nonlinear optimization process, a3 and b3 are intercept positions that can be set to fixed values ​​based on the fixed lane line spacing. Taking {H} as the parameter object to be optimized, nonlinear optimization can be performed using, for example, the Levenberg-Marquardt method, and ultimately the optimal perspective transformation matrix H is obtained so that the left and right lane lines are parallel to each other after the perspective transformation.

[0070] Step 303: Based on the perspective transformation matrices and the stitching order of the two-dimensional images, information fusion is performed on each image data based on the image features of the overlapping areas.

[0071] Through step 302, the perspective transformation matrix H corresponding to each image data is obtained, and each image data is transformed using the perspective transformation matrix H corresponding to each image data obtained, and feature points (i.e., image features) are extracted from each converted image data, and the two-dimensional images are sequentially spliced ​​according to the extracted feature points in the splicing order of the two-dimensional images. For example, feature points (such as vehicles, lane lines, signboards, etc.) that are distinguished from the image background can be extracted from the converted image, and RANSAC (Random Sample Consensus) is used to sequentially splice according to the splicing order of the two-dimensional images. The spliced ​​two-dimensional images can be fused using a gradient fusion algorithm. In this way, the fusion of the first image data corresponding to the on-board unit and the second image data corresponding to the roadside unit is realized, providing comprehensive beyond-visual-range visual information for the automatic driving decision system, and improving the vehicle's intelligent driving capability.

[0072] In some embodiments, when it is determined that the second image data acquired by each roadside unit corresponds to the image data acquired by the multi-eye camera, the first image data acquired by the vehicle-mounted unit is also the image data acquired by the multi-eye camera on the vehicle side. Figure 5 , performing information fusion on the target image data set based on image features corresponding to the overlapping perception areas, including:

[0073] Step 501, determining a splicing order of three-dimensional visual information based on position data corresponding to the first image data and each of the second image data;

[0074] The position data corresponding to the first image data acquired by the vehicle-mounted unit, and the position data corresponding to the second image data acquired by the roadside unit. The splicing order of the three-dimensional visual information can be determined based on the position data corresponding to the first image data and each of the second image data. Exemplarily, each image data is sorted along the direction in which the lane extends according to the position data corresponding to each image data. Specifically, the first image data comes from the second image data corresponding to the roadside unit that is closest to the target vehicle and located at the rear end of its driving direction, the second image data comes from the first image data corresponding to the vehicle-mounted unit of the target vehicle, and the following image data are arranged in order from near to far from the target vehicle. The second image data corresponding to a set number of roadside units, thereby determining the splicing order of the three-dimensional visual information can greatly reduce the computational consumption of search and matching during splicing.

[0075] Step 502, determining a second conversion relationship between the second image data of two adjacent roadside units based on the first conversion relationship corresponding to each of the second image data and the image features of the overlapping area between the second image data of two adjacent roadside units; wherein the first conversion relationship is a conversion relationship between the image coordinate system of the second image data and the world coordinate system;

[0076] In some embodiments, the conversion relationship between the image coordinate system of each second image data and the world coordinate system (i.e., the first conversion relationship) is first determined, and based on the determined first conversion relationship corresponding to each second image data, the images of each second image data are unified into the world coordinate system, and then the second conversion relationship between the second image data of two adjacent roadside units is determined based on the image features of the second image data of the two adjacent roadside units in the overlapping area.

[0077] Specifically, obtaining the conversion relationship between the image coordinate system corresponding to each second image data and the world coordinate system (i.e., the first conversion relationship) specifically includes: determining the relationship between the image coordinate system of the second image data of the roadside unit and the multi-eye stereo camera coordinate system through camera calibration technology, and then determining the relationship between the multi-eye stereo camera coordinate system and the world coordinate system with the help of calibration technology of multi-eye camera and GPS or combined inertial navigation equipment, thereby obtaining the first conversion relationship. For example, for the i-th roadside unit, the relationship T1 between the image coordinate system of the roadside unit and the multi-eye stereo camera coordinate system can be determined by, for example, Zhang Zhengyou's calibration method i Then, determine the relationship between the multi-camera coordinate system and the world coordinate system T2 i , based on relation T1 i and relation T2 i A first conversion relationship can be determined, which can be relationship T1 i and relation T2 i The product of .

[0078] Optionally, in some embodiments, the second conversion relationship between the second image data of two adjacent roadside units is determined based on the first conversion relationship corresponding to each second image data and the image features of the overlapping area between the second image data of two adjacent roadside units, including: obtaining the first conversion relationship corresponding to each second image data; extracting abstract features based on target detection for a two-dimensional image corresponding to each second image data to generate a first abstract feature set; extracting a point cloud set of each second image data belonging to the first abstract feature set in three-dimensional space based on the first abstract feature set and the first conversion relationship; matching the point cloud set of each second image data based on the three-dimensional visual features of the overlapping area to determine the second conversion relationship between the second image data of two adjacent roadside units.

[0079] Since each roadside unit uses a multi-camera to collect and generate second image data, the second image data corresponds to at least two two-dimensional images (for example, when a binocular camera is used, the second image data corresponds to two two-dimensional images), and abstract features are extracted from a two-dimensional image corresponding to each of the second image data to generate a first abstract feature set. For example, target detection is performed on a two-dimensional image corresponding to the second image data of each roadside unit, and abstract features (such as image features that can distinguish the background, such as vehicles, lane lines, and signboards) are extracted to generate a first abstract feature set corresponding to the two-dimensional image. Exemplarily, abstract features may be extracted based on machine learning methods such as SVM and Adaboost or methods such as SSD and YOLO based on deep learning, which will not be described in detail here.

[0080] According to the first abstract feature set Extract the point cloud set of each second image data belonging to the first abstract feature set in the three-dimensional space. For example, after obtaining the first transformation relationship corresponding to each second image data, the index of the image coordinates in each image coordinate system to the three-dimensional coordinates in the world coordinate system can be recorded, so that the point cloud set belonging to the first abstract feature set can be quickly determined. For the i-th roadside unit, determine the point cloud set belonging to the first abstract feature set in the three-dimensional space

[0081] Since the image features of the overlapping area have a corresponding relationship on the three-dimensional point cloud, optionally, the point cloud set of the second image data of the two adjacent roadside units is matched based on the three-dimensional visual features of the overlapping area by an optimal method to determine the second conversion relationship between the second image data of the two adjacent roadside units. For example, the point cloud set corresponding to the adjacent i-th roadside unit and the j-th roadside unit is matched based on the three-dimensional visual features of the overlapping area by using the LM (Levenberg-Marquard) algorithm and other methods to obtain the conversion relationship from the i-th roadside unit to the j-th roadside unit. That is, a second conversion relationship between the three-dimensional visual information corresponding to the second image data of two adjacent roadside units is obtained.

[0082] Step 503, splicing the three-dimensional visual information corresponding to the second image data according to the splicing order of the three-dimensional visual information based on each of the second conversion relationships;

[0083] After determining the second conversion relationship between the second image data of two adjacent roadside units, corresponding three-dimensional visual information can be generated according to the second image data of each adjacent roadside unit, and then the three-dimensional visual information is spliced ​​according to the splicing order of the three-dimensional visual information based on the corresponding second conversion relationship, so as to obtain the spliced ​​point cloud P of each roadside unit RSU .

[0084] Step 504, determining a fourth transformation relationship corresponding to the first image data based on the third transformation relationship corresponding to the first image data and image features of an overlapping area between the first image data and the second image data overlapping therewith; wherein the third transformation relationship is a transformation relationship between an image coordinate system of the first image data and a world coordinate system, and the fourth transformation relationship is a transformation relationship for transforming the first image data into the spliced ​​second image data;

[0085] In order to combine the first image data acquired by the on-board unit with the point cloud P RSU The fusion is performed, and optionally, a third transformation relationship corresponding to the first image data can be determined, that is, a calibration method corresponding to the second image data is used to determine the transformation relationship between the image coordinate system of the first image data and the world coordinate system; in this way, the first image data can be transformed into the world coordinate system to form a corresponding three-dimensional point cloud P' OBU . Then, extract abstract features from a two-dimensional image corresponding to the first image data to generate a second abstract feature set, and extract abstract features from a two-dimensional image corresponding to the second image data that overlaps with the first image data to generate a third abstract feature set; extract a point cloud set of the first image data belonging to the second abstract feature set in three-dimensional space according to the second abstract feature set and the third conversion relationship; determine a point cloud set of the second image data belonging to the third abstract feature set in three-dimensional space according to the third abstract feature set and the first conversion relationship; match the point cloud set of the first image data with the point cloud set of the second image data based on the three-dimensional visual features of the overlapping area to determine the fourth conversion relationship.

[0086] Step 505: Based on the fourth conversion relationship, information fusion is performed on the three-dimensional visual information corresponding to the first image data and the three-dimensional visual information corresponding to the stitched second image data.

[0087] The three-dimensional point cloud P' corresponding to the first image data OBU The fourth transformation relationship is obtained with the point cloud P RSU Matched point cloud P OBU , the point cloud P RSU and P OBU By integrating the 3D visual information acquired by the on-board unit and the 3D visual information of the corresponding roadside unit, the beyond-visual-range visual information corresponding to the target vehicle can be obtained. This can realize the integration of visual perception information of different road sections, provide comprehensive beyond-visual-range image information for the autonomous driving decision-making system, and improve the vehicle's intelligent driving capabilities.

[0088] The embodiment of the present application is based on a visual information fusion processing method for vehicle-road collaboration. Compared with direct stitching and fusion using the position information of the camera in the world coordinate system, and using the redundant data content between the cameras to calculate the homography matrix to complete image stitching, the information fusion is based on the image features corresponding to the overlapping perception areas, so that the visual information (two-dimensional or three-dimensional visual information) stitching is more accurate, and pixel-level fusion between visual information can be achieved. It also satisfies the spatiotemporal synchronization of visual information in a state of motion change in relative posture between the on-board unit and each roadside unit, and the system has higher robustness. In addition, the method of this embodiment can obtain visual information corresponding to multiple roadside units distributed in different sections, and fuse each visual information with the visual information obtained by the on-board unit, so as to achieve the fusion of visual information of different sections, thereby providing comprehensive beyond-visual-range information for the autonomous driving decision system and improving the vehicle's intelligent driving capabilities.

[0089] It should be noted that the visual information fusion processing method based on vehicle-road collaboration in the embodiment of the present application can be applied to a roadside unit, a vehicle-mounted unit or a server.

[0090] Taking the application to the vehicle-mounted unit as an example, the vehicle-mounted unit obtains the first image data corresponding to its own first perception module, the second perception module of each roadside unit obtains the corresponding second image data, and the second communication module of the roadside unit broadcasts the corresponding second image data to each vehicle-mounted unit within the communication range. The first communication module of the vehicle-mounted unit receives the second image data broadcast by the second communication module of the corresponding roadside unit, and the vehicle-mounted unit selects the image data whose difference with the time information of the timestamp of the acquired first image data is within a set threshold as the matching second image data, so that the first image data and the corresponding second image data form a target image data set. The target image data set is subjected to information fusion based on the image features corresponding to the overlapping perception areas to obtain the beyond-visual-range visual information corresponding to the target vehicle.

[0091] Optionally, when the onboard unit receives the second image data broadcast by each roadside unit, the receiving frequency of the onboard unit receiving the second image data broadcast by the corresponding roadside unit can be adjusted according to the distance between the onboard unit and the corresponding roadside unit and / or the moving speed of the onboard unit. For example, the receiving frequency can be increased according to the increase of the moving speed of the onboard unit, and the receiving frequency can be decreased according to the decrease of the moving speed of the onboard unit; or the receiving frequency can be decreased according to the increase of the distance between the onboard unit and the roadside unit, and the receiving frequency can be increased according to the decrease of the distance between the onboard unit and the roadside unit. In this way, the receiving frequency of the second image data broadcast by each roadside unit at different times or the receiving frequency of the second image data broadcast by the roadside units of different sections at the same time can be different. Through the corresponding receiving frequency adjustment strategy, the fusion speed of useful or more relevant visual information can be faster, which is conducive to improving the intelligent driving level of the vehicle.

[0092] The present application also provides a visual information fusion processing device based on vehicle-road collaboration, see Figure 6 , the visual information fusion processing device comprises:

[0093] An acquisition module 601 is used to acquire first image data corresponding to the vehicle-mounted unit of the target vehicle;

[0094] A determination module 602 is used to determine, according to a time synchronization matching rule, second image data acquired by a roadside unit corresponding to the first image data, and form a target image data set according to the first image data and the corresponding second image data; wherein the roadside units are arranged at intervals along the lane direction and the perception areas of adjacent roadside units overlap;

[0095] The information fusion module 603 is used to fuse the target image data set based on the image features corresponding to the overlapping perception areas to obtain the beyond-visual-range visual information corresponding to the target vehicle.

[0096] In some embodiments, when it is determined that the second image data acquired by the roadside unit is image data collected by a monocular camera, the information fusion module 603 is specifically used to: determine the stitching order of the two-dimensional images based on the position data corresponding to the first image data and each of the second image data; determine the perspective transformation matrix corresponding to the conversion of each image data to the same plane viewing angle based on the lane line features in the first image data and each of the second image data; and perform information fusion on the image features of the overlapping areas of each image data based on the perspective transformation matrices and the stitching order of the two-dimensional images.

[0097] In some embodiments, the information fusion module 603 is also used to: obtain the left lane line and the right lane line of the set lane in each image data; extract the intersection of the left lane line and the right lane line at at least two different image heights, and determine the corresponding perspective transformation matrix based on the principle of fixed width between lane lines.

[0098] In some embodiments, the information fusion module 603 is also used to: obtain the first lane line and the second lane line of the left lane line and the right lane line of the set lane in each image data after being converted by the perspective transformation matrix; based on the target point where the normal of any point on the first lane line intersects with the second lane line, optimize the perspective transformation matrix according to the equal slope of the arbitrary point and the target point.

[0099] In some embodiments, the information fusion module 603 is also used to: obtain the first lane line and the second lane line of the left lane line and the right lane line of the set lane in each image data after being converted by the perspective transformation matrix; obtain the first slope of multiple pixel points on the first lane line and the second slope of the intersection point where the normal of the multiple pixel points intersects with the second lane line, and determine the cost function based on the difference between each first slope and the corresponding second slope; and optimize the perspective transformation matrix using a nonlinear optimization method based on the cost function.

[0100] In some embodiments, when it is determined that the second image data acquired by the roadside unit is image data collected by a multi-eye camera, the information fusion module 603 is used to: determine the splicing order of the three-dimensional visual information based on the position data corresponding to the first image data and each of the second image data; determine the second conversion relationship between the second image data of the two adjacent roadside units based on the first conversion relationship corresponding to each of the second image data and the image features of the overlapping area between the second image data of the two adjacent roadside units; wherein the first conversion relationship is the conversion relationship between the image coordinate system of the second image data and the world coordinate system; based on each of the second conversion relationships, the three-dimensional visual information corresponding to the second image data is spliced ​​according to the splicing order of the three-dimensional visual information; based on the third conversion relationship corresponding to the first image data and the image features of the overlapping area between the first image data and the second image data overlapping therewith, determine the fourth conversion relationship corresponding to the first image data; wherein the third conversion relationship is the conversion relationship between the image coordinate system of the first image data and the world coordinate system, and the fourth conversion relationship is the conversion relationship for converting the first image data into the second image data after splicing; based on the fourth conversion relationship, the three-dimensional visual information corresponding to the first image data and the three-dimensional visual information corresponding to the second image data after splicing are information fused.

[0101] In some embodiments, the information fusion module 603 is also used to: obtain a first conversion relationship corresponding to each second image data; extract abstract features from a two-dimensional image corresponding to each second image data to generate a first abstract feature set; extract a point cloud set belonging to the first abstract feature set in the three-dimensional space of each second image data based on the first abstract feature set and the first conversion relationship; match the point cloud set of each second image data based on the three-dimensional visual features of the overlapping area to determine the second conversion relationship between the three-dimensional visual information corresponding to the second image data of two adjacent roadside units.

[0102] In some embodiments, the information fusion module 603 is also used to: obtain a third conversion relationship corresponding to the first image data; extract abstract features from a two-dimensional image corresponding to the first image data to generate a second abstract feature set, and extract abstract features from a two-dimensional image corresponding to the second image data that overlaps with the first image data to generate a third abstract feature set; extract a point cloud set of the first image data belonging to the second abstract feature set in three-dimensional space based on the second abstract feature set and the third conversion relationship; determine a point cloud set of the second image data belonging to the third abstract feature set in three-dimensional space based on the third abstract feature set and the first conversion relationship; match the point cloud set of the first image data with the point cloud set of the second image data based on the three-dimensional visual features of the overlapping area to determine the fourth conversion relationship.

[0103] It should be noted that: when the visual information fusion processing device provided in the above embodiment performs visual information fusion processing, only the division of the above program modules is used as an example. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the visual information fusion processing device provided in the above embodiment and the visual information fusion processing method embodiment based on vehicle-road collaboration belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0104] The present application also provides a visual information fusion processing device. Figure 7 , which shows a visual information fusion processing device, which can be a vehicle-mounted unit, a roadside unit, or a server, etc. Figure 7 Only an exemplary structure of the visual information fusion processing device is shown, not the entire structure, and it can be implemented as needed. Figure 7 Partial or complete structure shown.

[0105] The visual information fusion processing device 700 provided in the embodiment of the present application includes: at least one processor 701, a memory 702, a user interface 703 and at least one network interface 704. The various components in the visual information fusion processing device 700 are coupled together through a bus system 705. It can be understood that the bus system 705 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 705 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 7 Various buses are labeled as bus system 405 .

[0106] The user interface 703 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0107] It can be understood that the memory 702 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories.

[0108] The memory 702 in the embodiment of the present application is used to store various types of data to support the execution of the visual information fusion processing method based on vehicle-road collaboration. Examples of such data include: any executable program for running on the visual information fusion processing device 700, such as the executable program 7021, and the program for implementing the visual information fusion processing method based on vehicle-road collaboration in the embodiment of the present application can be included in the executable program 7021.

[0109] The visual information fusion processing method based on vehicle-road collaboration disclosed in the embodiment of the present application can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the visual information fusion method can be completed by the hardware integrated logic circuit in the processor 701 or the instructions in the form of software. The above-mentioned processor 701 may be a general-purpose processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 701 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present application, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in the memory 702. The processor 701 reads the information in the memory 702 and completes the steps of the visual information fusion method provided in the embodiment of the present application in combination with its hardware.

[0110] The embodiment of the present application also provides a readable storage medium, which may include: a mobile storage device, a random access memory (RAM), a read-only memory (ROM), a magnetic disk or an optical disk, and other media that can store program codes. The readable storage medium stores an executable program; the executable program is used to implement the visual information fusion processing method based on vehicle-road collaboration described in any embodiment of the present application when executed by a processor.

[0111] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the embodiments of the present application may take the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the embodiments of the present application may take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program codes.

[0112] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing system to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing system generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0113] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing system to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing system so that a series of operating steps are executed on the computer or other programmable system to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0115] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A visual information fusion processing method based on vehicle-road collaboration, characterized in that: include: Acquire first image data corresponding to the onboard unit of the target vehicle; The first image data is obtained based on a first perception module of the vehicle-mounted unit; According to a time synchronization matching rule, second image data respectively acquired by a plurality of roadside units corresponding to the first image data are determined, and a target image data set is formed according to the first image data and the corresponding plurality of second image data; wherein the roadside units are arranged at intervals along the lane direction and the perception areas of adjacent roadside units overlap; and the second image data is acquired based on a second perception module of the roadside unit; Performing information fusion on the target image data set based on image features corresponding to overlapping perception areas to obtain beyond-visual-range visual information corresponding to the target vehicle; If the second perception module is a monocular camera, and the first perception module is a monocular camera or a multi-camera, the target image data set is subjected to information fusion based on image features corresponding to overlapping perception areas, including: Determine the stitching order of the two-dimensional images based on the position data corresponding to the first image data and each of the second image data; determine the perspective transformation matrix corresponding to the conversion of each image data to the same plane viewing angle based on the lane line features in the first image data and each of the second image data; perform information fusion on the image features of the overlapping areas of each image data based on each of the perspective transformation matrices and the stitching order of the two-dimensional images; wherein the first image data is the two-dimensional image data collected by the monocular camera, or the two-dimensional image data collected by the multi-camera; or, If the second perception module is a multi-eye camera and the first perception module is a multi-eye camera, the information fusion of the target image data set based on the image features corresponding to the overlapping perception areas includes: Determine the stitching order of the three-dimensional visual information based on the position data corresponding to the first image data and each of the second image data; determine the second transformation relationship between the second image data of two adjacent roadside units based on the first transformation relationship corresponding to each of the second image data and the image features of the overlapping area between the second image data of two adjacent roadside units; wherein the first transformation relationship is the transformation relationship between the image coordinate system of the second image data and the world coordinate system; stitch the three-dimensional visual information corresponding to the second image data according to the stitching order of the three-dimensional visual information based on each of the second transformation relationships; determine the fourth transformation relationship corresponding to the first image data based on the third transformation relationship corresponding to the first image data and the image features of the overlapping area between the first image data and the second image data overlapping therewith; Among them, the third transformation relationship is the transformation relationship between the image coordinate system of the first image data and the world coordinate system, and the fourth transformation relationship is the transformation relationship for converting the first image data into the second image data after splicing; based on the fourth transformation relationship, the three-dimensional visual information corresponding to the first image data and the three-dimensional visual information corresponding to the second image data after splicing are fused; wherein, the first image data is image data collected by the multi-eye camera on the vehicle side.

2. The visual information fusion processing method based on vehicle-road collaboration according to claim 1, characterized in that: Determining a perspective transformation matrix corresponding to the conversion of each image data to the same plane viewing angle based on the lane line features in the first image data and each of the second image data, comprising: Obtaining the left lane marking and the right lane marking of the set lane in each image data; The intersection points of the left lane line and the right lane line at at least two different image heights are extracted, and based on the principle that the width between lane lines is fixed, the corresponding perspective transformation matrix is ​​determined.

3. The visual information fusion processing method based on vehicle-road collaboration as claimed in claim 2, characterized in that: The method further comprises: determining a perspective transformation matrix corresponding to the conversion of each image data to the same plane viewing angle based on the lane line features in the first image data and each of the second image data; Obtaining a first lane line and a second lane line of a left lane line and a right lane line of a set lane in each image data after being transformed by the perspective transformation matrix; Based on a target point where a normal line of an arbitrary point on the first lane line intersects with the second lane line, the perspective transformation matrix is ​​optimized according to the slopes of the arbitrary point and the target point being equal.

4. The visual information fusion processing method based on vehicle-road collaboration as claimed in claim 2, characterized in that: The method further comprises: determining a perspective transformation matrix corresponding to the conversion of each image data to the same plane viewing angle based on the lane line features in the first image data and each of the second image data; Obtaining a first lane line and a second lane line of a left lane line and a right lane line of a set lane in each image data after being transformed by the perspective transformation matrix; Obtaining first slopes of a plurality of pixel points on the first lane line and second slopes of intersection points where normal lines of the plurality of pixel points intersect the second lane line, and determining a cost function based on a difference between each first slope and the corresponding second slope; The perspective transformation matrix is ​​optimized using a nonlinear optimization method based on the cost function.

5. The visual information fusion processing method based on vehicle-road collaboration according to claim 1, characterized in that: Determining a second conversion relationship between the second image data of two adjacent roadside units based on the first conversion relationship corresponding to each of the second image data and image features of an overlapping area between the second image data of two adjacent roadside units includes: Acquire a first conversion relationship corresponding to each of the second image data; Extracting abstract features from a two-dimensional image corresponding to each of the second image data to generate a first abstract feature set; Extracting a point cloud set of each of the second image data belonging to the first abstract feature set in a three-dimensional space according to the first abstract feature set and the first conversion relationship; The point cloud sets of each second image data are matched based on the three-dimensional visual features of the overlapping area to determine the second conversion relationship between the three-dimensional visual information corresponding to the second image data of two adjacent roadside units.

6. The visual information fusion processing method based on vehicle-road collaboration according to claim 1, characterized in that: Determining a fourth conversion relationship corresponding to the first image data based on the third conversion relationship corresponding to the first image data and image features of an overlapping area between the first image data and the second image data overlapping therewith, includes: Acquire a third conversion relationship corresponding to the first image data; Extracting abstract features from a two-dimensional image corresponding to the first image data to generate a second abstract feature set, and extracting abstract features from a two-dimensional image corresponding to the second image data overlapping therewith to generate a third abstract feature set; Extracting a point cloud set of the first image data belonging to the second abstract feature set in three-dimensional space according to the second abstract feature set and the third conversion relationship; determining a point cloud set of the second image data belonging to the third abstract feature set in three-dimensional space according to the third abstract feature set and the first conversion relationship; The point cloud set of the first image data and the point cloud set of the second image data are matched based on the three-dimensional visual features of the overlapping area to determine the fourth conversion relationship.

7. A visual information fusion processing method based on vehicle-road collaboration, characterized in that: The on-board unit is applied to a vehicle to implement the visual information fusion processing method as described in any one of claims 1 to 6.

8. The visual information fusion processing method based on vehicle-road collaboration according to claim 7 is characterized in that: Before forming a target image data set according to the first image data and the corresponding plurality of second image data, the method further includes: Receive the second image data broadcasted by the corresponding multiple roadside units; wherein the receiving frequency of receiving the second image data broadcasted by each roadside unit is determined based on the distance between the vehicle-mounted unit and the corresponding roadside unit and / or the moving speed of the vehicle-mounted unit.

9. A visual information fusion processing device based on vehicle-road collaboration, characterized in that: include: An acquisition module, used for acquiring first image data corresponding to the on-board unit of the target vehicle; A determination module, configured to determine, according to a time synchronization matching rule, second image data acquired by a roadside unit corresponding to the first image data, and form a target image data set according to the first image data and the corresponding second image data; wherein the roadside units are arranged at intervals along a lane direction and the perception areas of adjacent roadside units overlap; An image fusion module is used to fuse the target image data set based on the image features corresponding to the overlapping perception areas to obtain the beyond-horizon visual information corresponding to the target vehicle; the image fusion module is also used to fuse the target image data set based on the image features corresponding to the overlapping perception areas if each of the second perception modules is a monocular camera and the first perception module is a monocular camera or a multi-eye camera, including: Determine the stitching order of the two-dimensional images based on the position data corresponding to the first image data and each of the second image data; determine the perspective transformation matrix corresponding to the conversion of each image data to the same plane viewing angle based on the lane line features in the first image data and each of the second image data; perform information fusion on the image features of the overlapping areas of each image data based on each of the perspective transformation matrices and the stitching order of the two-dimensional images; wherein the first image data is the two-dimensional image data collected by the monocular camera, or the two-dimensional image data collected by the multi-camera; or, If the second perception module of each of the drive test units is a multi-eye camera, and the first perception module is a multi-eye camera, the information fusion of the target image data set based on image features corresponding to overlapping perception areas includes: The splicing order of the three-dimensional visual information is determined based on the position data corresponding to the first image data and each of the second image data; the second conversion relationship between the second image data of the two adjacent roadside units is determined based on the first conversion relationship corresponding to each of the second image data and the image features of the overlapping area between the second image data of the two adjacent roadside units; wherein the first conversion relationship is the conversion relationship between the image coordinate system of the second image data and the world coordinate system; based on each of the second conversion relationships, the three-dimensional visual information corresponding to the second image data is spliced ​​according to the splicing order of the three-dimensional visual information; based on the third conversion relationship corresponding to the first image data and the image features of the overlapping area between the first image data and the second image data overlapping therewith, the fourth conversion relationship corresponding to the first image data is determined; wherein the third conversion relationship is the conversion relationship between the image coordinate system of the first image data and the world coordinate system, and the fourth conversion relationship is the conversion relationship for converting the first image data into the second image data after splicing; based on the fourth conversion relationship, the three-dimensional visual information corresponding to the first image data and the three-dimensional visual information corresponding to the second image data after splicing are fused; wherein the first image data is image data collected by the multi-eye camera on the vehicle side.

10. A visual information fusion processing device based on vehicle-road collaboration, characterized in that: The visual information fusion processing device is one of the following: a vehicle-mounted unit, a roadside unit, and a server; the image information processing device includes: A memory for storing executable programs; The processor is used to implement the visual information fusion processing method based on vehicle-road collaboration as described in any one of claims 1 to 6 when executing the executable program stored in the memory.

11. A vehicle-mounted unit, characterized in that: include: A memory for storing executable programs; The processor is used to implement the visual information fusion processing method based on vehicle-road collaboration as described in claim 8 when executing the executable program stored in the memory.

12. A computer storage medium, characterized in that: An executable program is stored, and when the executable program is executed by a processor, the visual information fusion processing method based on vehicle-road collaboration as described in any one of claims 1 to 8 is implemented.

Images