A method, apparatus and related medium for lane data fusion

Abstract

This application discloses a method, apparatus, and related medium for lane data fusion. The method may include the following steps: a vehicle receives first lane data sent by a target device within a preset range, the first lane data including a first lane image; the vehicle acquires an original lane image through a first image pickup; the vehicle acquires a second lane image based on the original lane image; and an image fusion operation is performed on the first lane image and the second lane image to obtain a third lane image. By implementing the method of this application, the vehicle can reasonably select the source of lane data and fuse the appropriate lane data source with the lane data acquired by the vehicle, which helps to supplement lane data in the blind spot of the vehicle and also helps the vehicle or the customer to understand lane information at a greater distance, thereby improving the accuracy and reliability of lane data.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to a method, apparatus and related medium for lane data fusion. Background Technology

[0002] With the development of technology, various industries are improving their products towards intelligence. For example, more and more car manufacturers are focusing on autonomous driving technology. The realization of autonomous driving technology relies on lane keeping assist and automatic cruise control, and lane data is fundamental to these functions. Current technology can collect lane data through cameras and / or sensors on the vehicle; however, this method is limited by the location of the cameras and sensors, resulting in blind spots and short viewing distances. This leads to unreliable lane data transmitted to users and / or vehicles. Users and / or vehicles relying on this lane data during driving may face safety hazards.

[0003] Therefore, how to provide a method for generating reliable lane data is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] One objective of this application is to provide a method, apparatus, and related medium for lane data fusion, which can fuse appropriate lane data sources with lane data acquired by the vehicle, helping to supplement lane data in the blind spot of the vehicle and also helping the vehicle or customer to understand lane information at a greater distance, thereby improving the accuracy and reliability of lane data.

[0005] In a first aspect, this application provides a method for lane data fusion, which may include the following steps:

[0006] The vehicle receives first lane data sent by a target device within a preset range. The first lane data may include a first lane image.

[0007] The vehicle acquires the original lane image through the first image pickup;

[0008] The vehicle acquires a second lane image based on the original lane image;

[0009] The first lane image and the second lane image are fused together to obtain the third lane image.

[0010] In one possible implementation, the first lane data may further include a first feature point matrix, which is generated by the target device based on the RGBD image corresponding to the original lane image it acquires.

[0011] The method may also include the following steps:

[0012] Generate an RGBD image based on the original lane image;

[0013] Extract the second feature point matrix from the RGBD image;

[0014] When the root mean square error between the first lane image and the second lane image is less than a preset value, the target transpose matrix is ​​determined based on the first feature point matrix and the second feature point matrix.

[0015] The image fusion operation performed on the first lane image and the second lane image to obtain the third lane image may include:

[0016] The first lane image and the second lane image are fused based on the target transpose matrix to obtain the third lane image.

[0017] In another possible implementation, the first image picker may include a first camera and a second camera, and the original lane image may include the original lane image captured by the first camera and the original lane image captured by the second camera.

[0018] Generating an RGBD image from the original lane image may include the following steps:

[0019] The original images of the first lane and the second lane are corrected according to the preset distortion correction coefficients to obtain the first corrected image and the second corrected image.

[0020] Determine the feature matching points of the first and second corrected images;

[0021] Generate a first matching image corresponding to the first corrected image based on the feature matching points, and generate a second matching image corresponding to the second corrected image.

[0022] Generate an RGBD image based on the first and second matching images.

[0023] In another possible implementation, the method may further include the following steps:

[0024] Perform lane line recognition on the first corrected image to generate a first reference lane line image;

[0025] Perform coordinate mapping on the first reference lane line image to generate the second lane image.

[0026] In another possible implementation, generating an RGBD image based on a first matching image and a second matching image may include the following steps:

[0027] Determine the similarity transformation parameters between the first matching image and the second matching image;

[0028] The first matching image and similarity transformation parameters are input into a preset neural network to obtain a depth map and an RGB image.

[0029] An RGBD image is generated by overlaying a depth map with an RGB image.

[0030] In another possible implementation, determining the target transpose matrix based on the first feature point matrix and the second feature point matrix may include the following steps:

[0031] N values ​​are randomly selected from the first feature point matrix in N rounds to generate N first matrices, where N is a positive integer and the preset number of groups is greater than or equal to 3.

[0032] N values ​​are randomly selected from the second feature point matrix in N steps, with the same number of groups as the first matrix, to generate N second matrices.

[0033] Divide the N first matrices and N second matrices into N groups in a one-to-one correspondence, and determine the reference transpose matrix of the second matrix in each group when it is transformed into the first matrix;

[0034] Calculate the average square error of N reference transpose matrices;

[0035] The reference transpose matrix that minimizes the average squared error is determined as the target transpose matrix.

[0036] In another possible implementation, the method may further include the following steps:

[0037] If the number of target devices is greater than or equal to 2, the importance of each target device is ranked.

[0038] Select a preset number of target devices based on their importance.

[0039] Secondly, this application provides a lane data fusion method applied to a target device, the target device including a second image pickup, the method including the following steps:

[0040] The target device acquires the original lane image corresponding to the target device through the second image pickup;

[0041] The target device obtains first lane data based on the original lane image corresponding to the target device. The first lane data may include the first lane image.

[0042] The target device sends the first lane data to vehicles within a preset range. The first lane data can be used to perform an image fusion operation between the vehicle and the second lane image collected by the vehicle to obtain a third lane image.

[0043] In one possible implementation, the first lane data may further include a first feature point matrix;

[0044] The method may also include the following steps:

[0045] Generate an RGBD image corresponding to the target device based on the original lane image corresponding to the target device;

[0046] The first feature point matrix is ​​extracted from the RGBD image corresponding to the target device. The first feature point matrix can be used by the vehicle to determine the target transpose matrix based on the first feature point matrix and the second feature point matrix corresponding to the vehicle. The target transpose matrix can be used by the vehicle to perform image fusion operation on the first lane image and the second lane image based on the target transpose matrix to obtain the third lane image.

[0047] In another possible implementation, the original lane image corresponding to the target device includes a third original image and a fourth original image;

[0048] Generating an RGBD image of the target device based on the original lane image corresponding to the target device may include the following steps:

[0049] The original images of the third lane and the fourth lane are corrected according to the preset distortion correction coefficients to obtain the third corrected image and the fourth corrected image.

[0050] Determine the second feature matching point between the third and fourth corrected images;

[0051] Generate a third matching image corresponding to the third corrected image based on the second feature matching point, and generate a fourth matching image corresponding to the fourth corrected image.

[0052] Generate the RGBD image corresponding to the target device based on the third and fourth matching images.

[0053] In another possible implementation, the method may further include the following steps:

[0054] Perform lane line recognition on the third corrected image to generate a second reference lane line image;

[0055] Perform coordinate mapping on the second reference lane line image to generate the first lane image.

[0056] In another possible implementation, generating the RGBD image corresponding to the target device based on the third and fourth matching images may include the following steps:

[0057] Determine the similarity transformation parameters between the third and fourth matching images;

[0058] The third matching image and similarity transformation parameters are input into a preset neural network to obtain a depth map and an RGB image.

[0059] The depth map is overlaid with the RGB image to generate the RGBD image corresponding to the target device.

[0060] Thirdly, this application provides a lane data fusion apparatus, which may include: a first communication module, a first image pickup, and a first computing module;

[0061] The first communication module can be used to receive first lane data sent by a target device within a preset range. The first lane data may include a first lane image.

[0062] The first image picker can be used to acquire the original image of the lane;

[0063] The first calculation module can be used to obtain the second lane image based on the original lane image;

[0064] The first calculation module can also be used to perform image fusion operations on the first lane image and the second lane image to obtain the third lane image.

[0065] In one possible implementation, the first lane data may further include a first feature point matrix, which is generated by the target device based on the RGBD image corresponding to the original lane image it acquires.

[0066] The device may also include:

[0067] The first calculation module can also be used to generate RGBD images based on the original lane images;

[0068] The first calculation module can also be used to extract the second feature point matrix of the RGBD image;

[0069] The first calculation module can also be used to determine the target transpose matrix based on the first feature point matrix and the second feature point matrix when the root mean square error between the first lane image and the second lane image is less than a preset value.

[0070] The first calculation module can also be used to perform image fusion operations on the first lane image and the second lane image based on the target transpose matrix to obtain the third lane image.

[0071] In another possible implementation, the first image picker may include a first camera and a second camera, and the original lane image may include the original lane image captured by the first camera and the original lane image captured by the second camera.

[0072] The device may also include:

[0073] The first calculation module can be used to correct the original image of the first lane and the original image of the second lane according to the preset distortion correction coefficient, so as to obtain the first corrected image and the second corrected image.

[0074] The first calculation module can also be used to determine the feature matching points of the first corrected image and the second corrected image;

[0075] The first calculation module can also be used to generate a first matching image corresponding to the first corrected image based on feature matching points, and to generate a second matching image corresponding to the second corrected image.

[0076] The first calculation module can also be used to generate an RGBD image based on the first matching image and the second matching image.

[0077] In another possible implementation, the device may further include:

[0078] The first calculation module can also be used to perform lane line recognition on the first corrected image to generate a first reference lane line image.

[0079] The first calculation module can also be used to perform coordinate mapping operations on the first reference lane line image to generate a second lane image.

[0080] In another possible implementation, the device may further include:

[0081] The first calculation module can also be used to determine the similarity transformation parameters between the first matching image and the second matching image;

[0082] The first calculation module can also be used to input the first matching image and similarity transformation parameters into a preset neural network to obtain a depth map and an RGB image.

[0083] The first calculation module can also be used to overlay the depth map with the RGB image to generate an RGBD image.

[0084] In another possible implementation, the device may further include:

[0085] The first calculation module can also be used to randomly select a preset number of values ​​from the first feature point matrix N times to generate N first matrices, where N is a positive integer and the preset number of groups is greater than or equal to 3.

[0086] The first calculation module can also be used to randomly select the same number of values ​​as the number of rows in the first matrix from the second feature point matrix N times to generate N second matrices;

[0087] The first calculation module can also be used to divide N first matrices and N second matrices into N groups in a one-to-one correspondence, and determine the reference transpose matrix of the second matrix in each group to be transformed into the first matrix;

[0088] The first calculation module can also be used to calculate the average square error of N reference transpose matrices;

[0089] The first calculation module can also be used to determine the target transpose matrix as the reference transpose matrix that minimizes the average square error.

[0090] In another possible implementation, the device may further include:

[0091] The first calculation module can also be used to rank the importance of various target devices;

[0092] The first calculation module can also be used to select a preset number of target devices based on their importance.

[0093] Fourthly, this application provides a target device, which may include a second image pickup, a second computing module, and a second communication module;

[0094] The second image picker can be used to acquire the original image of the lane corresponding to the target device;

[0095] The second calculation module can be used to obtain first lane data based on the original lane image corresponding to the target device. The first lane data may include the first lane image.

[0096] The second communication module can be used to send first lane data to vehicles within a preset range. The first lane data can be used to perform image fusion operations between the vehicle and the second lane image collected by the vehicle to obtain a third lane image.

[0097] In one possible implementation, the first lane data may further include a first feature point matrix;

[0098] The target device may also include:

[0099] The second calculation module can also be used to generate an RGBD image of the target device based on the original lane image of the target device.

[0100] The second calculation module can also be used to extract a first feature point matrix from the RGBD image corresponding to the target device. The first feature point matrix can be used by the vehicle to determine the target transpose matrix based on the first feature point matrix and the second feature point matrix corresponding to the vehicle. The target transpose matrix can be used by the vehicle to perform image fusion operation on the first lane image and the second lane image based on the target transpose matrix to obtain the third lane image.

[0101] In another possible implementation, the original lane image corresponding to the target device includes a third original image and a fourth original image;

[0102] The target device may also include:

[0103] The second calculation module can also be used to correct the original images of the third lane and the fourth lane according to the preset distortion correction coefficients to obtain the third corrected image and the fourth corrected image.

[0104] The second calculation module can also be used to determine the second feature matching points of the third and fourth corrected images;

[0105] The second calculation module can also be used to generate a third matching image corresponding to the third corrected image based on the second feature matching points, and to generate a fourth matching image corresponding to the fourth corrected image.

[0106] The second calculation module can also be used to generate an RGBD image corresponding to the target device based on the third and fourth matching images.

[0107] In another possible implementation, the target terminal may further include:

[0108] The second calculation module can also be used to perform lane line recognition on the third corrected image to generate a second reference lane line image.

[0109] The second calculation module can also be used to perform coordinate mapping operations on the second reference lane line image to generate the first lane image.

[0110] In another possible implementation, the target device may further include:

[0111] The second calculation module can also be used to determine the similarity transformation parameters between the third and fourth matching images;

[0112] The second calculation module can also be used to input the third matching image and similarity transformation parameters into a preset neural network to obtain a depth map and an RGB image.

[0113] The second calculation module can also be used to overlay the depth map with the RGB image to generate an RGBD image corresponding to the target device.

[0114] Fifthly, this application provides a processing apparatus, which may include the following components: a processor, a memory, and a bus;

[0115] The processor and memory are connected via a bus, wherein the memory is used to store a set of program code, and the processor is used to call the program code stored in the memory to execute the method described in the first aspect.

[0116] Sixthly, this application provides a computer-readable storage medium, comprising:

[0117] The computer-readable storage medium stores instructions that, when executed on a computer, implement the method described in the first aspect.

[0118] In a seventh aspect, this application provides a vehicle that may include a lane data fusion apparatus as described in the third aspect, a processing apparatus as described in the fifth aspect, and a computer-readable storage medium as described in the sixth aspect.

[0119] As can be seen, this application allows the vehicle to acquire lane data from other surrounding devices and filter out suitable lane data to fuse with its own collected lane data. This helps to fill in lane data in the vehicle's blind spot and allows the vehicle or customer to understand lane information at a greater distance, thereby improving the accuracy and reliability of lane data. Attached Figure Description

[0120] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0121] Figure 1 A flowchart illustrating a lane data fusion method provided in an embodiment of this application;

[0122] Figure 2 This application provides a schematic diagram of a scenario for acquiring lane data from a target device, as illustrated in an embodiment of the present application.

[0123] Figure 3 A flowchart illustrating a method for generating an RGBD image provided in an embodiment of this application;

[0124] Figure 4 A flowchart illustrating a method for determining a target transpose matrix provided in an embodiment of this application;

[0125] Figure 5 A schematic diagram of a scenario for feature point matrix registration provided in an embodiment of this application;

[0126] Figure 6 A schematic diagram illustrating a lane data transmission scenario provided in an embodiment of this application;

[0127] Figure 7 A schematic diagram illustrating the composition of a lane data fusion apparatus provided in an embodiment of this application;

[0128] Figure 8 A schematic diagram of another lane data fusion device provided in an embodiment of this application. Detailed Implementation

[0129] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0130] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0131] In this document, the term "embodiment" means that a particular feature, result, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0132] To better understand the technical solutions of the embodiments of this application, the following will be combined with Figure 1 This application provides a detailed description of a lane data fusion method according to an embodiment.

[0133] Please see Figure 1 This is a flowchart illustrating a lane data fusion method provided in an embodiment of this application. The method may include the following steps:

[0134] S101, the vehicle receives first lane data sent by a target device within a preset range.

[0135] It should be noted that the target device can use wireless communication technology to transmit lane data with the vehicle. The road test equipment can use the IEEE 802.11p protocol and / or 5G technology to transmit first requests and / or lane data with the vehicle. The vehicle can also use vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), and vehicle-to-network (V2N) communication technologies to transmit first requests and / or lane data with other vehicle-shaped target devices.

[0136] In one possible implementation, before the vehicle receives the first lane data sent by the target device, it may further include sending a first request to the target device within a preset range.

[0137] It should be noted that the first request can be used to obtain lane data from the target device, and the aforementioned vehicle may include a first image pickup. Specifically, the lane data may include the vehicle (referring to the vehicle that issued the first request, which will not be described again below) and / or a lane image generated by the target device based on the original lane image.

[0138] In the embodiments of this application, the target device can be a device with image pickup and data transmission functions, and its specific form can be other vehicles, roadside units (RSUs) (such as surveillance cameras located around the road), pedestrian mobile terminals, etc.

[0139] For example, if the preset range is a circular range with the vehicle as the center and a radius of 10m, the vehicle will send a first request to the target device located within the range, requesting the target device to send the lane data it has collected to the vehicle.

[0140] It should be noted that the number of target devices can be one or more. Target devices within the preset range that have communication or lane data sharing capabilities can receive the first request.

[0141] For example, Figure 2This is a schematic diagram illustrating a scenario for acquiring lane data from a target device, as provided in an embodiment of this application. Suppose vehicle 1 needs to generate a lane image of road 1 within a preset range of 10 meters. Vehicle 1 will send a first request to a target device (such as vehicle 2, vehicle 3, and camera 1) located within a circular area with a radius of 10 meters centered on vehicle 1. Since the distance between vehicle 4 and vehicle 1 exceeds 10 meters, the first request is sent to vehicle 4. In response to the first request from vehicle 1, the target device will transmit the lane data it has collected to vehicle 1.

[0142] S102, the vehicle acquires the original lane image through the first image pickup.

[0143] Specifically, the vehicle's first image acquisition device may include a first camera and a second camera, and the original lane image may include the original first lane image captured by the first camera and the original second lane image captured by the second camera. In this application, the first camera and the second camera can be distinguished using a left camera and a right camera. The first camera can be a main camera, and the second camera can be an auxiliary camera. Alternatively, the first camera can be an auxiliary camera, and the second camera can be a main camera; there is no limitation on this.

[0144] Optionally, the vehicle may control the first and second cameras to simultaneously capture raw images of the first lane and the second lane.

[0145] S103, the vehicle acquires a second lane image based on the original lane image.

[0146] More specifically, the second lane image can be generated based on the original first lane image captured by the first camera, which can be understood as the vehicle's main camera.

[0147] In one possible implementation, the generation of the second lane image may refer to the following steps:

[0148] The original image of the first lane is corrected according to the preset distortion correction coefficient to obtain the first corrected image;

[0149] Perform lane line recognition on the first corrected image to generate a first reference lane line image;

[0150] Perform coordinate mapping on the first reference lane line image to generate the second lane image.

[0151] It should be noted that images captured by different cameras will have varying degrees of distortion. Therefore, the vehicle needs to query its own database for the model information of the first and second cameras, and then retrieve the corresponding distortion correction coefficients from the database based on this model information. These distortion correction coefficients can be pre-saved by the manufacturer before the vehicle leaves the factory; alternatively, the camera model and distortion correction coefficient information in the database can be updated during vehicle use using Over-the-Air (OTA) technology.

[0152] It is possible that before correcting the original image of the first lane according to the distortion correction coefficient to obtain the first corrected image, image preprocessing can also be performed on the original image of the first lane. Image preprocessing may include image denoising and image enhancement, etc.

[0153] More specifically, distortion correction can include radial distortion correction and tangential distortion correction. The mathematical model for radial distortion correction can be found in the following formula:

[0154]

[0155] Where u is the x-coordinate of the original image of the first lane in the normalized plane, v is the y-coordinate of the original image of the first lane in the normalized plane, u′ represents the x-coordinate of the original image of the first lane after radial distortion correction in the normalized plane, v′ represents the y-coordinate of the original image of the first lane after radial distortion correction in the normalized plane, k1, k2, and k3 are all radial distortion correction coefficients, and r 2 =u 2 +v 2 .

[0156] The mathematical model for tangential distortion correction can be referenced by the following formula:

[0157]

[0158] Where u is the x-coordinate of the original image of the first lane in the normalized plane, v is the y-coordinate of the original image of the first lane in the normalized plane, u′ represents the x-coordinate of the original image of the first lane after tangential distortion correction in the normalized plane, v′ represents the y-coordinate of the original image of the first lane after radial shear correction in the normalized plane, p1 and p2 are both tangential distortion correction coefficients, and r 2 =u 2 +v 2 .

[0159] More specifically, the vehicle can input the first corrected image into the lane detection model, which will output a first reference lane image, which can be a 2D image. Then, the first reference lane image is mapped to obtain a second lane image. This can be understood as mapping the coordinates of the 2D image to a 3D coordinate system, thereby obtaining a 3D lane image.

[0160] As can be seen, the embodiments of this application can perform distortion correction processing on the original lane data collected by the vehicle, thereby obtaining second image data that is more consistent with the actual lane conditions, which helps to improve the accuracy and reliability of the lane image finally presented to the user or vehicle.

[0161] S104, perform an image fusion operation on the first lane image and the second lane image to obtain a third lane image.

[0162] In one possible implementation, the first lane data may further include a first feature point matrix, which is generated by the target device based on the RGBD image corresponding to the original lane image it acquires.

[0163] The method may also include the following steps:

[0164] Generate an RGBD image based on the original lane image;

[0165] Extract the second feature point matrix from the RGBD image;

[0166] When the root mean square error between the first lane image and the second lane image is less than a preset value, the target transpose matrix is ​​determined based on the first feature point matrix and the second feature point matrix.

[0167] The image fusion operation performed on the first lane image and the second lane image to obtain the third lane image may include:

[0168] The first lane image and the second lane image are fused based on the target transpose matrix to obtain the third lane image.

[0169] The target transpose matrix may include similarity transformation parameters that transform the first feature point matrix to the second feature point matrix through a geometric transformation. The similarity transformation parameters may include at least one of translation parameters, rotation parameters, and scaling parameters.

[0170] It should be noted that the first lane data may include the first lane image and the first feature point matrix.

[0171] More specifically, the feature point matrix may include the number, type, and location of feature points in the corresponding RGBD image. Specifically, the first feature point matrix may include the number, type, and location of feature points in the RGBD image corresponding to the original lane image acquired by the target device, and the second feature point matrix may include the number, type, and location of feature points in the RGBD image corresponding to the original lane image acquired by the vehicle. The specific form of the feature point matrix can be... Where x represents the number of feature points, y represents the type of feature points, and z represents the location of feature points.

[0172] Possibly, the feature points represented by the feature point matrix can be specific objects in the original lane image. These objects can be trees, lane lines, vehicles, pedestrians, utility poles, zebra crossings, green belts, or road barriers, etc., with different types corresponding to different numerical designations. It should be noted that the examples of object types represented by feature points above are merely for illustrating the embodiments of this application in more detail and should not be construed as limiting the scope of this application. This application does not limit the method for generating RGBD images; please refer to the following... Figure 3 A flowchart illustrating the method for generating RGBD images.

[0173] As can be seen, the embodiments of this application can acquire lane data from target devices within a preset range, which helps to expand the sources of lane data and supplement lane data in the blind spots of the vehicle, providing users and / or vehicles with lane images that are closer to the actual lane conditions. This application can also determine a target transpose matrix based on the lane data sent by the target device and the lane data collected by the vehicle, and then perform a fusion operation on the lane data acquired by the target device and the lane data collected by the vehicle according to the target transpose matrix. This helps to generate lane images that are more consistent with the actual situation, improving the accuracy and reliability of the lane images, thereby further enhancing the user experience.

[0174] In one possible implementation, the third lane is presented on the vehicle's display device.

[0175] like Figure 3 As shown, a method for generating an RGBD image may include the following steps:

[0176] S301, the original images of the first lane and the second lane are corrected according to the preset distortion correction coefficient to obtain the first corrected image and the second corrected image.

[0177] The specific execution details of steps S301-S302 can be found in the relevant embodiments of step S104 regarding the generation of the first corrected image, and will not be repeated here.

[0178] S302, determine the feature matching points of the first and second corrected images.

[0179] For example, if there are objects 1, 2 and 3 in the first corrected image and objects 1, 3 and 4 in the second corrected image, then the feature matching points in the first and second corrected images are the corresponding pixel points of objects 1 and 3 in the two images.

[0180] S303, Generate a first matching image corresponding to the first corrected image based on the feature matching points, and generate a second matching image corresponding to the second corrected image.

[0181] Possibly, by retaining the feature points (which can be understood as pixels belonging to the object's outline and within its outline) in the first corrected image and deleting other pixels, a first matching image can be generated. Similarly, by retaining the feature points in the second corrected image and deleting other pixels, a second matching image can be generated. For example, if the feature points representing the same object in the first and second corrected images are the pixels corresponding to object 1 and object 3, then deleting all pixels in the first corrected image except those corresponding to object 1 and object 3 will yield the first matching image; deleting all pixels in the second corrected image except those corresponding to object 1 and object 3 will yield the second matching image.

[0182] S304, Generate an RGBD image based on the first matching image and the second matching image.

[0183] It is possible that before correcting the original images of the first lane and the second lane according to the distortion correction coefficient to obtain the first corrected image and the second corrected image, image preprocessing can also be performed on the original images of the first lane and the second lane. Image preprocessing may include image denoising and image enhancement.

[0184] In one possible implementation, generating an RGBD image based on a first matching image and a second matching image may include the following steps:

[0185] Determine the similarity transformation parameters between the first matching image and the second matching image;

[0186] The first matching image and similarity transformation parameters are input into a preset neural network to obtain a depth map and an RGB image.

[0187] An RGBD image is generated by overlaying a depth map with an RGB image.

[0188] The matching information represents the similarity transformation parameters between the first matching image and the second matching image. The first matching image can be understood as the matching image corresponding to the original lane image captured by the vehicle's main camera.

[0189] Furthermore, mapping the depth information of the depth map to the brightness or color of the RGB image helps to better restore the structural information (such as height, width, and length) of objects on the lane.

[0190] As can be seen, the embodiments of this application can correct the acquired original lane images according to the model information of different cameras, which helps to improve the accuracy of the lane images. The embodiments of this application can also perform matching processing on the corrected images to extract feature points representing the same object in the two corrected images, and then generate an RGBD image. The simplified image helps to reduce the time required to determine the target transpose matrix of the lane data sent by the target device, and also reduces the influence of other factors (such as irrelevant distortion and image noise) on the generation of the target transpose matrix, thereby indirectly improving the reliability of the fused lane image to be closer to the lane.

[0191] More Figure 4 This is a flowchart illustrating a method for determining a target transpose matrix, provided in an embodiment of this application. The method may include the following steps:

[0192] S401, randomly select a preset number of values ​​from the first feature point matrix N times to generate N first matrices, where N is a positive integer.

[0193] The preset number of groups is greater than or equal to 3.

[0194] For example, let the first feature point matrix be S. S The first matrix is ​​represented by G. S express.

[0195] S402, randomly select the same number of values ​​as the number of rows in the first matrix from the second feature point matrix N times to generate N second matrices.

[0196] For example, let the second feature point matrix be S. R The second matrix is ​​represented by G. R express.

[0197] S403, divide the N first matrices and N second matrices into N groups in a one-to-one correspondence, and determine the reference transpose matrix for transforming the second matrix into the first matrix in each group.

[0198] For example, point A(x,y,z) in the second matrix can be transformed to the coordinate system of the first matrix using the transpose matrix, resulting in the coordinates of point A′ as (x′,y′,z′). The calculation process for this coordinate system transformation can be referenced in the following formula:

[0199]

[0200] S404, calculate the average squared error of N reference transpose matrices.

[0201] Understandably, the reference transpose matrix will contain rotation parameters, translation parameters, and scaling parameters, and correspondingly, there can be a rotation matrix and a translation vector.

[0202] It should be noted that here we need to calculate the minimum average squared error of each reference transpose matrix. For example, determining the e-values ​​between the reference transpose matrices... 2 The minimum mean square error of (R,t,c) can be obtained by referring to the following steps, wherein,

[0203]

[0204] Where R represents the rotation matrix corresponding to the rotation parameters in the reference transpose matrix, t represents the translation vector corresponding to the translation parameters in the reference transpose matrix, c represents the scaling parameter in the reference transpose matrix, and x i Represents the first matrix G S The column vector, y i Represents the second matrix G R The column vectors, where n represents the first matrix G. S Second matrix G R The number of columns.

[0205] Existence Lemma: Let A and B be m×n matrices, and let there exist an m×m rotation matrix R between matrices A and B. T (UU T =VV T =I,D=diag(d i ), d1≥d2≥…≥d m The singular value decomposition of (≥0) is UDV T (where UU) T =VV T =I,D=diag(d i ), d1≥d2≥…≥d m ≥0, U is an m-order orthogonal matrix, V is an n-order orthogonal matrix). Then ||A-RB|| 2 The minimum value relative to R is:

[0206] min R ‖A-RB‖ 2 =|A| 2 +‖B‖ 2 -2tr(DS),

[0207]

[0208] Existence Theorem: Let X = (x1, x2, ..., x...) n ) = G Sand Y = (y1, y2, ..., y n ) = G R It is possible

[0209]

[0210] Then the minimum mean square error ε of these two matrices X and Y relative to the similarity transformation parameters is... 2 as follows:

[0211]

[0212] in,

[0213]

[0214]

[0215]

[0216]

[0217]

[0218] Where ∑xy is the covariance matrix of X and Y, u x and u y It is the mean vector of X and Y. and It is the variance around the mean vector of X and Y.

[0219] The reasoning is as follows: Let X = (x1, x2, ..., x n ) = G S and Y = (y1, y2, ..., y n ) = G R Then, using an m×n matrix to represent X and Y, we get:

[0220]

[0221] h = (1, 1, ..., 1) T ,

[0222] Here we introduce an n×n normalization matrix. (K 2 =K T =K), therefore:

[0223]

[0224]

[0225]

[0226] at the same time:

[0227]

[0228]

[0229] so:

[0230]

[0231]

[0232] Therefore, we can obtain...

[0233] tr(K(Y T -cx T R T )t′h T ) = 0,

[0234] ||t′h T || 2 =n||t′|| 2 ,

[0235] have to

[0236]

[0237] In this equation, t′ needs to be equal to 0 in order to minimize e. 2 (R,t,c)

[0238]

[0239] Next, when UDV T It is ∑xy=(1 / nYKX) T The singular value decomposition of cnUDV T It is YK (cXK) T =cYKK T X T =cYKX T Therefore, the minimum value ε of R. 2 (c)=(1 / n)‖YK-cRXK‖ 2 Given by the following lemma:

[0240]

[0241]

[0242] From the lemma above, if rank(∑xy)≥m-1, then R=USV T ,

[0243] If rank(∑xy)=m-1, then

[0244]

[0245] Finally, the minimum ε can be obtained. 2 yes

[0246]

[0247] S405, the reference transpose matrix with the minimum mean square error is determined as the target transpose matrix.

[0248] Specifically, based on the minimum mean square error of each reference transpose matrix obtained in step S404, each reference transpose matrix is ​​screened, and the reference transpose matrix with the smallest mean square error is determined as the target transpose matrix.

[0249] in, Figure 5 This is a schematic diagram of a scenario for feature point matrix registration provided in an embodiment of this application.

[0250] The first and second matrices, containing three sets of values, can be viewed as triangles ABC and DEF respectively (assuming triangles ABC and DEF are of the same size). The process of transforming triangle ABC to triangle DEF, using the coordinate system of triangle DEF corresponding to the second matrix, is called feature point matrix registration. Figure 5 As shown, triangle ABC can be rotated first to obtain the rotation parameters, and then translated to obtain the translation parameters. Here, since triangles ABC and DEF are triangles of the same size, the scaling parameter can be considered as 1 (i.e., neither scaling nor shrinking is performed).

[0251] It should be noted that if the triangles corresponding to the first matrix and the triangles corresponding to the second matrix are not the same size, the final registration result is not necessarily two triangles of the same size, but two similar triangles.

[0252] in, Figure 6 This is a schematic diagram of a lane data transmission scenario provided in an embodiment of this application.

[0253] set up Figure 6 The specific type of the target device is a vehicle.

[0254] The target device acquires original lane images via a second image pickup. Specifically, the target device simultaneously acquires original images 611 (third lane) and 612 (fourth lane) via a left camera (designated as the main camera) and a right camera. Distortion correction processing is applied to these images to obtain corrected images 613 and 614. Image matching processing is then performed on these images to obtain matched images 615 and 6141. The similarity transformation parameters of matched images 615 and 6141 are input into a neural network to obtain a depth map 616 and an RGB image 617. Overlaying the depth map 616 and the RGB image 617 generates an RGBD image 618, and feature point extraction is performed on the RGBD image 618 to generate a first feature point matrix 619. Furthermore, the target device will perform lane line recognition on the third corrected image 613 to obtain a 2D lane image 6110. Then, a coordinate mapping operation will be performed on the 2D lane image 6110 to obtain a 3D lane image 6111 (equivalent to the first lane image mentioned above). Finally, after receiving the first request from the vehicle (i.e., the vehicle that sent the first request to the target device), the target device will send the first feature point matrix 619 and the 3D lane image 6111 to the vehicle (i.e., the vehicle that sent the first request to the target device).

[0255] The vehicle (i.e., the vehicle sending the first request to the target device) simultaneously acquires the first lane's original image 621 and the second lane's original image 622 via the left camera (designated as the main camera) and the right camera. Distortion correction processing is performed on the first lane's original image 621 and the second lane's original image 622 to obtain the first corrected image 623 and the second corrected image 624. Subsequently, image matching processing is performed on the first corrected image 623 and the second corrected image 624 to obtain the first matched image 625 and the second matched image 6241. The similarity transformation parameters of the first matched image 625 and the second matched image 6241 are input into a neural network to obtain a depth map 626 and an RGB image 627. Overlaying the depth map 626 and the RGB image 627 generates an RGBD image 628, and feature point extraction is performed on the RGBD image 628 to generate a second feature point matrix 629. Furthermore, the vehicle (i.e., the vehicle that sends the first request to the target device) will also perform lane line recognition on the first corrected image 623 to obtain a 2D lane image 6210. Then, a coordinate mapping operation is performed on the 2D lane image 6210 to obtain a 3D lane image 6211 (equivalent to the second lane image mentioned above).

[0256] Furthermore, after receiving the first feature point matrix 619 and the 3D lane image 6111, the vehicle (i.e., the vehicle that sent the first request to the target device) performs registration processing on the first feature point matrix 619 and the second feature point matrix 629 in the registration module 6212 to obtain similarity transformation parameters from the geometric transformation of the first feature point matrix 619 to the second feature point matrix 629. Based on these similarity transformation parameters, the target transpose matrix 6213 can be obtained. Finally, the 3D lane image 6111 and the 3D lane image 6212 are fused based on the target transpose matrix 6213 to obtain the third lane image 6214.

[0257] based on Figures 1-6 In another possible implementation, according to a related embodiment, the method of this application may further include the following steps:

[0258] If the number of target devices is greater than or equal to 2, the importance of each target device is ranked.

[0259] Select a preset number of target devices based on their importance.

[0260] Possibly, this application can rank the importance of each target device based on the mean square error of the target transpose matrix corresponding to the target device, and / or the distance between the target device and the vehicle. Specifically, the smaller the mean square error of the target transpose matrix corresponding to the target device, the higher the importance of the target device; the smaller the distance between the target device and the vehicle, the higher the importance of the target device.

[0261] In another possible implementation, the processing of the original lane image acquired by the target device (i.e., generating the first lane image and the first feature point matrix) can refer to the processing flow of the vehicle-related embodiments in this application, which will not be repeated here.

[0262] The apparatus involved in the embodiments of this application is described below with reference to the accompanying drawings.

[0263] Please see Figure 7 The present application provides a lane data fusion device, which may include: a first communication module 710, a first image pickup 720 and a first calculation module 730.

[0264] The first communication module 710 can be used to receive first lane data sent by a target device within a preset range. The first lane data may include a first lane image.

[0265] The first image picker 720 can be used to acquire the original image of the lane;

[0266] The first calculation module 730 can be used to obtain the second lane image based on the original lane image;

[0267] The first calculation module 730 can also be used to perform image fusion operations on the first lane image and the second lane image to obtain the third lane image.

[0268] In one possible implementation, the first lane data may further include a first feature point matrix, which is generated by the target device based on the RGBD image corresponding to the original lane image it acquires.

[0269] The device may also include:

[0270] The first calculation module 730 can also be used to generate an RGBD image based on the original lane image;

[0271] The first calculation module 730 can also be used to extract the second feature point matrix of the RGBD image;

[0272] The first calculation module 730 can also be used to determine the target transpose matrix based on the first feature point matrix and the second feature point matrix when the root mean square error between the first lane image and the second lane image is less than a preset value.

[0273] The first calculation module 730 can also be used to perform image fusion operation on the first lane image and the second lane image based on the target transpose matrix to obtain the third lane image.

[0274] In another possible implementation, the first image picker may include a first camera and a second camera, and the original lane image may include the original lane image captured by the first camera and the original lane image captured by the second camera.

[0275] The device may also include:

[0276] The first calculation module 730 can be used to perform correction processing on the original image of the first lane and the original image of the second lane according to the preset distortion correction coefficient, so as to obtain the first corrected image and the second corrected image.

[0277] The first calculation module 730 can also be used to determine the feature matching points of the first corrected image and the second corrected image;

[0278] The first calculation module 730 can also be used to generate a first matching image corresponding to the first corrected image based on feature matching points, and to generate a second matching image corresponding to the second corrected image.

[0279] The first calculation module 730 can also be used to generate an RGBD image based on the first matching image and the second matching image.

[0280] In another possible implementation, the device may further include:

[0281] The first calculation module 730 can also be used to perform lane line recognition operation on the first corrected image to generate a first reference lane line image.

[0282] The first calculation module 730 can also be used to perform coordinate mapping operations on the first reference lane line image to generate a second lane image.

[0283] In another possible implementation, the device may further include:

[0284] The first calculation module 730 can also be used to determine similarity transformation parameters between the first matching image and the second matching image;

[0285] The first calculation module 730 can also be used to input the first matching image and similarity transformation parameters into a preset neural network to obtain a depth map and an RGB image.

[0286] The first calculation module 730 can also be used to overlay the depth map with the RGB image to generate an RGBD image.

[0287] In another possible implementation, the device may further include:

[0288] The first calculation module 730 can also be used to randomly select a preset number of values ​​from the first feature point matrix N times to generate N first matrices, where N is a positive integer and the preset number of groups is greater than or equal to 3.

[0289] The first calculation module 730 can also be used to randomly select the same number of values ​​as the number of rows in the first matrix from the second feature point matrix N times to generate N second matrices;

[0290] The first calculation module 730 can also be used to divide N first matrices and N second matrices into N groups in a one-to-one correspondence, and determine the reference transpose matrix of the second matrix in each group to be transformed into the first matrix;

[0291] The first calculation module 730 can also be used to calculate the average square error of N reference transpose matrices;

[0292] The first calculation module 730 can also be used to determine the reference transpose matrix with the minimum average square error as the target transpose matrix.

[0293] In another possible implementation, the device may further include:

[0294] The first calculation module 730 can also be used to prioritize the importance of various target devices.

[0295] The first calculation module 730 can also be used to select a preset number of target devices based on their importance.

[0296] Please see Figure 8This is a schematic diagram illustrating the composition of another lane data fusion apparatus provided in this application embodiment. The apparatus may include:

[0297] The processor 810, memory 820, and I / O interface 830 are communicatively connected. The memory 820 stores instructions, and the processor 810 executes the instructions stored in the memory 820 to achieve the above-mentioned functions. Figure 1 , Figure 3 as well as Figure 4 The corresponding methods and steps.

[0298] The processor 810 executes the instructions stored in the memory 820 to control the I / O interface 830 to receive and send signals, thus completing the steps in the above method. The memory 820 may be integrated into the processor 810 or may be disposed separately from the processor 810.

[0299] The memory 820 may also include a storage system 821, a cache 822, and RAM 823. The cache 822 is a primary memory located between the RAM 823 and the CPU, composed of static RAM chips (SRAM). It has a relatively small capacity but a much higher speed than main memory, approaching the speed of the CPU. The RAM 823 is an internal memory that directly exchanges data with the CPU. It can be read and written at any time (except during refresh) and is very fast, typically serving as temporary data storage for the operating system or other running programs. The three components work together to realize the function of the memory 820.

[0300] As one implementation approach, the functionality of the I / O interface 830 can be implemented using transceiver circuitry or dedicated transceiver chips. The processor 810 can be implemented using dedicated processing chips, processing circuitry, processors, or general-purpose chips.

[0301] As another implementation method, the apparatus provided in this application embodiment can be implemented using a general-purpose computer. The program code that implements the functions of processor 810 and I / O interface 830 is stored in memory 820, and the general-purpose processor implements the functions of processor 810 and I / O interface 830 by executing the code in memory 820.

[0302] For the concepts, explanations, detailed descriptions, and other steps related to the technical solutions provided in the embodiments of this application, please refer to the descriptions of the method steps performed by the device in the foregoing method or other embodiments, which will not be repeated here.

[0303] As another implementation of this embodiment, a computer-readable storage medium is provided, on which instructions are stored, which, when executed, perform the methods in the above-described method embodiments.

[0304] As another implementation of this embodiment, a computer program product containing instructions is provided, which, when executed, perform the method in the above method embodiment.

[0305] Those skilled in the art will understand that, for ease of explanation, Figure 8 Only one memory and processor are shown in the illustration. In a real terminal or server, multiple processors and memories may exist. Memory can also be called storage medium or storage device, etc., and this application does not limit this.

[0306] It should be understood that in the embodiments of this application, the processor may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0307] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0308] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) is integrated into the processor.

[0309] It should be noted that the memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0310] In addition to the data bus, this bus may also include a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled "bus" in the diagram.

[0311] It should also be understood that the first, second, third, fourth and various numerical designations used herein are merely for descriptive convenience and are not intended to limit the scope of this application.

[0312] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0313] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0314] In the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0315] Those skilled in the art will recognize that the various illustrative logical blocks (ILBs) and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0316] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0317] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0318] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0319] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0320] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement some or all of the steps of any of the methods for managing wireless connections based on in-vehicle devices as described in the above method embodiments.

[0321] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments for managing wireless connections based on in-vehicle devices.

[0322] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for lane data fusion, characterized in that, The method is applied to a vehicle, the vehicle including a first image pickup, and the method includes the following steps: The vehicle receives first lane data sent by a target device within a preset range. The first lane data includes a first lane image and a first feature point matrix. The first feature point matrix is ​​generated by the target device based on the RGBD image corresponding to the original lane image it has acquired. The vehicle acquires the original lane image through the first image pickup; The vehicle acquires a second lane image based on the original lane image; A third lane image is obtained by performing an image fusion operation on the first lane image and the second lane image; The method further includes the following steps: Generate an RGBD image based on the original lane image; Extract the second feature point matrix from the RGBD image; When the root mean square error between the first lane image and the second lane image is less than a preset value, the target transpose matrix is ​​determined based on the first feature point matrix and the second feature point matrix. The step of performing an image fusion operation on the first lane image and the second lane image to obtain the third lane image includes: The first lane image and the second lane image are fused according to the target transpose matrix to obtain the third lane image; Determining the target transpose matrix based on the first feature point matrix and the second feature point matrix includes the following steps: N values ​​are randomly selected from the first feature point matrix in N rounds to generate N first matrices, where N is a positive integer and the preset number of groups is greater than or equal to 3. N values ​​are randomly selected from the second feature point matrix in N steps, with the same number of groups as the number of rows in the first matrix, to generate N second matrices; Divide the N first matrices and the N second matrices into N groups in a one-to-one correspondence, and determine the reference transpose matrix of the second matrix in each group to transform into the first matrix; Calculate the average square error of the N reference transpose matrices; The reference transpose matrix that minimizes the average squared error is determined as the target transpose matrix.

2. The method according to claim 1, characterized in that, The first image pickup includes a first camera and a second camera, and the original lane image includes the original lane image captured by the first camera and the original lane image captured by the second camera. The process of generating an RGBD image based on the original lane image includes the following steps: The original images of the first lane and the second lane are corrected according to a preset distortion correction coefficient to obtain a first corrected image and a second corrected image. Determine the feature matching points of the first corrected image and the second corrected image; A first matching image corresponding to the first corrected image is generated based on the feature matching points, and a second matching image corresponding to the second corrected image is generated. The RGBD image is generated based on the first matching image and the second matching image.

3. The method according to claim 2, characterized in that, The method further includes the following steps: Perform lane line recognition on the first corrected image to generate a first reference lane line image; The first reference lane image is subjected to coordinate mapping to generate the second lane image.

4. The method according to claim 2 or 3, characterized in that, The step of generating an RGBD image based on the first matching image and the second matching image includes the following steps: Determine the similarity transformation parameters between the first matching image and the second matching image, and input the first matching image and the similarity transformation parameters into a preset neural network to obtain a depth map and an RGB image; The depth map is overlaid with the RGB image to generate the RGBD image.

5. The method according to claim 4, characterized in that, The method further includes the following steps: When the number of target devices is greater than or equal to 2, the importance of each target device is ranked. A preset number of target devices are selected based on their importance.

6. A method for lane data fusion, characterized in that, The method is applied to a target device, the target device including a second image pickup, and the method includes the following steps: The target device acquires the original lane image corresponding to the target device through the second image pickup; The target device acquires first lane data based on the original lane image corresponding to the target device. The first lane data includes a first lane image and a first feature point matrix. The first feature point matrix is ​​generated by the target device based on the RGBD image corresponding to the original lane image it acquires. The target device sends the first lane data to vehicles within a preset range. The first lane data is used by the vehicles to perform an image fusion operation on the first lane image and the second lane image collected by the vehicles to obtain a third lane image. The method further includes the following steps: Generate an RGBD image corresponding to the target device based on the original lane image corresponding to the target device; The first feature point matrix is ​​extracted from the RGBD image corresponding to the target device. When the root mean square error between the first lane image and the second lane image is less than a preset value, the target transpose matrix is ​​determined based on the first feature point matrix and the second feature point matrix corresponding to the vehicle. The step of performing an image fusion operation on the first lane image and the second lane image to obtain a third lane image includes: performing an image fusion operation on the first lane image and the second lane image based on the target transpose matrix to obtain the third lane image; the step of determining the target transpose matrix based on the first feature point matrix and the second feature point matrix... The transpose matrix includes the following steps: N times, randomly selecting a predetermined number of values ​​from the first feature point matrix to generate N first matrices, where N is a positive integer and the predetermined number of groups is greater than or equal to 3; N times, randomly selecting a number of values ​​from the second feature point matrix with the same number of rows as the first matrices to generate N second matrices; dividing the N first matrices and the N second matrices into N groups, and determining a reference transpose matrix for each group to be transformed from the second matrix to the first matrix; calculating the average square error of the N reference transpose matrices; and determining the reference transpose matrix with the smallest average square error as the target transpose matrix.

7. The method according to claim 6, characterized in that, The original lane images corresponding to the target device include the original images of the third lane and the fourth lane. The step of generating the RGBD image corresponding to the target device based on the original lane image corresponding to the target device includes the following steps: The original images of the third lane and the fourth lane are corrected according to the preset distortion correction coefficient to obtain the third corrected image and the fourth corrected image. Determine the feature matching points of the third corrected image and the fourth corrected image; A third matching image corresponding to the third corrected image is generated based on the feature matching points, and a fourth matching image corresponding to the fourth corrected image is generated. An RGBD image corresponding to the target device is generated based on the third matching image and the fourth matching image.

8. The method according to claim 7, characterized in that, The method further includes the following steps: Perform lane line recognition operation on the third corrected image to generate a second reference lane line image; The second reference lane image is subjected to coordinate mapping to generate the first lane image.

9. The method according to claim 7 or 8, characterized in that, The step of generating the RGBD image corresponding to the target device based on the third matching image and the fourth matching image includes the following steps: Determine the similarity transformation parameters between the third matching image and the fourth matching image; The third matching image and the similarity transformation parameters are input into a preset neural network to obtain a depth map and an RGB image. The depth map is superimposed on the RGB image to generate an RGBD image corresponding to the target device.

10. A device for lane data fusion, characterized in that, The device includes: a first communication module, a first image pickup, and a first computing module; The first communication module is used to receive first lane data sent by a target device within a preset range. The first lane data includes a first lane image and a first feature point matrix. The first feature point matrix is ​​generated by the target device based on the RGBD image corresponding to the original lane image it has acquired. The first image picker is used to acquire the original image of the lane; The first calculation module is used to obtain a second lane image based on the original lane image obtained by the first image picker; The first calculation module is further configured to perform an image fusion operation on the first lane image and the second lane image to obtain a third lane image; The lane data fusion device also includes: The first calculation module is further configured to generate an RGBD image based on the original lane image acquired by the first image picker; The first calculation module is also used to extract the second feature point matrix of the RGBD image; The first calculation module is further configured to determine the target transpose matrix based on the first feature point matrix and the second feature point matrix when the root mean square error between the first lane image and the second lane image is less than a preset value. The lane data fusion device also includes: The first calculation module is further configured to randomly select a preset number of values ​​from the first feature point matrix N times to generate N first matrices, where N is a positive integer and the preset number of groups is greater than or equal to 3. The first calculation module is also used to randomly select the same number of values ​​as the number of rows in the first matrix from the second feature point matrix N times to generate N second matrices; The first calculation module is further configured to divide the N first matrices and the N second matrices into N groups in a one-to-one correspondence, and determine the reference transpose matrix of the second matrix in each group to be transformed into the first matrix; The first calculation module is also used to calculate the average square error of the N reference transpose matrices; The first calculation module is further configured to determine the reference transpose matrix with the minimum average squared error as the target transpose matrix.

11. A target device for acquiring lane images, characterized in that, The target device includes: a second image pickup, a second computing module, and a second communication module; The second image picker is used to acquire the original lane image corresponding to the target device; The second calculation module is used to obtain first lane data based on the original lane image corresponding to the target device. The first lane data includes a first lane image and a first feature point matrix. The first feature point matrix is ​​generated by the target device based on the RGBD image corresponding to the original lane image it has obtained. The second communication module is used to send first lane data to vehicles within a preset range. The first lane data is used by the vehicle to perform an image fusion operation on the first lane image and the second lane image collected by the vehicle to obtain a third lane image. The target device for acquiring lane images also includes: The second calculation module is further configured to generate an RGBD image corresponding to the target device based on the original lane image corresponding to the target device; The second calculation module is further configured to extract the first feature point matrix based on the RGBD image corresponding to the target device; when the root mean square error between the first lane image and the second lane image is less than a preset value, determine the target transpose matrix based on the first feature point matrix and the second feature point matrix corresponding to the vehicle; the step of performing an image fusion operation on the first lane image and the second lane image to obtain a third lane image includes: performing an image fusion operation on the first lane image and the second lane image based on the target transpose matrix to obtain the third lane image; the step of performing an image fusion operation on the first lane image and the second lane image based on the first feature point matrix and the second feature point matrix... The method for determining the target transpose matrix includes the following steps: N times, randomly selecting a predetermined number of values ​​from the first feature point matrix to generate N first matrices, where N is a positive integer and the predetermined number of groups is greater than or equal to 3; N times, randomly selecting a number of values ​​from the second feature point matrix with the same number of rows as the first matrices to generate N second matrices; dividing the N first matrices and the N second matrices into N groups, and determining the reference transpose matrix for transforming the second matrix into the first matrix in each group; calculating the average square error of the N reference transpose matrices; and determining the reference transpose matrix with the smallest average square error as the target transpose matrix.

12. A processing apparatus for performing a method of lane data fusion, characterized in that, include: The processor, memory, and I / O interface are communicatively connected, wherein the memory is used to store a set of program code, and the processor is used to call the program code stored in the memory to perform the method as described in any one of claims 1-5, or to perform the method as described in any one of claims 6-9.

13. A computer-readable storage medium, characterized in that, include: The computer-readable storage medium stores instructions that, when executed on a computer, implement the method as described in any one of claims 1-5, or the method as described in any one of claims 6-9.

14. A vehicle, characterized in that, The vehicle includes the lane data fusion apparatus as claimed in claim 10, the processing apparatus for performing the lane data fusion method as claimed in claim 12, and the computer-readable storage medium as claimed in claim 13.

Images