Lane data fusion method and device and related medium

Abstract

The embodiment of the invention discloses a lane data fusion method and device and a related medium, and the method comprises the following steps: a vehicle receives first lane data sent by a target device in a preset range, and the first lane data comprises a first lane image; the vehicle obtains an original image of the lane through the first image pickup device; the vehicle obtains a second lane image according to the lane original image; and carrying out image fusion operation on the first lane image and the second lane image to obtain a third lane image. By implementing the method provided by the embodiment of the invention, the vehicle can reasonably select the source of the lane data, and the appropriate source of the lane data and the lane data acquired by the vehicle can be fused, so that the lane data in the visual angle blind area of the vehicle can be supplemented, the vehicle or a customer can know the lane information in a longer distance, and the user experience is improved. Therefore, the correctness and reliability of the lane data are improved.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a lane data fusion method, device and related media. Background Art

[0002] With the development of science and technology, all walks of life have improved their products in the direction of intelligence. For example, more and more automobile manufacturers are focusing on driverless technology. The realization of driverless technology is inseparable from the lane keeping function and automatic cruise function, and lane data information is the basis for the realization of these functions. The existing technology can collect lane data information through cameras and / or sensors on the vehicle, but this method is limited by the location of the camera and sensor and has problems of blind spots and short visual distance, resulting in low reliability of lane data information transmitted to users and / or vehicles. When users and / or vehicles refer to this lane data information during driving, there may be safety hazards.

[0003] Therefore, how to provide a method for generating reliable lane data is a problem that those skilled in the art need to solve urgently. Summary of the invention

[0004] One purpose of the present application is to provide a method, device and related media for lane data fusion, which can fuse appropriate lane data sources with lane data acquired by the vehicle, help supplement lane data in the blind spot of the vehicle's field of view, and help the vehicle or customers understand lane information at a longer distance, thereby improving the accuracy and reliability of lane data.

[0005] In a first aspect, the present application provides a method for lane data fusion, which may include the following steps:

[0006] The vehicle receives first lane data sent by a target device within a preset range, where the first lane data may include a first lane image;

[0007] The vehicle acquires an original lane image through a first image pickup device;

[0008] The vehicle obtains a second lane image based on the original lane image;

[0009] An image fusion operation is performed on the first lane image and the second lane image to obtain a third lane image.

[0010] In a possible implementation, the first lane data may further include a first feature point matrix, which is generated by the target device according to an RGBD image corresponding to the lane original image acquired by the target device;

[0011] The method may further comprise the following steps:

[0012] Generate an RGBD image based on the original lane image;

[0013] Extract the second feature point matrix of the RGBD image;

[0014] When a root mean square error between the first lane image and the second lane image is less than a preset value, determining a target transposed matrix according to the first feature point matrix and the second feature point matrix;

[0015] Performing an image fusion operation on the first lane image and the second lane image to obtain a third lane image may include:

[0016] An image fusion operation is performed on the first lane image and the second lane image according to the target transposed matrix to obtain a third lane image.

[0017] In another possible implementation, the first image pickup device may include a first camera and a second camera, and the lane original image may include a first lane original image captured by the first camera and a second lane original image captured by the second camera;

[0018] Generating an RGBD image based on the original lane image may include the following steps:

[0019] Correcting the original image of the first lane and the original image of the second lane according to a preset distortion correction coefficient to obtain a first corrected image and a second corrected image;

[0020] Determining feature matching points of the first rectified image and the second rectified image;

[0021] generating a first matching image corresponding to the first corrected image according to the feature matching points, and generating a second matching image corresponding to the second corrected image;

[0022] An RGBD image is generated according to the first matching image and the second matching image.

[0023] In another possible implementation, the method may further include the following steps:

[0024] Performing a lane line recognition operation on the first corrected image to generate a first reference lane line image;

[0025] A coordinate mapping operation is performed on the first reference lane line image to generate a second lane image.

[0026] In another possible implementation, generating an RGBD image according to the first matching image and the second matching image may include the following steps:

[0027] determining a similarity transformation parameter between the first matching image and the second matching image;

[0028] Inputting the first matching image and the similarity transformation parameters into a preset neural network to obtain a depth map and an RGB image;

[0029] The depth map is superimposed with the RGB image to generate an RGBD image.

[0030] In another possible implementation, determining a target transposed matrix according to the first feature point matrix and the second feature point matrix may include the following steps:

[0031] Randomly select a preset number of groups of values ​​from the first feature point matrix N times to generate N first matrices, where N is a positive integer and the preset number of groups is greater than or equal to 3;

[0032] Randomly select from the second feature point matrix N times the same number of values ​​as the number of rows in the first matrix, to generate N second matrices;

[0033] Divide the N first matrices and the N second matrices into N groups in a one-to-one correspondence, and determine a reference transposed matrix for transforming the second matrix into the first matrix in each group;

[0034] Calculate the mean square error of N reference transposed matrices;

[0035] The reference transposed matrix with the smallest mean square error is determined as the target transposed matrix.

[0036] In another possible implementation, the method may further include the following steps:

[0037] When the number of target devices is greater than or equal to 2, ranking the respective target devices in order of importance;

[0038] A preset number of target devices are selected according to importance.

[0039] In a second aspect, the present application provides a lane data fusion method, which is applied to a target device, the target device including a second image picker, and the method may include the following steps:

[0040] The target device acquires an original image of a lane corresponding to the target device through a second image pickup device;

[0041] The target device acquires first lane data according to the original lane image corresponding to the target device, and the first lane data may include a first lane image;

[0042] The target device sends the first lane data to the vehicle within a preset range, and the first lane data can be used by the vehicle to perform an image fusion operation on the first lane image and the second lane image collected by the vehicle to obtain a third lane image.

[0043] In a possible implementation, the first lane data may further include a first feature point matrix;

[0044] The method may further comprise the following steps:

[0045] Generate an RGBD image corresponding to the target device according to the original lane image corresponding to the target device;

[0046] A first feature point matrix is ​​extracted based on the RGBD image corresponding to the target device. The first feature point matrix can be used by the vehicle to determine a target transposed matrix based on the first feature point matrix and a second feature point matrix corresponding to the vehicle. The target transposed matrix can be used by the vehicle to perform an image fusion operation on the first lane image and the second lane image based on the target transposed matrix to obtain a third lane image.

[0047] In another possible implementation manner, the lane original image corresponding to the target device includes a third original image and a fourth original image;

[0048] Generating an RGBD image corresponding to the target device according to the original lane image corresponding to the target device may include the following steps:

[0049] Correcting the original image of the third lane and the original image of the fourth lane according to a preset distortion correction coefficient to obtain a third corrected image and a fourth corrected image;

[0050] Determining second feature matching points of the third corrected image and the fourth corrected image;

[0051] generating a third matching image corresponding to the third corrected image according to the second feature matching points, and generating a fourth matching image corresponding to the fourth corrected image;

[0052] An RGBD image corresponding to the target device is generated according to the third matching image and the fourth matching image.

[0053] In another possible implementation, the method may further include the following steps:

[0054] Performing a lane line recognition operation on the third corrected image to generate a second reference lane line image;

[0055] A coordinate mapping operation is performed on the second reference lane line image to generate a first lane image.

[0056] In another possible implementation, generating an RGBD image corresponding to the target device according to the third matching image and the fourth matching image may include the following steps:

[0057] determining similarity transformation parameters between the third matching image and the fourth matching image;

[0058] Inputting the third matching image and the similarity transformation parameters into a preset neural network to obtain a depth map and an RGB image;

[0059] The depth map is superimposed on the RGB image to generate an RGBD image corresponding to the target device.

[0060] In a third aspect, the present application provides a lane data fusion device, which may include: a first communication module, a first image pickup device, and a first calculation module;

[0061] A first communication module may be used to receive first lane data sent by a target device within a preset range, where the first lane data may include a first lane image;

[0062] A first image picker can be used to obtain an original lane image;

[0063] A first calculation module can be used to obtain a second lane image according to the original lane image;

[0064] The first calculation module can also be used to perform an image fusion operation on the first lane image and the second lane image to obtain a third lane image.

[0065] In a possible implementation, the first lane data may further include a first feature point matrix, which is generated by the target device according to an RGBD image corresponding to the lane original image acquired by the target device;

[0066] The device may also include:

[0067] The first calculation module can also be used to generate an RGBD image based on the original lane image;

[0068] The first calculation module can also be used to extract a second feature point matrix of the RGBD image;

[0069] The first calculation module may also be used to determine a target transposed matrix according to the first feature point matrix and the second feature point matrix when a root mean square error between the first lane image and the second lane image is less than a preset value;

[0070] The first calculation module can also be used to perform an image fusion operation on the first lane image and the second lane image according to the target transposed matrix to obtain a third lane image.

[0071] In another possible implementation, the first image pickup device may include a first camera and a second camera, and the lane original image may include a first lane original image captured by the first camera and a second lane original image captured by the second camera;

[0072] The device may also include:

[0073] A first calculation module may be used to perform correction processing on the first lane original image and the second lane original image according to a preset distortion correction coefficient to obtain a first corrected image and a second corrected image;

[0074] The first calculation module may also be used to determine feature matching points of the first corrected image and the second corrected image;

[0075] The first calculation module may also be used to generate a first matching image corresponding to the first corrected image and a second matching image corresponding to the second corrected image according to the feature matching points;

[0076] The first calculation module can also be used to generate an RGBD image according to the first matching image and the second matching image.

[0077] In another possible implementation, the device may further include:

[0078] The first calculation module may also be used to perform a lane line recognition operation on the first rectified image to generate a first reference lane line image;

[0079] The first calculation module can also be used to perform a coordinate mapping operation on the first reference lane line image to generate a second lane image.

[0080] In another possible implementation, the device may further include:

[0081] The first calculation module may also be used to determine a similarity transformation parameter between the first matching image and the second matching image;

[0082] The first calculation module can also be used to input the first matching image and the similarity transformation parameters into a preset neural network to obtain a depth map and an RGB image;

[0083] The first calculation module can also be used to superimpose the depth map and the RGB image to generate an RGBD image.

[0084] In another possible implementation, the device may further include:

[0085] The first calculation module may also be used to randomly select a preset number of groups of values ​​from the first feature point matrix N times to generate N first matrices, where N is a positive integer and the preset number of groups is greater than or equal to 3;

[0086] The first calculation module may also be used to randomly select values ​​of the same number of groups as the number of rows of the first matrix from the second feature point matrix N times to generate N second matrices;

[0087] The first calculation module may also be used to divide the N first matrices and the N second matrices into N groups in a one-to-one correspondence, and determine a reference transposed matrix for transforming the second matrix into the first matrix in each group;

[0088] The first calculation module can also be used to calculate the mean square error of N reference transposed matrices;

[0089] The first calculation module can also be used to determine the reference transposed matrix with the smallest mean square error as the target transposed matrix.

[0090] In another possible implementation, the device may further include:

[0091] The first calculation module can also be used to rank the importance of each target device;

[0092] The first calculation module may also be used to select a preset number of target devices according to importance.

[0093] In a fourth aspect, the present application provides a target device, which may include a second image pickup device, a second computing module, and a second communication module;

[0094] A second image picker may be used to obtain an original lane image corresponding to the target device;

[0095] A second calculation module may be used to obtain first lane data according to the lane original image corresponding to the target device, and the first lane data may include a first lane image;

[0096] The second communication module can be used to send the first lane data to vehicles within a preset range. The first lane data can be used for the vehicle to perform an image fusion operation on the first lane image and the second lane image collected by the vehicle to obtain a third lane image.

[0097] In a possible implementation, the first lane data may further include a first feature point matrix;

[0098] The target device may also include:

[0099] The second calculation module may also be used to generate an RGBD image corresponding to the target device according to the original lane image corresponding to the target device;

[0100] The second calculation module can also be used to extract a first feature point matrix based on the RGBD image corresponding to the target device. The first feature point matrix can be used by the vehicle to determine a target transposed matrix based on the first feature point matrix and the second feature point matrix corresponding to the vehicle. The target transposed matrix can be used by the vehicle to perform an image fusion operation on the first lane image and the second lane image based on the target transposed matrix to obtain a third lane image.

[0101] In another possible implementation manner, the lane original image corresponding to the target device includes a third original image and a fourth original image;

[0102] The target device may also include:

[0103] The second calculation module may also be used to perform correction processing on the third lane original image and the fourth lane original image according to a preset distortion correction coefficient to obtain a third corrected image and a fourth corrected image;

[0104] The second calculation module may also be used to determine second feature matching points of the third corrected image and the fourth corrected image;

[0105] The second calculation module may also be used to generate a third matching image corresponding to the third corrected image according to the second feature matching points, and generate a fourth matching image corresponding to the fourth corrected image;

[0106] The second calculation module may also be used to generate an RGBD image corresponding to the target device according to the third matching image and the fourth matching image.

[0107] In another possible implementation manner, the target terminal may further include:

[0108] The second calculation module can also be used to perform a lane line recognition operation on the third corrected image to generate a second reference lane line image;

[0109] The second calculation module can also be used to perform a coordinate mapping operation on the second reference lane line image to generate a first lane image.

[0110] In another possible implementation manner, the target device may further include:

[0111] The second calculation module may also be used to determine similarity transformation parameters between the third matching image and the fourth matching image;

[0112] The second calculation module can also be used to input the third matching image and the similarity transformation parameters into a preset neural network to obtain a depth map and an RGB image;

[0113] The second calculation module can also be used to superimpose the depth map and the RGB image to generate an RGBD image corresponding to the target device.

[0114] In a fifth aspect, the present application provides a processing device, which may include the following parts: a processor, a memory, and a bus;

[0115] The processor and the memory are connected via a bus, wherein the memory is used to store a set of program codes, and the processor is used to call the program codes stored in the memory to execute the method described in the first aspect.

[0116] In a sixth aspect, the present application provides a computer-readable storage medium, including:

[0117] The computer-readable storage medium stores instructions, which, when executed on a computer, implement the method described in the first aspect.

[0118] In a seventh aspect, the present application provides a vehicle, which may include the lane data fusion device as described in the third aspect, the processing device as described in the fifth aspect, and the computer-readable storage medium as described in the sixth aspect.

[0119] It can be seen that the present application can allow the vehicle to obtain lane data from other devices around it, and screen out appropriate lane data to merge with the lane data collected by itself, which helps to fill the lane data in the blind spot of the vehicle's field of view, and allows the vehicle or customer to understand lane information at a longer distance, thereby improving the accuracy and reliability of lane data. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0121] Figure 1 A flowchart of a lane data fusion method provided in an embodiment of the present application;

[0122] Figure 2 A schematic diagram of a scenario for obtaining lane data from a target device provided in an embodiment of the present application;

[0123] Figure 3 A flowchart of a method for generating an RGBD image provided in an embodiment of the present application;

[0124] Figure 4 A flowchart of a method for determining a target transposed matrix provided in an embodiment of the present application;

[0125] Figure 5 A schematic diagram of a scene for feature point matrix registration provided in an embodiment of the present application;

[0126] Figure 6 A schematic diagram of a scenario of lane data transmission provided in an embodiment of the present application;

[0127] Figure 7 A schematic diagram of the composition of a lane data fusion device provided in an embodiment of the present application;

[0128] Figure 8 A schematic diagram of the composition of another lane data fusion device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0129] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0130] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0131] Reference to "embodiments" herein means that a particular feature, result, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0132] In order to better understand the technical solution of the embodiment of the present application, Figure 1 A lane data fusion method provided in an embodiment of the present application is described in detail.

[0133] See also Figure 1 , is a flow chart of a lane data fusion method provided in an embodiment of the present application, the method may include the following steps:

[0134] S101, the vehicle receives first lane data sent by a target device within a preset range.

[0135] It should be noted that the target device can use wireless communication technology to transmit lane data with the vehicle. The road test equipment can use the IEEE 802.11p protocol and / or the fifth generation mobile communication technology (5th Generation Mobile Communication Technology, 5G) to transmit the first request and / or lane data with the vehicle, and the vehicle can also use vehicle-to-vehicle communication technology (V2V), vehicle-to-infrastructure communication technology (V2I) and vehicle-to-network communication technology (V2N) to transmit the first request and / or lane data with other target devices in the form of vehicles.

[0136] In a possible implementation manner, before the vehicle receives the first lane data sent by the target device, the vehicle may further include: sending a first request to the target device within a preset range.

[0137] It should be noted that the first request may be used to obtain lane data from the target device, and the above-mentioned vehicle may include a first image pickup device. Specifically, the lane data may include the vehicle (in this article, it refers to the vehicle that issues the first request, which will not be repeated below) and / or a lane image generated by the target device based on the original lane image.

[0138] In an embodiment of the present application, the target device may be a device having an image pickup device and data transmission functions, and its specific form may be other vehicles, road side equipment (RSU) (such as surveillance cameras located around the road, etc.), pedestrians' mobile terminals, etc.

[0139] Exemplarily, assuming that the preset range is a circular range with the vehicle as the center and a radius of 10 m, the vehicle will send a first request to a target device within the range, requesting the target device to send the lane data collected by it to the vehicle.

[0140] It should be noted that the number of the target devices may be 1 or greater than 1. The target devices within the preset range may receive the first request when they have a communication function or a lane data sharing function.

[0141] For example, Figure 2A schematic diagram of a scenario for obtaining lane data from a target device provided in an embodiment of the present application. Assuming that vehicle 1 now needs to generate a lane image of road 1 and the preset range is 10 meters, vehicle 1 will send a first request to target devices (such as vehicle 2, vehicle 3, and camera 1) located within a circular range with vehicle 1 as the center and a radius of 10 meters. Among them, the distance between vehicle 4 and vehicle 1 exceeds 10 meters, so the first request is sent to vehicle 4. In response to the first request sent by vehicle 1, the target device will transmit the lane data collected by itself to vehicle 1.

[0142] S102: The vehicle obtains an original lane image through the first image pickup device.

[0143] Specifically, the first image pickup device of the vehicle may include a first camera and a second camera, and the lane original image may include a first lane original image captured by the first camera and a second lane original image captured by the second camera. In the present application, the first camera and the second camera may be distinguished by a left camera and a right camera. The first camera may be a main camera, and the second camera may be an auxiliary camera. Alternatively, the first camera may be an auxiliary camera, and the second camera may be a main camera, which is not limited.

[0144] Optionally, the vehicle controls the first camera and the second camera to simultaneously capture the original image of the first lane and the original image of the second lane.

[0145] S103: The vehicle acquires a second lane image according to the original lane image.

[0146] More specifically, the second lane image can be generated based on the original image of the first lane captured by the first camera, and the first camera can be understood as the main camera of the vehicle.

[0147] In a possible implementation manner, the generation of the second lane image may refer to the following steps:

[0148] Correcting the original image of the first lane according to a preset distortion correction coefficient to obtain a first corrected image;

[0149] Performing a lane line recognition operation on the first corrected image to generate a first reference lane line image;

[0150] A coordinate mapping operation is performed on the first reference lane line image to generate a second lane image.

[0151] It should be noted that images collected by different cameras will have different degrees of distortion, so the vehicle needs to query the model information of the first camera and the second camera from its own database, and obtain the corresponding distortion correction coefficient from the database based on the model information. The distortion correction coefficient can be pre-saved by the manufacturer before the vehicle leaves the factory; the camera model and its distortion correction coefficient related information in the database can also be updated using Over-the-Air (OTA) technology during the use of the vehicle.

[0152] Possibly, before the first lane original image is corrected according to the distortion correction coefficient to obtain the first corrected image, the first lane original image may be preprocessed, wherein the image preprocessing may include image denoising and image enhancement.

[0153] Furthermore, distortion correction can include radial distortion correction and tangential distortion correction. The mathematical model of radial distortion correction can refer to the following formula:

[0154]

[0155] Among them, u is the horizontal coordinate of the original image of the first lane in the normalized plane, v is the vertical coordinate of the original image of the first lane in the normalized plane, u′ represents the horizontal coordinate of the original image of the first lane after radial distortion correction in the normalized plane, v′ represents the vertical coordinate of the original image of the first lane after radial distortion correction in the normalized plane, k 1 , k 2 , k 3 are radial distortion correction coefficients, r 2 =u 2 +v 2 .

[0156] The mathematical model of tangential distortion correction can refer to the following formula:

[0157]

[0158] Among them, u is the horizontal coordinate of the original image of the first lane in the normalized plane, v is the vertical coordinate of the original image of the first lane in the normalized plane, u′ represents the horizontal coordinate of the original image of the first lane after tangential distortion correction in the normalized plane, v′ represents the vertical coordinate of the original image of the first lane after radial shear correction in the normalized plane, p 1 and p 2 are the tangential distortion correction coefficients, r 2 =u 2 +v 2 .

[0159] Furthermore, the vehicle can input the first corrected image into the lane line detection model, and the model will output a first reference lane line image, which can be a 2D image. Then, the first reference lane line image is coordinate mapped to obtain a second lane line image, which can be understood as mapping the coordinates of the 2D image to a 3D coordinate system, thereby obtaining a 3D lane line image.

[0160] It can be seen that the embodiment of the present application can perform distortion correction processing on the original lane data collected by the vehicle, so as to obtain second image data that is more in line with the actual situation of the lane, which helps to improve the accuracy and reliability of the lane image finally presented to the user or vehicle.

[0161] S104: Perform an image fusion operation on the first lane image and the second lane image to obtain a third lane image.

[0162] In a possible implementation, the first lane data may further include a first feature point matrix, which is generated by the target device according to an RGBD image corresponding to the lane original image acquired by the target device;

[0163] The method may further comprise the following steps:

[0164] Generate an RGBD image based on the original lane image;

[0165] Extract the second feature point matrix of the RGBD image;

[0166] When a root mean square error between the first lane image and the second lane image is less than a preset value, determining a target transposed matrix according to the first feature point matrix and the second feature point matrix;

[0167] Performing an image fusion operation on the first lane image and the second lane image to obtain a third lane image may include:

[0168] An image fusion operation is performed on the first lane image and the second lane image according to the target transposed matrix to obtain a third lane image.

[0169] The target transposed matrix may include similarity transformation parameters of the first feature point matrix transformed into the second feature point matrix by a geometric transformation method. Possibly, the similarity transformation parameters may include at least one of a translation parameter, a rotation parameter, and a scaling parameter.

[0170] It should be noted that the first lane data may include a first lane image and a first feature point matrix.

[0171] More specifically, the feature point matrix may include the number of feature points, the type of feature points, and the position of feature points of the RGBD image corresponding to the feature point matrix. Specifically, the first feature point matrix may include the number of feature points, the type of feature points, and the position of feature points of the RGBD image corresponding to the original lane image collected by the target device, and the second feature point matrix may include the number of feature points, the type of feature points, and the position of feature points of the RGBD image corresponding to the original lane image collected by the vehicle. The specific form of the feature point matrix can be Among them, x represents the number of feature points, y represents the type of feature points, and z represents the position of feature points.

[0172] Possibly, the feature points represented by the feature point matrix may be specific objects in the original lane image, and their types may include trees, lane lines, vehicles, pedestrians, electric poles, zebra crossings, green belts or road barriers, etc. Different types may correspond to different digital numbers. It should be noted that the above examples of the types of objects represented by feature points are only for the purpose of illustrating the embodiments of the present application in more detail, and should not be construed as limiting the present application. The present application does not limit the method for generating RGBD images, and can refer to the subsequent Figure 3 A flowchart of a method for generating an RGBD image is described.

[0173] It can be seen that the embodiment of the present application can obtain lane data from a target device within a preset range, which helps to expand the source of lane data, and then supplement the lane data of the blind spot of the vehicle's field of view, and can provide users and / or vehicles with a lane image that is closer to the actual situation of the lane. The present application can also determine a target transposed matrix based on the lane data sent by the target device and the lane data collected by the vehicle, and then perform a fusion operation on the lane data obtained by the target device and the lane data collected by the vehicle according to the target transposed matrix, which helps to generate a lane image that is more in line with the actual situation, improves the accuracy and reliability of the lane image, and further improves the user experience.

[0174] In one possible implementation, the third lane is presented on a display device of the vehicle.

[0175] like Figure 3 As shown, the method for generating an RGBD image may include the following steps:

[0176] S301, performing correction processing on the first lane original image and the second lane original image according to a preset distortion correction coefficient to obtain a first corrected image and a second corrected image.

[0177] The specific execution details of step S301 to step S302 may refer to the relevant embodiment regarding the generation of the first corrected image in step S104, which will not be described in detail here.

[0178] S302: Determine feature matching points of the first corrected image and the second corrected image.

[0179] For example, if object 1, object 2 and object 3 exist in the first corrected image, and object 1, object 3 and object 4 exist in the second corrected image, then the feature matching points of the first corrected image and the second corrected image are the corresponding pixel points of object 1 and object 3 in the two images.

[0180] S303: Generate a first matching image corresponding to the first corrected image according to the feature matching points, and generate a second matching image corresponding to the second corrected image.

[0181] Possibly, the feature points in the first corrected image (which can be understood as the pixels belonging to the object contour and its inner contour) are retained and other pixels are deleted, so as to generate the first matching image; similarly, the feature points in the second corrected image are retained and other pixels are deleted, so as to generate the second matching image. Exemplarily, if the feature points representing the same object in the first corrected image and the second corrected image are the pixels corresponding to object 1 and object 3, then the first matching image can be obtained by deleting the pixels in the first corrected image except the pixels corresponding to object 1 and object 3; and the second matching image can be obtained by deleting the pixels in the second corrected image except the pixels corresponding to object 1 and object 3.

[0182] S304: Generate an RGBD image according to the first matching image and the second matching image.

[0183] Possibly, before the first lane original image and the second lane original image are corrected according to the distortion correction coefficient to obtain the first corrected image and the second corrected image, the first lane original image and the second lane original image may be subjected to image preprocessing, wherein the image preprocessing may include image denoising and image enhancement.

[0184] In a possible implementation, generating an RGBD image according to the first matching image and the second matching image may include the following steps:

[0185] determining a similarity transformation parameter between the first matching image and the second matching image;

[0186] Inputting the first matching image and the similarity transformation parameters into a preset neural network to obtain a depth map and an RGB image;

[0187] The depth map is superimposed with the RGB image to generate an RGBD image.

[0188] The matching information represents similarity transformation parameters between the first matching image and the second matching image, and the first matching image can be understood as a matching image corresponding to the original lane image captured by the main camera of the vehicle.

[0189] Furthermore, mapping the depth information of the depth map to the brightness or color of the RGB image helps to better restore the structural information (such as height, width, and length) of each object on the lane.

[0190] It can be seen that the embodiment of the present application can correct the collected original lane image according to the model information of different cameras, which helps to improve the accuracy of the lane image. The embodiment of the present application can also extract the feature points representing the same object in the two corrected images by matching the corrected images, and then generate an RGBD image. The simplified image helps to reduce the time required to determine the target transposed matrix of the lane data sent to the target device, and also reduces the influence of other factors (such as irrelevant distortion and image noise, etc.) on the generation of the target transposed matrix, thereby indirectly improving the reliability of the fused lane image to be closer to the lane.

[0191] More, Figure 4 A flowchart of a method for determining a target transposed matrix provided in an embodiment of the present application. The method may include the following steps:

[0192] S401, randomly selecting a preset number of groups of values ​​from the first feature point matrix N times to generate N first matrices, where N is a positive integer.

[0193] The preset number of groups is greater than or equal to 3.

[0194] Exemplarily, let the first feature point matrix be S S Indicated by G. S express.

[0195] S402, randomly selecting from the second feature point matrix N times the same number of groups of values ​​as the number of rows in the first matrix, to generate N second matrices.

[0196] Exemplarily, let the second feature point matrix be S R The second matrix is ​​represented by G R express.

[0197] S403, dividing the N first matrices and the N second matrices into N groups in a one-to-one correspondence, and determining a reference transposed matrix for transforming the second matrix into the first matrix in each group.

[0198] For example, point A(x, y, z) in the second matrix is ​​transformed into the coordinate system of the first matrix through the transposed matrix, and the coordinates of point A' can be obtained as (x', y', z'). The calculation process of the coordinate system transformation can refer to the following formula:

[0199]

[0200] S404, calculating the mean square error of N reference transposed matrices.

[0201] It can be understood that the reference transposed matrix may include rotation parameters, translation parameters and scaling parameters, and correspondingly, there may be a rotation matrix and a translation vector.

[0202] It should be noted that the minimum mean square error of each reference transposed matrix needs to be found here. 2 The minimum mean square error of (R, t, c) can refer to the following steps, where:

[0203]

[0204] Among them, R represents the rotation matrix corresponding to the rotation parameter in the reference transposed matrix, t represents the translation vector corresponding to the translation parameter in the reference transposed matrix, c represents the scaling parameter in the reference transposed matrix, and x i Represents the first matrix G S Column vector, y i Denotes the second matrix G R The column vector of n represents the first matrix G S and the second matrix G R The number of columns.

[0205] Existence Lemma: Let A and B be m×n matrices. There is an m×m rotation matrix R between matrices A and B. AB T (UU T =VV T =I,D=diag(d i ),d 1 ≥d 2 ≥…≥d m ≥0) is decomposed into UDV T (Among them, UU T =VV T =I,D=diag(d i ),d 1 ≥d 2 ≥…≥d m ≥0, U is an m-order orthogonal matrix, V is an n-order orthogonal matrix). Then ‖A-RB‖ 2 The minimum value relative to R is:

[0206] min R ‖A-RB‖ 2 =‖A‖ 2 +‖B‖ 2 -2tr(DS),

[0207]

[0208] Existence theorem: Let X = (x 1 ,x 2 ,…,x n )=G S and Y=(y 1 ,y 2 ,…,y n )=G R , you can have

[0209]

[0210] Then the minimum value of the mean square error of these two matrices X and Y relative to the similarity transformation parameters is 2 as follows:

[0211]

[0212] in,

[0213]

[0214]

[0215]

[0216]

[0217]

[0218] Where ∑xy is the covariance matrix of X and Y, u x and u y is the mean vector of X and Y, and is the variance around the mean vector of X and Y.

[0219] The reason is as follows: Let X = (x 1 ,x 2 ,…,x n )=G S and Y=(y 1 ,y 2 ,…,y n )=G R , and then use m×n matrices to represent X and Y, we get:

[0220]

[0221] h=(1,1,…,1) T ,

[0222] Here we introduce an n×n normalized matrix (K 2 =K T =K), we get:

[0223]

[0224]

[0225]

[0226] at the same time:

[0227]

[0228]

[0229] so:

[0230]

[0231]

[0232] So you can get

[0233] tr(K(Y T -cx T R T )t′h T )=0,

[0234] ||t′h T || 2 =n||t′|| 2 ,

[0235] have to

[0236]

[0237] In this equation, t′ needs to be equal to 0 to minimize e 2 (R,t,c)

[0238]

[0239] Next, when UDV T ∑xy=(1 / nYKX T )’s singular value decomposition, cnUDV T is YK(cXK) T =cYKK T X T =cYKX T Therefore, the minimum value ε about R 2 (c)=(1 / n)‖YK-cRXK‖ 2 It is given by the following lemma:

[0240]

[0241]

[0242] From the above lemma, if rank(∑xy)≥m-1, then R=USV T ,

[0243] If rank(∑xy)=m-1, then

[0244]

[0245] Finally, the minimum ε can be obtained 2 yes

[0246]

[0247] S405: Determine the reference transposed matrix with the smallest mean square error as the target transposed matrix.

[0248] Specifically, based on the minimum mean square error of each reference transposed matrix obtained in step S404, each reference transposed matrix is ​​screened, and the reference transposed matrix with the minimum mean square error is determined as the target transposed matrix.

[0249] in, Figure 5 A schematic diagram of a scene for feature point matrix registration provided in an embodiment of the present application.

[0250] The first matrix and the second matrix containing three sets of values ​​can be regarded as triangle ABC and triangle DEF respectively (assuming that triangle ABC and triangle DEF are triangles of the same size). The coordinate system of triangle DEF corresponding to the second matrix is ​​taken as the final coordinate system. The process of transforming triangle ABC to triangle DEF is called the process of feature point matrix registration. Figure 5 As shown, triangle ABC can be rotated first to obtain rotation parameters, and then translated to obtain translation parameters. Here, since triangle ABC and triangle DEF are triangles of the same size, the scaling parameter can be regarded as 1 (ie, neither zoom-in nor zoom-out operation is performed).

[0251] It should be noted that if the sizes of the triangles corresponding to the first matrix and the triangles corresponding to the second matrix are inconsistent, the final registration result is not necessarily two triangles of the same size, but may be two similar triangles.

[0252] in, Figure 6 A schematic diagram of a lane data transmission scenario provided in an embodiment of the present application.

[0253] set up Figure 6 The specific type of the target device in is a vehicle.

[0254] The target device will obtain the original image of the lane through the second image picker. Specifically, the target device will simultaneously obtain the original image 611 of the third lane and the original image 612 of the fourth lane through the left camera (set as the main camera) and the right camera, and perform distortion correction processing on the original image 611 of the third lane and the original image 612 of the fourth lane to obtain the third corrected image 613 and the fourth corrected image 614. Subsequently, the third corrected image 613 and the fourth distortion corrected image 614 are image matched to obtain the third matching image 615 and the fourth matching image 6141, and the similarity transformation parameters of the third matching image 615 and the third matching image 615 and the first matching image 6141 are input into the neural network to obtain the depth map 616 and the RGB image 617. The depth map 616 and the RGB image 617 are superimposed to generate the RGBD image 618, and the feature points of the RGBD image 618 are extracted to generate the first feature point matrix 619. Furthermore, the target device will also perform lane line recognition on the third corrected image 613 to obtain a 2D lane image 6110. A coordinate mapping operation is then performed on the 2D lane image 6110 to obtain a 3D lane image 6111 (equivalent to the first lane image mentioned above). Finally, after receiving the first request sent by the vehicle (i.e., the vehicle that sends the first request to the target device), the target device will send the first feature point matrix 619 and the 3D lane image 6111 to the vehicle (i.e., the vehicle that sends the first request to the target device).

[0255] The vehicle (i.e., the vehicle that sends the first request to the target device) will simultaneously obtain the first lane original image 621 and the second lane original image 622 through the left camera (set as the main camera) and the right camera, and perform distortion correction processing on the first lane original image 621 and the second lane original image 622 to obtain the first corrected image 623 and the second corrected image 624. Subsequently, the first corrected image 623 and the second corrected image 624 are image matched to obtain the first matching image 625 and the second matching image 6241, and the first matching image 625 and the similarity transformation parameters of the first matching image 625 and the second matching image 6241 are input into the neural network to obtain the depth map 626 and the RGB image 627. The depth map 626 and the RGB image 627 are superimposed to generate an RGBD image 628, and the feature points of the RGBD image 628 are extracted to generate the second feature point matrix 629. Furthermore, the vehicle (i.e., the vehicle that sends the first request to the target device) also performs lane line recognition on the first corrected image 623 to obtain a 2D lane image 6210. A coordinate mapping operation is then performed on the 2D lane image 6210 to obtain a 3D lane image 6211 (equivalent to the second lane image mentioned above).

[0256] Further, after receiving the first feature point matrix 619 and the 3D lane image 6111, the vehicle (i.e., the vehicle that sends the first request to the target device) performs a registration process on the first feature point matrix 619 and the second feature point matrix 629 in the registration module 6212, and obtains similarity transformation parameters of the geometric transformation of the first feature point matrix 619 to the second feature point matrix 629. Based on the similarity transformation parameters, a target transposed matrix 6213 can be obtained. Finally, the 3D lane image 6111 and the 3D lane image 6211 are fused based on the target transposed matrix 6213, and finally a third lane image 6214 is obtained.

[0257] based on Figure 1-Figure 6 Related embodiments, in another possible implementation manner, the method of the present application may further include the following steps:

[0258] When the number of target devices is greater than or equal to 2, ranking the respective target devices in order of importance;

[0259] A preset number of target devices are selected according to importance.

[0260] Possibly, the present application can rank the importance of each target device according to the size of the mean square error of the target transposed matrix corresponding to the target device, and / or the distance between the target device and the vehicle. Specifically, the smaller the mean square error of the target transposed matrix corresponding to the target device, the higher the importance of the target device; the smaller the distance between the target device and the vehicle, the higher the importance of the target device.

[0261] In another possible implementation, the target device may process the original lane image it collects (i.e., generate a first lane image and a first feature point matrix) by referring to the processing flow of the vehicle-related embodiments in the present application, which will not be elaborated here.

[0262] The device involved in the embodiment of the present application is described below with reference to the accompanying drawings.

[0263] See also Figure 7 , is a lane data fusion device provided in an embodiment of the present application, the device may include: a first communication module 710, a first image pickup device 720 and a first calculation module 730;

[0264] The first communication module 710 may be used to receive first lane data sent by a target device within a preset range, where the first lane data may include a first lane image;

[0265] The first image pickup device 720 may be used to obtain an original lane image;

[0266] The first calculation module 730 may be used to obtain a second lane image according to the original lane image;

[0267] The first calculation module 730 may also be used to perform an image fusion operation on the first lane image and the second lane image to obtain a third lane image.

[0268] In a possible implementation, the first lane data may further include a first feature point matrix, which is generated by the target device according to an RGBD image corresponding to the lane original image acquired by the target device;

[0269] The device may also include:

[0270] The first calculation module 730 may also be used to generate an RGBD image based on the original lane image;

[0271] The first calculation module 730 may also be used to extract a second feature point matrix of the RGBD image;

[0272] The first calculation module 730 may also be used to determine a target transposed matrix according to the first feature point matrix and the second feature point matrix when a root mean square error between the first lane image and the second lane image is less than a preset value;

[0273] The first calculation module 730 may also be used to perform an image fusion operation on the first lane image and the second lane image according to the target transposed matrix to obtain a third lane image.

[0274] In another possible implementation, the first image pickup device may include a first camera and a second camera, and the lane original image may include a first lane original image captured by the first camera and a second lane original image captured by the second camera;

[0275] The device may also include:

[0276] The first calculation module 730 may be used to perform correction processing on the first lane original image and the second lane original image according to a preset distortion correction coefficient to obtain a first corrected image and a second corrected image;

[0277] The first calculation module 730 may also be used to determine feature matching points of the first corrected image and the second corrected image;

[0278] The first calculation module 730 may also be used to generate a first matching image corresponding to the first corrected image and a second matching image corresponding to the second corrected image according to the feature matching points;

[0279] The first calculation module 730 may also be configured to generate an RGBD image according to the first matching image and the second matching image.

[0280] In another possible implementation, the device may further include:

[0281] The first calculation module 730 may also be used to perform a lane line recognition operation on the first rectified image to generate a first reference lane line image;

[0282] The first calculation module 730 may also be used to perform a coordinate mapping operation on the first reference lane line image to generate a second lane image.

[0283] In another possible implementation, the device may further include:

[0284] The first calculation module 730 may also be used to determine similarity transformation parameters between the first matching image and the second matching image;

[0285] The first calculation module 730 may also be used to input the first matching image and the similarity transformation parameters into a preset neural network to obtain a depth map and an RGB image;

[0286] The first calculation module 730 may also be used to superimpose the depth map and the RGB image to generate an RGBD image.

[0287] In another possible implementation, the device may further include:

[0288] The first calculation module 730 may also be used to randomly select a preset number of groups of values ​​from the first feature point matrix N times to generate N first matrices, where N is a positive integer and the preset number of groups is greater than or equal to 3;

[0289] The first calculation module 730 may also be used to randomly select values ​​of the same number of groups as the number of rows of the first matrix from the second feature point matrix N times to generate N second matrices;

[0290] The first calculation module 730 may also be used to divide the N first matrices and the N second matrices into N groups in a one-to-one correspondence, and determine a reference transposed matrix for transforming the second matrix into the first matrix in each group;

[0291] The first calculation module 730 may also be used to calculate the mean square error of N reference transposed matrices;

[0292] The first calculation module 730 may also be used to determine a reference transposed matrix with the smallest mean square error as a target transposed matrix.

[0293] In another possible implementation, the device may further include:

[0294] The first calculation module 730 may also be used to rank the importance of each target device;

[0295] The first calculation module 730 may also be used to select a preset number of target devices according to importance.

[0296] See also Figure 8, is a schematic diagram of another lane data fusion device provided in an embodiment of the present application, and the device may include:

[0297] Processor 810, memory 820 and I / O interface 830. The processor 810, memory 820 and I / O interface 830 can be connected to each other in a communicative manner. The memory 820 is used to store instructions, and the processor 810 is used to execute the instructions stored in the memory 820 to implement the above Figure 1 , Figure 3 as well as Figure 4 The corresponding method steps.

[0298] The processor 810 is used to execute the instructions stored in the memory 820 to control the I / O interface 830 to receive and send signals and complete the steps in the above method. The memory 820 can be integrated in the processor 810 or can be set separately from the processor 810.

[0299] The memory 820 may also include a storage system 821, a cache 822, and a RAM 823. The cache 822 is a primary memory between the RAM 823 and the CPU, which is composed of a static memory chip (SRAM). It has a relatively small capacity but a much higher speed than the main memory, close to the speed of the CPU; the RAM 823 is an internal memory that directly exchanges data with the CPU, which can be read and written at any time (except when refreshed), and has a very high speed. It is usually used as a temporary data storage medium for the operating system or other running programs. The three are combined to realize the functions of the memory 820.

[0300] As an implementation, the function of the I / O interface 830 may be implemented by a transceiver circuit or a dedicated transceiver chip. The processor 810 may be implemented by a dedicated processing chip, a processing circuit, a processor or a general-purpose chip.

[0301] As another implementation method, it is possible to use a general-purpose computer to implement the device provided in the embodiment of the present application. That is, the program code that implements the functions of the processor 810 and the I / O interface 830 is stored in the memory 820, and the general-purpose processor implements the functions of the processor 810 and the I / O interface 830 by executing the code in the memory 820.

[0302] For the concepts, explanations, detailed descriptions and other steps involved in the device and related to the technical solutions provided in the embodiments of the present application, please refer to the description of the method steps performed by the device in the aforementioned method or other embodiments, which will not be repeated here.

[0303] As another implementation of this embodiment, a computer-readable storage medium is provided, on which instructions are stored. When the instructions are executed, the method in the above method embodiment is executed.

[0304] As another implementation of this embodiment, a computer program product including instructions is provided, and when the instructions are executed, the method in the above method embodiment is performed.

[0305] Those skilled in the art will appreciate that for ease of description, Figure 8 Only one memory and processor are shown. In an actual terminal or server, there may be multiple processors and memories. The memory may also be referred to as a storage medium or a storage device, etc., which is not limited in the embodiments of the present application.

[0306] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.

[0307] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

[0308] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) is integrated in the processor.

[0309] It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0310] In addition to the data bus, the bus may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are marked as buses in the figure.

[0311] It should also be understood that the first, second, third, fourth and various numerical numbers involved in this document are only distinctions made for the convenience of description and are not intended to limit the scope of the present application.

[0312] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0313] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0314] In various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0315] Those skilled in the art will appreciate that the various illustrative logical blocks (ILBs) and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0316] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0317] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0318] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0319] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0320] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement part or all of the steps of any method for managing wireless connections based on a vehicle-mounted device as recorded in the above method embodiments.

[0321] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute part or all of the steps of any method for managing wireless connections based on a vehicle-mounted device as recorded in the above method embodiments.

[0322] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A lane data fusion method, characterized in that: The method is applied to a vehicle, the vehicle comprising a first image pickup device, and the method comprises the following steps: The vehicle receives first lane data sent by a target device within a preset range, where the first lane data includes a first lane image; The vehicle acquires an original lane image through the first image pickup device; The vehicle acquires a second lane image according to the lane original image; An image fusion operation is performed on the first lane image and the second lane image to obtain a third lane image.

2. The method according to claim 1, characterized in that The first lane data further includes a first feature point matrix, which is generated by the target device according to an RGBD image corresponding to the lane original image acquired by the target device; The method further comprises the following steps: Generate an RGBD image based on the lane original image; Extracting a second feature point matrix of the RGBD image; When a root mean square error between the first lane image and the second lane image is less than a preset value, determining a target transposed matrix according to the first feature point matrix and the second feature point matrix; The performing an image fusion operation on the first lane image and the second lane image to obtain a third lane image includes: An image fusion operation is performed on the first lane image and the second lane image according to the target transposed matrix to obtain the third lane image.

3. The method according to claim 2, characterized in that The first image pickup device includes a first camera and a second camera, and the lane original image includes a first lane original image captured by the first camera and a second lane original image captured by the second camera; The step of generating an RGBD image according to the lane original image comprises the following steps: Correcting the first lane original image and the second lane original image according to a preset distortion correction coefficient to obtain a first corrected image and a second corrected image; Determining feature matching points of the first rectified image and the second rectified image; generating a first matching image corresponding to the first corrected image according to the feature matching points, and generating a second matching image corresponding to the second corrected image; The RGBD image is generated according to the first matching image and the second matching image.

4. The method according to claim 3, characterized in that: The method further comprises the following steps: Performing a lane line recognition operation on the first corrected image to generate a first reference lane line image; A coordinate mapping operation is performed on the first reference lane line image to generate the second lane image.

5. The method according to claim 3 or 4, characterized in that: The step of generating an RGBD image according to the first matching image and the second matching image comprises the following steps: Determine similarity transformation parameters between the first matching image and the second matching image, input the first matching image and the similarity transformation parameters into a preset neural network, and obtain a depth map and an RGB image; The depth map is superimposed on the RGB image to generate the RGBD image.

6. The method according to claim 5, characterized in that Determining a target transposed matrix according to the first feature point matrix and the second feature point matrix comprises the following steps: Randomly selecting a preset number of groups of values ​​from the first feature point matrix N times to generate N first matrices, where N is a positive integer and the preset number of groups is greater than or equal to 3; Randomly select from the second feature point matrix N times the same number of groups of values ​​as the number of rows in the first matrix, to generate N second matrices; Divide the N first matrices and the N second matrices into N groups in a one-to-one correspondence, and determine a reference transposed matrix for transforming the second matrix into the first matrix in each group; Calculate the mean square error of N reference transposed matrices; A reference transposed matrix with the smallest mean square error is determined as the target transposed matrix.

7. The method according to claim 6, characterized in that The method further comprises the following steps: When the number of the target devices is greater than or equal to 2, ranking the respective target devices in order of importance; A preset number of target devices are selected according to the importance.

8. A method for lane data fusion, characterized in that: The method is applied to a target device, the target device includes a second image pickup device, and the method includes the following steps: The target device acquires an original lane image corresponding to the target device through the second image picker; The target device acquires first lane data according to the lane original image corresponding to the target device, where the first lane data includes a first lane image; The target device sends the first lane data to a vehicle within a preset range, and the first lane data is used by the vehicle to perform an image fusion operation on the first lane image and a second lane image captured by the vehicle to obtain a third lane image.

9. The method according to claim 8, characterized in that The first lane data also includes a first feature point matrix; The method further comprises the following steps: Generate an RGBD image corresponding to the target device according to the original lane image corresponding to the target device; The first feature point matrix is ​​extracted according to the RGBD image corresponding to the target device, the first feature point matrix is ​​used by the vehicle to determine a target transposed matrix according to the first feature point matrix and a second feature point matrix corresponding to the vehicle, and the target transposed matrix is ​​used by the vehicle to perform an image fusion operation on the first lane image and the second lane image according to the target transposed matrix to obtain the third lane image.

10. The method according to claim 9, characterized in that The lane original image corresponding to the target device includes a third original image and a fourth original image; The step of generating an RGBD image corresponding to the target device according to the original lane image corresponding to the target device comprises the following steps: Correcting the third lane original image and the fourth lane original image according to a preset distortion correction coefficient to obtain a third corrected image and a fourth corrected image; Determining feature matching points of the third rectified image and the fourth rectified image; generating a third matching image corresponding to the third corrected image according to the feature matching points, and generating a fourth matching image corresponding to the fourth corrected image; An RGBD image corresponding to the target device is generated according to the third matching image and the fourth matching image.

11. The method according to claim 10, characterized in that The method further comprises the following steps: Performing a lane line recognition operation on the third corrected image to generate a second reference lane line image; A coordinate mapping operation is performed on the second reference lane line image to generate the first lane image.

12. The method according to claim 10 or 11, characterized in that: The step of generating an RGBD image corresponding to the target device according to the third matching image and the fourth matching image comprises the following steps: determining a similarity transformation parameter between the third matching image and the fourth matching image; Inputting the third matching image and the similarity transformation parameters into a preset neural network to obtain a depth map and an RGB image; The depth map is superimposed on the RGB image to generate an RGBD image corresponding to the target device.

13. A lane data fusion device, characterized in that: The device comprises: a first communication module, a first image pickup device and a first calculation module; The first communication module is used to receive first lane data sent by a target device within a preset range, wherein the first lane data includes a first lane image; The first image picker is used to obtain an original lane image; The first calculation module is used to obtain a second lane image according to the lane original image; The first computing module is further configured to perform an image fusion operation on the first lane image and the second lane image to obtain a third lane image.

14. A target device, characterized in that: The target device includes: a second image pickup device, a second calculation module and a second communication module; The second image picker is used to obtain an original lane image corresponding to the target device; The second calculation module is used to obtain first lane data according to the lane original image corresponding to the target device, and the first lane data includes a first lane image; The second communication module is used to send first lane data to vehicles within a preset range, and the first lane data is used by the vehicle to perform an image fusion operation on the first lane image and the second lane image collected by the vehicle to obtain a third lane image.

15. A processing device, characterized in that: include: A processor, a memory and an I / O interface, wherein the processor, the memory and the I / O interface are communicatively connected, wherein the memory is used to store a set of program codes, and the processor is used to call the program codes stored in the memory to execute the method according to any one of claims 1 to 7, or to execute the method according to any one of claims 8 to 12.

16. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores instructions, which, when executed on a computer, implement the method according to any one of claims 1 to 7, or implement the method according to any one of claims 8 to 12.

17. A vehicle, characterized in that: The vehicle comprises the lane data fusion device according to claim 13, the processing device according to claim 15, and the computer-readable storage medium according to claim 16.

Images