Lane line fusion method and device, automobile, electronic equipment and computer readable storage medium

Abstract

The application relates to the technical field of automatic driving, in particular to a lane line fusion method and device, an automobile, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring a plurality of to-be-processed images, wherein the plurality of to-be-processed images are collected by a plurality of image collection devices of a vehicle; inputting each to-be-processed image into a preset lane line detection model to obtain a plurality of initial lane lines corresponding to each to-be-processed image; fusing each initial lane line to obtain a fused lane line; and matching the fused lane line with a tracked lane line to obtain a lane line result. The beneficial effect is that the initial lane line is obtained by processing the plurality of to-be-processed images, and the initial lane line is fused and tracked, thereby meeting the demand of automatic driving for the lane line.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a lane line fusion method and device, a car, an electronic device and a computer readable storage medium. BACKGROUND

[0002] The visual perception of automatic driving of a car mainly uses a camera configured on the car and a computer vision-based method to identify, detect and track objects such as lane lines, drivable areas, static targets and dynamic targets. Automatic driving has high requirements for the accuracy of the detection and tracking results of lane lines. Currently, common lane line processing based on a single camera is used in front view, that is, a camera is installed at the front of the car and lane lines within the visual angle range of the camera are processed. The lane line processing based on a single camera is difficult to meet the use requirements of automatic driving of the car. SUMMARY

[0003] In view of the above-mentioned shortcomings of the prior art, the present application provides a lane line fusion method, device, car, electronic device and computer readable storage medium based on lane line processing of multiple cameras to solve the technical problem that the lane line information based on a single camera is single and difficult to meet the use requirements.

[0004] The present application provides a lane line fusion method, which comprises:

[0005] obtaining a plurality of to-be-processed images, wherein the plurality of to-be-processed images are collected by a plurality of image collection devices of a vehicle;

[0006] inputting each of the to-be-processed images into a preset lane line detection model to obtain a plurality of initial lane lines corresponding to each of the to-be-processed images;

[0007] fusing each of the initial lane lines to obtain a fused lane line;

[0008] matching the fused lane line with a tracked lane line to obtain a lane line result.

[0009] In an embodiment of the present application, inputting each of the to-be-processed images into a preset lane line detection model to obtain a plurality of initial lane lines corresponding to each of the to-be-processed images comprises:

[0010] detecting each of the to-be-processed images to obtain a plurality of initial lane line points in the to-be-processed image;

[0011] sampling and projecting a plurality of the initial lane line points to obtain a plurality of projected lane line points, and performing curve fitting on a plurality of the projected lane line points to obtain a plurality of the initial lane lines.

[0012] In an embodiment of the present application, the fusion of each initial lane line to obtain a fused lane line includes at least one of the following:

[0013] If the difference between the lateral offsets of at least two initial lane lines is less than a preset offset threshold, and the Euclidean distance is less than a preset distance threshold, the at least two initial lane lines are fused to obtain a fused lane line.

[0014] If the line type attribute parameters of at least two initial lane lines are consistent, the at least two initial lane lines are fused to obtain a fused lane line, wherein the line type attribute parameters include lane line color and lane line type.

[0015] In an embodiment of the present application, after each of the to-be-processed images is input to a preset lane line detection model to obtain a plurality of initial lane lines corresponding to each of the to-be-processed images, the method further includes:

[0016] The initial lane lines are sorted according to the lateral offsets to obtain an initial lane line sequence.

[0017] If the difference between the lateral offsets of a plurality of consecutive initial lane lines in the initial lane line sequence is less than a preset offset threshold, the Euclidean distance between the plurality of consecutive initial lane lines is determined.

[0018] If the Euclidean distance between the plurality of consecutive initial lane lines is less than a preset distance threshold, the plurality of consecutive initial lane lines are determined as a fused lane line subset to obtain a lane line set including a plurality of fused lane line subsets.

[0019] In an embodiment of the present application, after the lane line set including a plurality of fused lane line subsets is obtained, the method further includes:

[0020] According to the image acquisition devices corresponding to each of the initial lane lines in the fused lane line subset, the fused lane line subset is divided into a first sub-set and a second sub-set, the first sub-set includes initial lane lines determined by to-be-processed images collected by image acquisition devices located in front of or behind the vehicle, and the second sub-set includes initial lane lines determined by to-be-processed images collected by image acquisition devices located on the side of the vehicle.

[0021] If the first sub-set is not empty, the image acquisition devices corresponding to the initial lane lines in the second sub-set are pose optimized by the initial lane lines in the first sub-set.

[0022] In an embodiment of the present application, the pose optimization of the image acquisition devices corresponding to the initial lane lines in the second sub-set by the initial lane lines in the first sub-set includes:

[0023] acquire a plurality of line points of initial lane line points in a target processing image of an initial lane line in the second subset;

[0024] acquire line point position information of each of the line points, and lane line position information of the initial lane line in the first subset in a preset bird's eye view;

[0025] determine distances of each of the line points to the initial lane line in the first subset based on the line point position information and the lane line position information;

[0026] adjust a pose of the image acquisition device corresponding to the initial lane line in the second subset to minimize the sum of distances.

[0027] To achieve the above object and other related objects, the present application also provides a lane line fusion device, comprising:

[0028] an image acquisition module, configured to acquire a plurality of to-be-processed images;

[0029] a lane line detection module, configured to input each of the to-be-processed images into a preset lane line detection model to obtain a plurality of initial lane lines corresponding to each of the to-be-processed images;

[0030] a lane line fusion module, configured to fuse each of the initial lane lines to obtain a fused lane line;

[0031] a lane line matching module, configured to match the fused lane line with a tracking lane line to obtain a lane line result.

[0032] To achieve the above object and other related objects, the present application also provides an automobile, comprising the lane line fusion device as described above.

[0033] To achieve the above object and other related objects, the present application also provides an electronic device, comprising:

[0034] one or more processors;

[0035] a storage device, configured to store one or more programs, which, when executed by the one or more processors, cause the automobile to implement the lane line fusion method as described above.

[0036] To achieve the above object and other related objects, the present application also provides a computer readable storage medium, which stores a computer program, which, when executed by a processor of a computer, causes the computer to execute the lane line fusion method as described above.

[0037] The lane line fusion method in the present application, through processing a plurality of to-be-processed images collected by a plurality of image collection devices to obtain initial lane lines, and performing fusion, tracking and other processing on the initial lane lines, the obtained lane line information is more abundant, so as to meet the needs of automatic driving for lane line related requirements.

[0038] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0039] The drawings incorporated into the specification and forming a part thereof, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application. It is to be expressly understood that the drawings are only exemplary of some embodiments of the application, and that they are therefore to be interpreted in the scope of what is possible for a person skilled in the art, without having to resort to inventive work, in order to obtain other drawings from these. In the drawings:

[0040] Figure 1 is a flowchart of the lane line fusion method according to an exemplary embodiment of the present application;

[0041] Figure 2 is a block diagram of the lane line fusion device according to an exemplary embodiment of the present application;

[0042] Figure 3 shows a structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application;

[0043] Figure 4 shows a flowchart of an embodiment of the lane line fusion method of the present application;

[0044] Figure 5 shows Figure 4 a flowchart of a single camera lane line processing module. DETAILED DESCRIPTION

[0045] The embodiments of the present application will be described below with reference to the drawings and preferred embodiments, and other advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure in the specification. The present application can also be implemented or applied by different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustration of the present application, and are not intended to limit the protection scope of the present application.

[0046] It is to be understood that the figures provided in the following embodiments are only schematic and that the drawings only show components related to the application, and that the drawings are not drawn to scale, and that the shapes, positions and dimensions of the components can be varied in actual implementation, and that the layout of the components can be more complex.

[0047] In the following description, numerous specific details are discussed in order to provide a thorough understanding of the embodiments of the application. However, it will be apparent to one of ordinary skill in the art that the embodiments of the application can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the embodiments of the application.

[0048] First of all, it needs to be explained that the union set is a tree type data structure used to process the merging and query problems of some disjoint sets. In some set application problems with N elements, we usually let each element form a single-element set at the beginning, and then merge the sets in which the elements belonging to the same group are located according to a certain order, and repeatedly find in which set an element is located.

[0049] The Euclidean distance, also known as the Euclidean metric, is a commonly used distance definition, which refers to the real distance between two points in m-dimensional space, or the natural length of the vector (i.e. the distance from the origin to the point), and the Euclidean distance in two-dimensional and three-dimensional space is the actual distance between two points.

[0050] Figure 1 is a flowchart of a lane line fusion method shown in an exemplary embodiment of the application.

[0051] As shown in Figure 1 The lane line fusion method at least includes steps S101 to S104, which are described in detail as follows:

[0052] In step S101, a plurality of to-be-processed images are acquired, wherein the plurality of to-be-processed images are collected by a plurality of image collection devices of a vehicle.

[0053] In an embodiment of the application, the image collection device can be a camera, and the number and installation position of the camera can be set according to requirements. For example, in this embodiment, six cameras are taken as an example, and the six cameras are front-view cameras, rear-view cameras, left-front cameras, left-rear cameras, right-front cameras, and right-rear cameras, respectively, and images are collected from different directions by the cameras distributed at different positions of the vehicle.

[0054] In step S102, each of the to-be-processed images is input into a preset lane line detection model to obtain a plurality of initial lane lines corresponding to each of the to-be-processed images.

[0055] In an embodiment of the present application, inputting each of the to-be-processed images into a preset lane line detection model to obtain a plurality of initial lane lines corresponding to each of the to-be-processed images comprises:

[0056] S401, detecting each of the to-be-processed images to obtain a plurality of initial lane line points in the to-be-processed images.

[0057] In an embodiment of the present application, the initial lane line points can be detected by using a traditional method or a deep learning algorithm. The deep learning algorithm is a kind of machine learning and is prior art, which will not be described here.

[0058] S402, sampling and projecting the plurality of initial lane line points to obtain a plurality of projected lane line points, and curve fitting the plurality of projected lane line points to obtain the plurality of initial lane lines.

[0059] In an embodiment of the present application, the initial lane line point sampling comprises: sampling line points in an image frame to generate line points for fitting, the input of this part is the original line points detected by the lane line detection module, and part of the points is extracted by the sampling algorithm for subsequent fitting. The initial lane line point projection comprises: projecting the line points for fitting to a preset bird's eye view to obtain projected lane line points. For each camera, there is a pre-calibrated configuration file, and based on the transformation matrix in the configuration file, the initial lane line points in the camera frame sampled above can be projected to the coordinate system corresponding to the preset bird's eye view. The curve fitting comprises: curve fitting based on the line points on the preset bird's eye view, and for the projected lane line points obtained in the initial lane line point projection, a cubic curve is used for fitting operation. The cubic polynomial equation fitting expression is as follows:

[0060] x = C0 + C1 * y + C2 * y 2 +C3 * y 3 ,

[0061] Wherein, C0 represents the lateral offset of the road edge line near end relative to the center of the vehicle body; C1 represents the road edge line direction; C2 represents the road edge line curvature; and C3 represents the curvature change rate.

[0062] S403, lane line tracking based on the initial lane lines in the historical images and the current images.

[0063] After step S102 of inputting each of the to-be-processed images into a preset lane line detection model to obtain a plurality of initial lane lines corresponding to each of the to-be-processed images, the method further comprises steps S501-S503.

[0064] S501, sort according to the lateral offset of each initial lane line, and obtain an initial lane line sequence.

[0065] In an embodiment of the present application, the initial lane line sequence is sorted according to the lateral offset C0 of each initial lane line.

[0066] S502, if the difference between the lateral offsets of the continuous multiple initial lane lines in the initial lane line sequence is less than a preset offset threshold, determine the Euclidean distance between the continuous multiple initial lane lines.

[0067] S503, if the Euclidean distance between the continuous multiple initial lane lines is less than a preset distance threshold, determine the continuous multiple initial lane lines as a fusion lane line subset, and obtain a lane line set including multiple fusion lane line subsets.

[0068] After obtaining the lane line set including multiple fusion lane line subsets in step S503, the method further includes steps S601-S602.

[0069] S601, according to the image acquisition device corresponding to each initial lane line in the fusion lane line subset, divide the fusion lane line subset into two first and second subsets, the first subset includes initial lane lines determined by the to-be-processed images collected by the image acquisition device located in front or behind the vehicle, and the second subset includes initial lane lines determined by the to-be-processed images collected by the image acquisition device located on the side of the vehicle.

[0070] S602, if the first subset is not empty, perform pose optimization on the image acquisition device corresponding to the initial lane line in the second subset with the initial lane line in the first subset.

[0071] In step S602, performing pose optimization on the image acquisition device corresponding to the initial lane line in the second subset with the initial lane line in the first subset includes steps S701-S704.

[0072] S701, obtain multiple line points of the initial lane line point of an initial lane line in the second subset in the target processed image.

[0073] S702, obtain line point position information of each line point, and lane line position information of the initial lane line in the first subset in a preset bird's eye view.

[0074] S703, determine the distance of each line point to the initial lane line in the first subset based on each line point position information and the lane line position information.

[0075] S704, adjusting the pose of the image acquisition device corresponding to the initial lane line in the second subset, so that the distance is minimum.

[0076] For example, in a specific embodiment, for the initial lane lines L i (1<=i<=l0), the algorithm first sorts them according to their C0 parameters to get the initial lane line sequence, and then traverses them according to the index i from 1 to l0, for each i, traverses the index j from i+1 to l0, and judges whether the jth and ith lane lines belong to the same lane line according to the above difference in lateral offset and Euclidean distance judgment criterion. This process is similar to the process of constructing a union-find set. If the j1th and ith lane lines are the same lane line, and the j2th and ith lane lines are the same lane line, then the j1th and j2th lane lines should also belong to the same lane line. According to the above process, the algorithm converts the initial l0 lane lines into l1 lane lines L i (1<=i<=l1), where L i =[l i,1 ,...,l i,ni ], 1<=l i,j =l 0, 1<=j<=n i , indicates that the lane line with the original index l i,j now corresponds to the lane line set L i obtained after merging.

[0077] For the lane line set L i , the algorithm divides it into two disjoint subsets L i1 and L i2 , where the initial lane lines in L i1 are detected from the front or rear camera, and L i2 is the initial lane line detected from the front left, front right, rear left, and rear. If L i1 is not empty, then based on the line points belonging to the initial lane lines in L i1 , the lane line fitting is performed, and the fitting result is taken as the curve fitting of the lane line in L i , and the result is also taken as the curve fitting of the lane line in L i2The true value of the result obtained after lane line fitting in the image. In this way, for the left front, right front, left rear and right rear, the algorithm can establish four sets of mapping sets FL, FR, RL and RR. Taking FL as an example, each mapping element {l->L} in the mapping set represents that a certain initial lane line l detected from the left front camera is mapped to the lane line result L in the preset bird's eye view. The line point coordinates p in l are saved from the original image. Based on the criterion that the sum of the distances of all p projected onto L on the preset bird's eye view is minimized, the algorithm can optimize the corresponding camera pose.

[0078] In step S103, the initial lane lines are fused to obtain a fused lane line.

[0079] The initial lane lines of the plurality of cameras are fused to obtain a fused lane line, and the camera correlation relationship is obtained by using the fused lane line, and the pose of the camera is optimized according to the camera correlation relationship. For example, due to the overlap of the viewing angles between some cameras, the plurality of lane lines projected onto the bird's eye view by the six cameras can be redundant. Therefore, the algorithm needs to fuse these lane line results, and update the pose of the camera by using the overlap information. Generally, due to the existence of two parallel lane lines in the pictures of the front-view camera and the rear-view camera, the poses of the two cameras have been optimized in the corresponding single-camera processing module.

[0080] In an embodiment of the present application, fusing the initial lane lines to obtain a fused lane line comprises at least one of the following:

[0081] S801, if the difference between the lateral offsets C0 of at least two initial lane lines is less than a preset offset threshold, and the Euclidean distance is less than a preset distance threshold, at least two initial lane lines are fused to obtain a fused lane line.

[0082] In an embodiment of the present application, the specific value of the preset offset threshold and the preset distance threshold can be set according to requirements. Wherein, the position of the initial lane line in space can be basically confirmed by the lateral offset C0; and the parameters of the initial lane line are known, and by taking some specific points, for example, taking the same x on two initial lane lines to obtain y1 and y2 respectively, the difference between y1 and y2 is the Euclidean distance of the two initial lane lines.

[0083] S802, if the line type attribute parameters of at least two initial lane lines are consistent, at least two initial lane lines are fused to obtain a fused lane line. Wherein, the line type attribute parameters include lane line color and lane line type.

[0084] In an embodiment of the present application, the criterion that the algorithm considers two initial lane lines as corresponding to the same lane line must be met S801, and S802 can be used as an auxiliary judgment criterion, but not as a necessary judgment criterion.

[0085] In step S104, the fusion lane line is matched with the tracking lane line to obtain a lane line result.

[0086] In an embodiment of the present application, based on the fusion lane line in the historical image and the current image, lane line tracking is performed to obtain a tracking lane line, the fusion lane line is matched with the tracking lane line to obtain a lane line structure.

[0087] Figure 4 Flowchart of an embodiment of the lane line fusion method of the present application.

[0088] As shown in Figure 4 , the to-be-processed images input by multiple cameras are input to a single camera lane line processing module. The multiple camera input (multiple image acquisition devices) includes camera 1, camera 2, …, camera n, and camera 1, camera 2, …, camera n input the to-be-processed images acquired by themselves to the corresponding single camera lane line processing module (lane line detection model) respectively.

[0089] Each single camera lane line processing module obtains a detected lane line (initial lane line) after processing the to-be-processed image.

[0090] The detected lane line is fused to obtain a fused lane line (fusion lane line).

[0091] The fused lane line is tracked and updated to obtain a lane line result.

[0092] Figure 5 Flowchart of a single camera lane line processing module. Figure 4

[0093] As shown in Figure 5 , the to-be-processed image acquired by single camera input is processed after lane line detection to obtain lane line points (initial lane line points).

[0094] The lane line points are sampled and projected to obtain lane line points on the bird's eye view (projected lane line points).

[0095] Curve fitting is performed on the lane line points on the bird's eye view, and the fitted lane line (initial lane line) is tracked and updated with the tracked lane line to obtain a lane line result.

[0096] ​The device embodiment of the present application is introduced below, which can be used to execute the lane line fusion method in the above embodiment of the present application. For details not disclosed in the device embodiment of the present application, refer to the above-mentioned embodiment of the lane line fusion method.

[0097] Figure 2 The block diagram of the lane line fusion device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 the implementation environment shown. The device can also be applicable to other exemplary implementation environments and be specifically configured in other devices, and the present embodiment does not limit the implementation environment to which the device is applicable.

[0098] As Figure 2 shown, the lane line fusion device of the example includes an image acquisition module 201, a lane line detection module 202, a lane line fusion module 203, and a lane line matching module 204.

[0099] The image acquisition module 201 is configured to acquire a plurality of to-be-processed images. The to-be-processed images in the present embodiment can be acquired by a camera.

[0100] In an embodiment, the lane line detection module 202 is configured to input each of the to-be-processed images into a preset lane line detection model to obtain a plurality of initial lane lines corresponding to each of the to-be-processed images.

[0101] In an embodiment, the lane line fusion module 203 is configured to fuse each of the initial lane lines to obtain a fused lane line.

[0102] In an embodiment, the lane line matching module 204 is configured to match the fused lane line with a tracked lane line to obtain a lane line result.

[0103] In an embodiment, the lane line fusion device further includes a lane line display module and a lane line output module. The lane line display module is configured to display the lane line result on a camera screen and a bird's eye view. The lane line output module is configured to output the lane line result to other modules outside the lane line fusion device.

[0104] In an embodiment, the lane line fusion device further includes a synchronization module. The synchronization module is configured to extract time-synchronized images from image sequences of a plurality of image acquisition modules. The number of image acquisition modules is a plurality, and the image acquisition modules can be cameras. The to-be-processed images acquired by the plurality of cameras are input into the synchronization module. Since the input of the present algorithm system is a time-synchronized camera image sequence, i.e., it is necessary to ensure that the image inputs of the plurality of cameras in a single processing process are as consistent as possible in the time stamp, the synchronization module is responsible for extracting time-synchronized images from the image sequences of the plurality of cameras.

[0105] The lane line fusion method and the lane line fusion device provided by the application can perform lane line detection, fitting, tracking, fusion and other processes under the condition of omnidirectional multi-camera input, thereby meeting the requirements of automatic driving on lane lines. The lane line fusion device is built by multiple modules, and the modules are less strongly coupled, so that the sub-modules can be replaced, modified and the like, the efficiency of analyzing and solving problems when local functions encounter problems is higher, and the cost of re-embedding the original algorithm system is also relatively low. Compared with the traditional lane line processing algorithm system which only relies on one camera for processing, the lane line information obtained by the application based on dynamic calibration of lane lines is richer and more complete.

[0106] In an embodiment, the application also provides an automobile comprising the lane line fusion device in any of the preceding embodiments.

[0107] Embodiments of the application also provide an electronic device comprising: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the automobile to implement the lane line fusion method provided in each of the above embodiments.

[0108] Embodiments of the application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of a computer, causes the computer to perform the lane line fusion method provided in each of the above embodiments.

[0109] Figure 3 A structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the application is shown. It should be noted that, Figure 3 The computer system 300 of the electronic device shown is only an example and should not limit the functions and use range of embodiments of the application.

[0110] As Figure 3 shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 302 or programs loaded from a storage portion 308 into a random access memory (RAM) 303, such as performing the methods described in the above embodiments. In the RAM 303, various programs and data required for system operation are also stored. The CPU 301, the ROM 302 and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0111] The following components are connected to the I / O interface 305: an input part 306 including a keyboard, a mouse, etc.; an output part 307 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc., and a speaker, etc.; a storage part 308 including a hard disk, etc.; and a communication part 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as necessary. A removable medium 311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 310 as necessary, so that a computer program read out therefrom is installed in the storage part 308 as necessary.

[0112] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing a computer program for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the system of the present application are executed.

[0113] It should be noted that the computer-readable medium in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may, for example, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, the computer-readable signal medium can include a data signal propagated in a baseband or as a carrier wave in a propagated data signal, in which the computer-readable computer program is carried. Such a propagated data signal can take on many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate, or transmit the program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted in any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination thereof.

[0114] The flowcharts and block diagrams in the drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. Each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by special-purpose hardware-based systems, which perform the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0115] The units described in the embodiments of the present application can be implemented by software or by hardware, and the units described can be arranged in a processor. In some cases, the names of the units do not limit the units themselves.

[0116] Another aspect of the present application also provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the lane line fusion method described above. The computer readable storage medium can be included in the electronic device described in the above embodiments, or can exist separately and not be assembled into the electronic device.

[0117] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, such division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into a plurality of modules or units.

[0118] Another aspect of the present application also provides a computer program product or computer program, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device perform the lane line fusion method provided in the above embodiments.

[0119] From the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) execute the methods according to the embodiments of the present application.

[0120] In the above embodiments, unless otherwise specified, the common objects are described by using the ordinal numbers such as "first", "second", etc., which only represent different instances of the same object, and do not mean that the described objects must be arranged in a given order, whether in time, space, sequence or any other manner.

[0121] In the above embodiments, references to "this embodiment," "an embodiment," "another embodiment," or "other embodiments" in the specification indicate that a specific feature, structure, or characteristic described in connection with the embodiments is included in at least some embodiments, but not necessarily all embodiments. Multiple appearances of "this embodiment," "an embodiment," or "another embodiment" do not necessarily all refer to the same embodiment. If the specification describes a component, feature, structure, or characteristic as "may," "may," or "can" be included, then that specific component, feature, structure, or characteristic is not necessarily required to be included. If the specification or claims refer to an element "a," it does not mean that there is only one element. If the specification or claims refer to "an additional" element, it does not exclude the existence of more than one additional element.

[0122] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory structures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed. The embodiments of the invention are intended to cover all such substitutions, modifications, and variations falling within the broad scope of the appended claims.

[0123] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A lane line fusion method, characterized in that, The method includes: Multiple images to be processed are acquired, wherein the multiple images to be processed are acquired through multiple image acquisition devices of the vehicle; Each of the images to be processed is input into a preset lane detection model to obtain multiple initial lane lines corresponding to each image to be processed. The initial lane lines are sorted according to their lateral offsets to obtain an initial lane line sequence. If the difference in the lateral offsets of multiple consecutive initial lane lines in the initial lane line sequence is less than a preset offset threshold, the Euclidean distance between the multiple consecutive initial lane lines is determined. If the Euclidean distance between multiple consecutive initial lane lines is less than a preset distance threshold, the multiple consecutive initial lane lines are determined as a fused lane line subset to obtain a lane line set including multiple fused lane line subsets. The initial lane lines are merged to obtain merged lane lines; The fused lane lines are matched with the tracked lane lines to obtain the lane line results.

2. The lane line fusion method according to claim 1, characterized in that, Each of the images to be processed is input into a preset lane detection model to obtain multiple initial lane lines corresponding to each image to be processed, including: Each of the images to be processed is detected to obtain multiple initial lane line points in the images to be processed. Multiple initial lane line points are sampled and projected to obtain multiple projected lane line points. Curve fitting is then performed on the multiple projected lane line points to obtain multiple initial lane lines.

3. The lane line fusion method according to claim 1, characterized in that, The initial lane lines are merged to obtain merged lane lines that include at least one of the following: If the difference in lateral offset between at least two initial lane lines is less than a preset offset threshold, and the Euclidean distance is less than a preset distance threshold, then the at least two initial lane lines are merged to obtain a merged lane line. If at least two initial lane lines have the same line type attribute parameters, the at least two initial lane lines are merged to obtain a merged lane line, wherein the line type attribute parameters include lane line color and lane line type.

4. The lane line fusion method according to claim 1, characterized in that, After obtaining a set of lane lines that includes multiple fused lane line subsets, the method further includes: Based on the image acquisition device corresponding to each initial lane line in the fused lane line subset, the fused lane line subset is divided into two first subsets and a second subset. The first subset includes the initial lane lines determined by the images to be processed acquired by the image acquisition device located in front of or behind the vehicle, and the second subset includes the initial lane lines determined by the images to be processed acquired by the image acquisition device located on the side of the vehicle. If the first subset is not empty, the image acquisition device corresponding to the initial lane line in the second subset is used to perform pose optimization based on the initial lane line in the first subset.

5. The lane line fusion method according to claim 4, characterized in that, Pose optimization of the image acquisition device corresponding to the initial lane lines in the second subset based on the initial lane lines in the first subset includes: Obtain multiple line points of the initial lane line point in the target processed image from the second subset of the initial lane line; Obtain the position information of each line point, and the position information of the initial lane line in the first subset in the preset bird's-eye view; Based on the position information of each line point and the position information of the lane line, determine the distance from each line point to the initial lane line in the first subset; Adjust the pose of the image acquisition device corresponding to the initial lane lines in the second subset to minimize the distance.

6. A lane line fusion device, characterized in that, include: The image acquisition module is used to acquire multiple images to be processed; The lane line detection module is used to input each of the images to be processed into a preset lane line detection model to obtain multiple initial lane lines corresponding to each of the images to be processed. The lane line fusion module is used to fuse the initial lane lines to obtain fused lane lines, sort the initial lane lines according to their lateral offsets to obtain an initial lane line sequence; if the difference in the lateral offsets of multiple consecutive initial lane lines in the initial lane line sequence is less than a preset offset threshold, the Euclidean distance between the multiple consecutive initial lane lines is determined. If the Euclidean distance between multiple consecutive initial lane lines is less than a preset distance threshold, the multiple consecutive initial lane lines are determined as a subset of fused lane lines, resulting in a set of lane lines including multiple subsets of fused lane lines. The lane line matching module is used to match the fused lane line with the tracked lane line to obtain the lane line result.

7. A car, characterized in that, Includes the lane line fusion device as described in claim 6.

8. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause a vehicle to implement the lane fusion method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the lane fusion method as described in any one of claims 1 to 5.

Images