Multi-camera lane line fusion method, apparatus, computer equipment and storage medium
By acquiring multi-view images and fusing lane line key points and features through a deep learning lane line detection model, the accuracy and robustness issues of multi-camera lane line fusion are solved, achieving high-precision fusion in complex scenes.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN JIMU INTELLIGENT TECH CO LTD
- Filing Date
- 2022-12-05
- Publication Date
- 2026-06-30
Smart Images
Figure CN116246130B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent driving technology, and in particular to a multi-camera lane line fusion method, apparatus, computer equipment, and storage medium. Background Technology
[0002] Current lane detection methods mostly rely on forward-looking monocular systems, lacking reliable multi-camera lane fusion solutions. Existing multi-camera lane fusion methods largely depend on traditional multi-view geometry, which places high demands on the calibration parameters of the addition, the flatness of the ground plane, and the stability of the camera structure. Therefore, the accuracy of the calibration parameters of each lens's intrinsic and extrinsic parameters is crucial. Errors in lane fusion can easily lead to significant errors in the lane fitting parameters, significantly impacting downstream control and planning tasks in autonomous driving. Furthermore, due to the high complexity of natural scenes, existing algorithms cannot adequately support lane fusion under multi-camera conditions, resulting in poor accuracy, scene adaptability, and robustness in lane fusion.
[0003] Existing lane line fusion methods for multiple shots mostly rely on traditional multi-view geometry, which makes it difficult to achieve lane line fusion in natural scenes with multiple shots.
[0004] Tasks closely related to this invention include:
[0005] 1. Deep learning-based lane detection and segmentation model. This model uses deep learning detection algorithms to detect lane lines from the foreground through segmentation or point regression.
[0006] 2. AVM multi-lens stitching based on camera calibration: The intrinsic parameters and relative positions of multiple lenses are calibrated using traditional calibration algorithms. Finally, the images from each camera are mapped to the same coordinate system and fused.
[0007] 3. A deep learning-based multi-camera target tracking algorithm detects target ROIs between multiple cameras, performs cropping and scaling, feeds them into a deep learning algorithm for feature extraction, and finally performs feature matching by calculating the Euclidean distance between features to complete target tracking under multiple cameras. Summary of the Invention
[0008] In view of this, embodiments of the present invention provide a multi-camera lane line fusion method to solve the technical problems of poor accuracy, scene adaptability, and robustness in existing multi-camera lane line fusion techniques. The method includes:
[0009] Acquire lane line images from multiple perspectives;
[0010] The lane line image from each viewpoint is input into the lane line detection model corresponding to each viewpoint, and the lane line detection model outputs the lane line key points and lane line features of the lane line image from the corresponding viewpoint.
[0011] Based on the lane line key points and lane line features of the lane line images from various perspectives, the lane lines in the lane line images from various perspectives are fused.
[0012] This invention also provides a multi-camera lane line fusion device to solve the technical problems of poor accuracy, scene adaptability, and robustness in existing multi-camera lane line fusion technologies. The device includes:
[0013] The image acquisition module is used to acquire lane line images from multiple perspectives.
[0014] The lane line detection module is used to input the lane line image from each viewpoint into the lane line detection model corresponding to each viewpoint, and the lane line detection model outputs the lane line key points and lane line features of the lane line image from the corresponding viewpoint.
[0015] The lane line fusion module is used to fuse the lane lines in the lane line images from different viewpoints based on the lane line key points and lane line features.
[0016] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned multi-camera lane line fusion methods to solve the technical problems of poor accuracy, scene adaptability, and robustness in existing multi-camera lane line fusion technologies.
[0017] This invention also provides a computer-readable storage medium storing a computer program that executes any of the above-described multi-camera lane line fusion methods, in order to solve the technical problems of poor accuracy, scene adaptability, and robustness in the existing multi-camera lane line fusion technology.
[0018] Compared with existing technologies, the beneficial effects achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: After acquiring lane line images from multiple perspectives, the lane line images from each perspective are input into the lane line detection model corresponding to each perspective, and then the lane line detection model outputs the lane line key points and lane line features of the lane line images from the corresponding perspectives; finally, the lane lines in the lane line images from each perspective are fused based on the lane line key points and lane line features. This implements a lane line detection model based on deep learning to detect lane line key points and lane line features of lane line images from each perspective, and then fuses the lane lines in the lane line images from each perspective based on the lane line key points and lane line features. This avoids using multi-view geometry-based methods for multi-view lane line fusion, which is beneficial to improving the accuracy and robustness of multi-view lane line fusion. Furthermore, the lane line detection model is applicable to lane line images in any scenario, and can perform lane line detection on lane line images in complex natural scenes, which is beneficial to improving the scene adaptability of multi-view lane line fusion. Attached Figure Description
[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a flowchart of a multi-camera lane line fusion method provided in an embodiment of the present invention;
[0021] Figure 2 This is a flowchart illustrating an implementation of the multi-camera lane line fusion method provided in an embodiment of the present invention;
[0022] Figure 3 This is a structural block diagram of a computer device provided in an embodiment of the present invention;
[0023] Figure 4 This is a structural block diagram of a multi-lens lane line fusion device provided in an embodiment of the present invention. Detailed Implementation
[0024] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0025] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0026] In this embodiment of the invention, a multi-camera lane line fusion method is provided, such as... Figure 1 As shown, the method includes:
[0027] Step S101: Obtain lane line images from multiple perspectives;
[0028] Step S102: Input the lane line image from each viewpoint into the lane line detection model corresponding to each viewpoint. The lane line detection model outputs the lane line key points and lane line features of the lane line image from the corresponding viewpoint.
[0029] Step S103: Based on the lane line key points and lane line features of the lane line images from each viewpoint, fuse the lane lines in the lane line images from each viewpoint.
[0030] Depend on Figure 1 As shown in the flowchart, in this embodiment of the invention, after acquiring lane line images from multiple perspectives, the lane line images from each perspective are input into the lane line detection model corresponding to each perspective. The lane line detection model then outputs the lane line key points and lane line features of the corresponding perspective. Finally, based on the lane line key points and lane line features of the lane line images from each perspective, the lane lines in the lane line images from each perspective are fused. This achieves the detection of lane line key points and lane line features from lane line images from various perspectives using a deep learning-based lane line detection model, and then fuses the lane lines in the lane line images from various perspectives based on these key points and features. This avoids using multi-view geometry-based methods for multi-view lane line fusion, which is beneficial for improving the accuracy and robustness of multi-view lane line fusion. Furthermore, the lane line detection model is applicable to lane line images in any scenario, and can perform lane line detection on lane line images in complex natural scenes, which is beneficial for improving the scene adaptability of multi-view lane line fusion.
[0031] In practical implementation, the aforementioned multi-view lane line images refer to lane line images acquired from multiple different perspectives of the vehicle, for example, such as... Figure 2 As shown, lane line images can be viewed from multiple perspectives, such as front view, side view, and rear view of the vehicle.
[0032] Specifically, lane line images can be captured from multiple perspectives using image acquisition devices such as cameras and webcams.
[0033] In specific implementation, in order to achieve lane line detection through deep learning and improve the accuracy and stability of multi-view lane line fusion, in this embodiment, the lane line detection model can be obtained by training a convolutional neural network (CNN) using lane line images and lane line key points and lane line features of lane line images as samples.
[0034] In practical implementation, the aforementioned lane detection model, based on deep learning algorithms, achieves a novel method for predicting and encoding the location of key lane points. Through the constraints of the loss function, it automatically obtains the feature encoding and location encoding information of lane lines in the image. For example...
[0035] (1) Prediction of lane line key points: The lane line detection model above draws on the prediction method of UFLD, but makes improvements on the basis of UFLD. a. On the basis of UFLD prediction, the original fully connected layer is replaced with Upsample+conv layer, which reduces the computational cost of the network. b. On the basis of the original classification of UFLD, a regression component is added, which makes the regression of lane line key points more stable and accurate. c. The original UFLD divides the prediction cells based on the horizontal direction of the image, which performs very poorly in the scene of large curves. Therefore, the vertical direction of the image is divided into prediction cells to improve the stability and accuracy of the scene of large curves.
[0036] (2) Position encoding method: The original position encoding is only the encoding of the relative position of the lane lines. In the algorithm of this application, the type of lane lines (solid line, dashed line, herringbone line, double yellow line), the color of the lane lines, and the relative position of the lane lines (i.e. the above position information) are encoded in a new way through permutation and combination.
[0037] (3) Lane line feature encoding: The lane line key point prediction layer performs pooling operation to transform the (n,m,1) feature map into a (n,1) or (m,1) feature vector. The lane line feature encoding is the aforementioned feature vector.
[0038] (4) Loss Function: New lane line regression loss, lane line confidence loss, lane line type loss, and lane line color loss are added to the original UFLD. Based on the current loss function, the performance of the deep learning algorithm is predicted during network training, making the prediction of lane line key points and lane line position encoding more accurate, and the lane line feature vectors have better discriminability (able to better distinguish different lane lines through similarity calculation).
[0039] In practice, the lane detection model described above can provide each lane line with a (n,1) float-shaped feature vector output by deep learning.
[0040] In practice, the aforementioned location information is a one-hot integer classification vector predicted by the lane detection model (assuming that the position of the lane line can be divided into 4 categories: the first lane on the left, the second lane on the left, the first lane on the right, and the second lane on the right, plus the possibility of no lane line, so the location information can be divided into 5 categories. The location information can be a specific textual description of the location or a location code. For example, the location code of the first lane on the left is 10000, the location code of no lane line is 00001, and so on).
[0041] In practical implementation, the lane line detection model has low requirements for the scene and strong applicability. It is suitable for various complex scenes and multi-view geometric schemes. The installation conditions and calibration accuracy of the image acquisition equipment are very high. In contrast, the lane line detection model has low requirements for the installation conditions and calibration accuracy of the image acquisition equipment and can be adapted to most products, thus improving the applicability of the scene.
[0042] In practical implementation, the lane line detection model is more portable than the Transformer-based BEV solution. The Transformer has a much higher difficulty in embedded porting and data acquisition. Compared with the Transformer-based BEV solution, the lane line detection model adopted in this application has stronger portability and easier sample acquisition. The overall solution is more suitable for mass production in terms of cost, time efficiency and portability.
[0043] In practical implementation, to further improve the accuracy of multi-view lane line fusion, this embodiment also proposes a specific method for lane line fusion based on lane line key points and lane line features, for example,
[0044] Based on the key points of the lane lines in the lane line images from each viewpoint, fit the lane line curves and calculate the physical distance between the lane line curves corresponding to the lane line images from each viewpoint.
[0045] Based on the lane line features of the lane line images from each viewpoint, calculate the similarity between lane lines in the lane line images from each viewpoint.
[0046] Based on the physical distance between the lane line curves corresponding to the lane line images from different viewpoints and the similarity between the lane lines in the lane line images from different viewpoints, the lane lines in the lane line images from different viewpoints are fused.
[0047] Specifically, such as Figure 2 As shown, after inputting lane line images from various perspectives into the lane line detection model, the model can output lane line key points and lane line features for each perspective. Then, based on the lane line features, the similarity between lane lines in lane line images from various perspectives can be calculated. Furthermore, lane line curves can be fitted based on lane line key points, and the physical distance between lane line curves corresponding to lane line images from various perspectives can be calculated.
[0048] In specific implementation, such as Figure 2 As shown, lane line curves can be fitted in the following way: for example, the key points of lane lines in lane line images from various viewpoints are mapped to the same world coordinate system, wherein the world coordinate system has the vehicle center as the origin; in the world coordinate system, lane line curves are fitted based on the key points of lane lines in the lane line images from each viewpoint to obtain the lane line curves corresponding to the lane line images from each viewpoint, so as to calculate the physical distance between the lane line curves corresponding to the lane line images from each viewpoint in the world coordinate system.
[0049] In specific implementation, such as Figure 2 As shown, lane line features may include any one or any combination of the following:
[0050] Feature vector, location information, color, and type.
[0051] In specific implementation, such as Figure 2 As shown, after obtaining the lane line features of the lane line images from various viewpoints, the similarity between lane lines in the lane line images from different viewpoints can be calculated based on the feature vectors in the lane line features. For example, the similarity between lane lines can be calculated in the following way:
[0052] The cosine similarity between two lane lines is calculated using the following formula:
[0053]
[0054] Where cosθ is the cosine similarity; A is the feature vector of a lane line; B is the feature vector of a lane line. For example, A and B are feature vectors with dimensions (n,1) and (n,1) respectively.
[0055] The cosine similarity is normalized using the following formula to obtain a similarity value in the range of 0 to 1:
[0056] Sim=(cosθ+1) / 2
[0057] Where Sim is the similarity value obtained after normalization.
[0058] In practice, after obtaining the physical distance between the lane line curves corresponding to the lane line images from each viewpoint and the similarity between the lane lines in the lane line images from each viewpoint, the lane lines in the lane line images from each viewpoint can be fused by combining the physical distance between the lane line curves corresponding to the lane line images from each viewpoint and the similarity between the lane lines in the lane line images from each viewpoint. For example, the Hungarian matching algorithm can be used to fuse the lane lines in the lane line images from each viewpoint.
[0059] In specific implementation, the Hungarian matching algorithm is used to fuse the lane lines in the lane line images from different viewpoints based on the physical distance between the lane line curves corresponding to the lane line images from different viewpoints and the similarity between the lane lines in the lane line images from different viewpoints. Figure 2 As shown, this can be achieved through the following steps: for example, calculating a cost matrix based on the similarity between lane lines in lane line images from different perspectives, and then fusing lane lines in lane line images from different perspectives based on the cost matrix and the physical distance between the lane line curves corresponding to the lane line images from different perspectives to obtain the fusion result of lane lines under multiple shots.
[0060] In practical implementation, the multi-lens lane line fusion method described in this application has no limitations on the number of lenses (i.e., the number of viewpoints in the lane line image), the number of lane lines, or the usage environment. Furthermore, the calculation processes for the similarity between lane lines and the distance between lane line curves can be run independently or jointly, ensuring a balance between accuracy and efficiency. In addition, compared to single-view lane line detection schemes, where the front-view camera cannot directly observe lane lines on the sides and rear of the vehicle and can only infer the vehicle body through fitted curves, resulting in poor stability and applicability of the fitted lane lines, this application achieves higher fitting stability through the fusion of lane lines from multiple viewpoints, and has significant potential for future functional development.
[0061] In this specific implementation, taking one lane as an example, the process of fusion of lane lines under multiple perspectives based on the above-mentioned multi-lens lane line fusion method is described in detail. The fusion method for other lanes is the same as that for this lane.
[0062] First, the input multi-view lane line images (e.g., front view image, side view image, and rear view image) are preprocessed and then input into the lane line detection model corresponding to each view. The lane line detection model performs lane line detection and segmentation on the lane line images, extracts the key point information and mask information of the lane lines in the image, and simultaneously obtains lane line feature information such as feature vector, position encoding (i.e., the aforementioned position information), color, and line type (i.e., type).
[0063] Then, the lane line key points of the lane line images obtained from each viewpoint are mapped to the world coordinate system with the center of the vehicle body as the origin. The lane line curve is fitted based on the lane line key points of the lane line images from each viewpoint to obtain the lane line curves corresponding to the lane line images from each viewpoint. The physical distance between the lane line curves corresponding to the lane line images from multiple viewpoints is calculated in the world coordinate system.
[0064] Finally, using the lane line features themselves, such as position encoding, color, and line type, the similarity between lane lines in lane line images from different viewpoints is calculated. Combining physical distance and similarity, the traditional Hungarian matching algorithm is used to perform dynamic fusion of lane lines under multiple shots.
[0065] In this embodiment, a computer device is provided, such as... Figure 3 As shown, it includes a memory 301, a processor 302, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described multi-camera lane line fusion methods.
[0066] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.
[0067] In this embodiment, a computer-readable storage medium is provided, which stores a computer program that performs any of the above-described multi-camera lane fusion methods.
[0068] Specifically, computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.
[0069] Based on the same inventive concept, this invention also provides a multi-camera lane line fusion device, as described in the following embodiments. Since the principle of the multi-camera lane line fusion device in solving the problem is similar to that of the multi-camera lane line fusion method, the implementation of the multi-camera lane line fusion device can refer to the implementation of the multi-camera lane line fusion method, and repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0070] Figure 4 This is a structural block diagram of a multi-lens lane line fusion device according to an embodiment of the present invention, such as... Figure 4 As shown, the device includes:
[0071] Image acquisition module 401 is used to acquire lane line images from multiple perspectives;
[0072] The lane line detection module 402 is used to input the lane line image from each viewpoint into the lane line detection model corresponding to each viewpoint, and the lane line detection model outputs the lane line key points and lane line features of the lane line image from the corresponding viewpoint.
[0073] The lane line fusion module 403 is used to fuse the lane lines in the lane line images from different viewpoints based on the lane line key points and lane line features of the lane line images from different viewpoints.
[0074] In one embodiment, the lane line fusion module includes:
[0075] The distance calculation unit is used to fit the lane line curve based on the lane line key points of the lane line image under each viewpoint, and calculate the physical distance between the lane line curves corresponding to the lane line images under each viewpoint.
[0076] The similarity calculation unit is used to calculate the similarity between lane lines in the lane line images under each viewpoint based on the lane line features of the lane line images under each viewpoint.
[0077] The lane line fusion unit is used to fuse the lane lines in the lane line images from different viewpoints based on the physical distance between the lane line curves corresponding to the lane line images from different viewpoints and the similarity between the lane lines in the lane line images from different viewpoints.
[0078] In one embodiment, the distance calculation unit is further configured to map the lane line key points of the lane line images from various viewpoints to the same world coordinate system, wherein the world coordinate system has the vehicle center as the origin; and in the world coordinate system, based on the lane line key points of the lane line images from each viewpoint, fit a lane line curve to obtain the lane line curve corresponding to the lane line images from each viewpoint.
[0079] In one embodiment, the similarity calculation unit is used to calculate the cosine similarity between two lane lines using the following formula:
[0080]
[0081] Where cosθ is the cosine similarity; A is the feature vector of a lane line; B is the feature vector of a lane line;
[0082] The cosine similarity is normalized using the following formula to obtain a similarity value in the range of 0 to 1:
[0083] Sim=(cosθ+1) / 2
[0084] Where Sim is the similarity value obtained after normalization.
[0085] In one embodiment, the lane line fusion unit is used to fuse lane lines in the lane line images from different viewpoints using a Hungarian matching algorithm, based on the physical distance between lane line curves corresponding to the lane line images from different viewpoints and the similarity between lane lines in the lane line images from different viewpoints.
[0086] This invention achieves the following technical effects: After acquiring lane line images from multiple perspectives, the model inputs the lane line image from each perspective into a corresponding lane line detection model. The model then outputs lane line key points and lane line features for each perspective. Finally, based on these key points and features, the lane lines in the lane line images from each perspective are fused. This invention implements a deep learning-based lane line detection model to detect lane line key points and features from various perspectives, and then fuses the lane lines based on these key points and features. This avoids using multi-view geometry-based methods for multi-view lane line fusion, improving the accuracy and robustness of multi-view lane line fusion. Furthermore, the lane line detection model is applicable to lane line images in any scenario, including complex natural scenes, enhancing the scene adaptability of multi-view lane line fusion.
[0087] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.
[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A multi-camera lane line fusion method, characterized in that, include: Acquire lane line images from multiple perspectives; The lane line images from each viewpoint are input into the lane line detection model corresponding to each viewpoint. The lane line detection model outputs the lane line key points and lane line features of the lane line images from the corresponding viewpoint. The lane line detection model implements a new prediction method and position encoding method for lane line key points based on deep learning algorithms. For the prediction of lane line key points, the original fully connected layer is replaced with an Upsample+conv layer based on the prediction of UFLD. A regression component is added to the original classification of UFLD. A prediction cell based on the vertical direction of the image is added to the UFLD model. Based on the key points and features of the lane lines in the lane line images from various perspectives, the lane lines in the lane line images from various perspectives are fused. Based on the lane line key points and lane line features of the lane line images from various viewpoints, the lane lines in the lane line images from various viewpoints are fused, including: Based on the key points of the lane lines in the lane line images from each viewpoint, fit the lane line curves and calculate the physical distance between the lane line curves corresponding to the lane line images from each viewpoint. Based on the lane line features of the lane line images from each viewpoint, calculate the similarity between lane lines in the lane line images from each viewpoint. Based on the physical distance between the lane line curves corresponding to the lane line images from each viewpoint and the similarity between the lane lines in the lane line images from each viewpoint, the lane lines in the lane line images from each viewpoint are fused. Based on the lane line key points of the lane line image from each viewpoint, fit a lane line curve, including: The key points of the lane lines in the lane line images from various perspectives are mapped to the same world coordinate system, wherein the world coordinate system has the center of the vehicle body as the origin. In the world coordinate system, based on the key points of the lane lines in the lane line image at each viewpoint, the lane line curve is fitted to obtain the lane line curve corresponding to the lane line image at each viewpoint.
2. The multi-camera lane line fusion method as described in claim 1, characterized in that, Based on the lane line features of the lane line images from each viewpoint, the similarity between lane lines in the lane line images from each viewpoint is calculated, including: The cosine similarity between two lane lines is calculated using the following formula: = in, Let A be the cosine similarity; let B be the feature vector of a lane line; and let C be the feature vector of a lane line. The cosine similarity is normalized using the following formula to obtain a similarity value in the range of 0 to 1: Try = Where Sim is the similarity value obtained after normalization.
3. The multi-camera lane line fusion method as described in claim 1, characterized in that, Based on the physical distance between lane line curves corresponding to the lane line images from different viewpoints and the similarity between lane lines in the lane line images from different viewpoints, the lane lines in the lane line images from different viewpoints are fused, including: The Hungarian matching algorithm is used to fuse the lane lines in the lane line images from different viewpoints based on the physical distance between the lane line curves corresponding to the lane line images from different viewpoints and the similarity between the lane lines in the lane line images from different viewpoints.
4. The multi-lens lane line fusion method as described in any one of claims 1 to 3, characterized in that, The lane line features include any one or any combination of the following: Feature vector, location information, color, and type.
5. The multi-lens lane line fusion method as described in any one of claims 1 to 3, characterized in that, The lane line detection model is obtained by training a convolutional neural network using lane line images, lane line key points, and lane line features as samples.
6. A multi-camera lane line fusion device, characterized in that, include: The image acquisition module is used to acquire lane line images from multiple perspectives. The lane detection module is used to input the lane image from each viewpoint into the lane detection model corresponding to each viewpoint. The lane detection model outputs the lane key points and lane features of the lane image from the corresponding viewpoint. The lane detection model implements a new prediction method and position encoding method for lane key points based on deep learning algorithms. For the prediction of lane key points, based on the prediction of UFLD, the original fully connected layer is replaced with an Upsample+conv layer. A regression component is added to the original classification of UFLD. A prediction cell based on the vertical direction of the image is added to the UFLD. The lane line fusion module is used to fuse the lane lines in the lane line images from different viewpoints based on the lane line key points and lane line features of the lane line images from different viewpoints. Lane line fusion module, including: The distance calculation unit is used to fit the lane line curve based on the lane line key points of the lane line image under each viewpoint, and calculate the physical distance between the lane line curves corresponding to the lane line images under each viewpoint. The similarity calculation unit is used to calculate the similarity between lane lines in the lane line images under each viewpoint based on the lane line features of the lane line images under each viewpoint. The lane line fusion unit is used to fuse the lane lines in the lane line images from different viewpoints based on the physical distance between the lane line curves corresponding to the lane line images from different viewpoints and the similarity between the lane lines in the lane line images from different viewpoints. The distance calculation unit is also used to map the lane line key points of the lane line images from various perspectives to the same world coordinate system, wherein the world coordinate system takes the center of the vehicle body as the origin; in the world coordinate system, based on the lane line key points of the lane line images from each perspective, the lane line curve is fitted to obtain the lane line curve corresponding to the lane line images from each perspective.
7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-lens lane line fusion method according to any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the multi-lens lane fusion method according to any one of claims 1 to 5.