Unmanned vehicle global lane line construction method and system based on double-end field of view fusion

Abstract

The application relates to the technical field of intelligent vehicles, and particularly provides a global lane line construction method and system for unmanned vehicles based on double-end field-of-view fusion, wherein the front end and the rear end of the unmanned vehicle are respectively provided with a front-end camera and a rear-end camera, the method comprises the following steps: acquiring front-end image data collected by the front-end camera and rear-end image data collected by the rear-end camera; performing lane line detection on the front-end image data and the rear-end image data respectively to obtain front-end lane line detection results and rear-end lane line detection results; projecting the front-end lane line and the rear-end lane line to a bird's-eye view space based on the front-end lane line detection results and the rear-end lane line detection results; and fusing and splicing the front-end lane line and the rear-end lane line in the bird's-eye view space to construct a global lane line. According to the global lane line construction method for the unmanned vehicle, the driving reliability and operation accuracy of the unmanned vehicle in tasks such as reversing and high-precision alignment can be improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicle technology, specifically to a method and system for constructing global lane lines for unmanned vehicles based on dual-end field-of-view fusion. Background Technology

[0002] With the increasing automation and intelligence of ports, the operational accuracy requirements for intelligent guided vehicles (IGVs) with bidirectional driving capabilities are constantly rising. However, most existing lane line perception solutions follow a unidirectional driving logic, meaning that the vehicle mainly relies on a single-direction camera for detection, switching to the other end's sensor when reversing or changing direction. This model has significant shortcomings in scenarios where IGVs frequently change direction: on the one hand, the information perceived by the two cameras is isolated and cannot be effectively fused in a unified space, resulting in a limited perception range and difficulty in constructing a continuous and complete global lane line; on the other hand, when the vehicle changes direction, due to the lack of an effective perception relay mechanism, the lane line information acquired by the original camera cannot be directly used by the new direction perception system, causing information breakage and reinitialization. These problems are particularly prominent when the lane lines are worn, obstructed, or discontinuous, severely restricting the driving reliability and operational accuracy of IGVs in tasks such as changing direction and high-precision alignment. Summary of the Invention

[0003] In view of this, the present invention provides a method and system for constructing global lane lines for unmanned vehicles based on dual-end field-of-view fusion, which can improve the driving reliability and operational accuracy of unmanned vehicles in tasks such as reversing and high-precision alignment.

[0004] To solve at least one of the above-mentioned technical problems, the present invention adopts the following technical solution:

[0005] The first aspect of this invention provides a method for constructing global lane lines for unmanned vehicles based on dual-end field-of-view fusion. The unmanned vehicle is equipped with a front-end camera and a rear-end camera, respectively. The construction method includes:

[0006] Acquire front-end image data captured by the front-end camera and back-end image data captured by the back-end camera respectively;

[0007] Lane line detection is performed on the front-end image data and the back-end image data respectively, and the front-end lane line detection results and the back-end lane line detection results are obtained.

[0008] Based on the front lane line detection results and the rear lane line detection results, the front lane lines and the rear lane lines are projected onto the bird's-eye view space;

[0009] In the bird's-eye view space, the front lane lines and the rear lane lines are merged and spliced ​​to construct the global lane lines.

[0010] In one embodiment of the present invention, lane line detection is performed after preprocessing the front-end image data and the back-end image data respectively. The preprocessing includes at least one of distortion correction, image enhancement and noise reduction.

[0011] In one embodiment of the present invention, lane line detection is performed on front-end image data and back-end image data respectively to obtain front-end lane line detection results and back-end lane line detection results, including:

[0012] Semantic segmentation is performed on the front-end image data and the back-end image data to identify the lane line pixel regions corresponding to the front-end lane lines and the back-end lane lines.

[0013] Post-process the lane line pixel regions corresponding to the front lane line and the back lane line to obtain the front lane line instance structure and the back lane line instance structure.

[0014] The front-end lane line detection results are generated based on the front-end lane line instance structure, and the back-end lane line detection results are generated based on the back-end lane line instance structure.

[0015] In one embodiment of the present invention, based on the front lane line detection results and the rear lane line detection results, the front lane lines and the rear lane lines are projected onto the bird's-eye view space, including:

[0016] Based on the front lane detection results and the rear lane detection results, and using the calibration parameters of the front and rear cameras, three-dimensional points of the front lane and the rear lane are constructed in the vehicle coordinate system of the unmanned vehicle.

[0017] Based on the real-time pose information of unmanned vehicles, the 3D points of the front lane and the rear lane are transformed into the bird's-eye view spatial coordinate system to construct the spatial projection of the front lane line and the rear lane line in the bird's-eye view space.

[0018] In one embodiment of the present invention, the front lane line detection result includes the pixel coordinates and confidence information corresponding to the front lane line, and the rear lane line detection result includes the pixel coordinates and confidence information corresponding to the rear lane line.

[0019] In one embodiment of the present invention, the calibration parameters include intrinsic parameter information and extrinsic parameter information;

[0020] Based on the front and rear lane detection results, and using the calibration parameters of the front and rear cameras, three-dimensional points of the front and rear lanes are constructed in the vehicle coordinate system of the autonomous vehicle, including:

[0021] Based on the intrinsic parameter information of the front-end camera and the back-end camera, the pixel coordinates corresponding to the front lane line are converted into the front lane 3D points in the front camera coordinate system, and the pixel coordinates corresponding to the back lane line are converted into the back lane 3D points in the back camera coordinate system.

[0022] Based on the extrinsic information from the front-end and rear-end cameras, the 3D points of the front-end lane and the 3D points of the rear-end lane are transformed into the vehicle coordinate system of the unmanned vehicle.

[0023] In one embodiment of the present invention, in a bird's-eye view space, the front lane lines and the rear lane lines are merged and stitched together to construct a global lane line, including:

[0024] Based on the spatial relationship between the three-dimensional points of the front lane and the three-dimensional points of the rear lane in the bird's-eye view spatial coordinate system, the overlapping area of ​​the front lane line and the rear lane line is determined.

[0025] Based on a preset weighted fusion strategy, the overlapping areas of the front lane lines and the rear lane lines are fused using confidence information to obtain fused lane segments.

[0026] Based on spatial continuity, the lane lines are spliced ​​together with the front and rear lane lines of the merged lane segments and non-overlapping areas to construct a global lane line.

[0027] In one embodiment of the present invention, the construction method further includes:

[0028] When there are gaps between the front and rear lane lines and the merging lane segments in non-overlapping areas, the gaps are filled by interpolation.

[0029] In one embodiment of the present invention, the construction method further includes:

[0030] When an autonomous vehicle performs forward or backward direction changes, the global lane lines are used as prior information to guide or constrain the autonomous vehicle.

[0031] A second aspect of the present invention also provides a global lane line construction system for unmanned vehicles based on dual-end field-of-view fusion, wherein a front-end camera and a rear-end camera are respectively installed at the front and rear ends of the unmanned vehicle, and the construction system includes:

[0032] The image data acquisition module is used to acquire front-end image data captured by the front-end camera and back-end image data captured by the back-end camera, respectively.

[0033] The lane line detection module is used to perform lane line detection on front-end image data and back-end image data, and obtain the front-end lane line detection results and the back-end lane line detection results.

[0034] The projection module is used to project the front lane lines and the rear lane lines onto the bird's-eye view space based on the front lane line detection results and the rear lane line detection results.

[0035] The building module is used to merge and stitch together the front and rear lane lines in the bird's-eye view space to construct the global lane lines.

[0036] The above-described technical solution of the present invention has at least one of the following beneficial effects:

[0037] This invention presents a method for constructing global lane lines for unmanned vehicles based on dual-view field fusion. By acquiring front-end image data from a front-end camera and back-end image data from a back-end camera, lane line detection is performed on both images separately, yielding front-end and back-end lane line detection results. This effectively expands the perceived physical range of lane lines, overcoming the limitations of a single camera's limited field of view and providing a data foundation for constructing longer and more complete global lane lines. Furthermore, by fusing and stitching the front-end and back-end lane lines in a bird's-eye view space to construct global lane lines, the integration and complementarity of lane lines from different perspectives can be achieved, generating more continuous and complete global lane lines. This improves the reliability and operational accuracy of unmanned vehicles in tasks such as direction changing and high-precision alignment. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the implementation environment for a global lane line construction method for unmanned vehicles based on dual-end field-of-view fusion in one embodiment of the present invention;

[0039] Figure 2 This is a flowchart of a method for constructing global lane lines for unmanned vehicles based on dual-end field-of-view fusion in one embodiment of the present invention;

[0040] Figure 3 This is a flowchart illustrating the generation of front-end lane line detection results and rear-end lane line detection results in one embodiment of the present invention;

[0041] Figure 4 This is a flowchart illustrating the construction of front and rear lane lines in a bird's-eye view according to one embodiment of the present invention;

[0042] Figure 5 This is a flowchart illustrating the construction of three-dimensional points of the front lane and the rear lane in the vehicle coordinate system of an unmanned vehicle according to one embodiment of the present invention.

[0043] Figure 6 This is a flowchart illustrating the construction of global lane lines in one embodiment of the present invention;

[0044] Figure 7This is a schematic diagram of the structure of a global lane line construction system for unmanned vehicles based on dual-end field-of-view fusion in one embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0046] The following describes, with reference to the accompanying drawings, a method for constructing global lane lines for unmanned vehicles based on dual-end field-of-view fusion according to an embodiment of the present invention.

[0047] Reference manual attached Figure 1 , Figure 1 A schematic diagram illustrating the implementation environment of the global lane line construction method for unmanned vehicles based on dual-end field-of-view fusion of the present invention is shown. Figure 1 As shown, the implementation environment may include an unmanned vehicle 1001 and a computing device 1002 electrically connected to the unmanned vehicle 1001 via wired / wireless means. The unmanned vehicle 1001 is equipped with a front-end camera 1003 and a rear-end camera 1004 at its front and rear ends, respectively. The computing device 1002 may be, but is not limited to, various servers, personal computers, laptops, smartphones, tablets, and portable wearable devices. The server may be an independent server, a server cluster composed of multiple servers, or a distributed computing device. It may also be an edge server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0048] The global lane line construction method for an unmanned vehicle 1001 based on dual-view field fusion can include the following steps: The construction device can acquire front-end image data collected by the front-end camera 1003 and back-end image data collected by the back-end camera 1004, and perform lane line detection on the front-end and back-end image data respectively, obtaining the front-end lane line detection results and the back-end lane line detection results. This effectively expands the perceived physical range of the lane lines, overcomes the limitation of the limited field of view of a single camera, and provides a data foundation for constructing longer and more complete global lane lines. Then, based on the front-end and back-end lane line detection results, the construction device can project the front-end and back-end lane lines onto the bird's-eye view space, and fuse and stitch the front-end and back-end lane lines in the bird's-eye view space to construct the global lane lines. Thus, the lane lines collected by the front-end camera 1003 and the back-end camera 1004 from different perspectives are integrated and complemented in the bird's-eye view space, thereby generating more continuous and complete global lane lines and improving the driving reliability and operational accuracy of the unmanned vehicle 1001 in tasks such as reversing and high-precision alignment.

[0049] Below, please refer to the attached instruction manual. Figure 2 This illustrates a method for constructing global lane lines for unmanned vehicles based on dual-view field fusion, provided by an embodiment of the present invention. This method can be applied to... Figure 1 In the computing device 1002. Specifically, the method may include the following steps:

[0050] S100: Acquire front-end image data captured by the front-end camera and back-end image data captured by the back-end camera, respectively.

[0051] In this embodiment, during the operation of the unmanned vehicle, the computing device can acquire front-end image data collected by the front-end camera installed at the front of the unmanned vehicle and rear-end image data collected by the rear-end camera installed at the rear of the unmanned vehicle, thereby providing raw visual input for subsequent lane line detection.

[0052] S200: Perform lane line detection on the front-end image data and the back-end image data respectively, and obtain the front-end lane line detection results and the back-end lane line detection results.

[0053] In this embodiment, the computing device can perform lane line detection on the front-end image data and the back-end image data respectively, identify and output the lane line detection results in each image. By obtaining the front-end lane line detection results and the back-end lane line detection results respectively, the physical range of lane line perception can be effectively expanded, overcoming the limitation of the limited field of view of a single camera, and providing a data foundation for constructing longer and more complete global lane lines.

[0054] S300 projects the front and rear lane lines onto the bird's-eye view space based on the front and rear lane line detection results.

[0055] In this embodiment, the computing device can project the front lane detection results and rear lane detection results, which exist in different image coordinate systems, onto the same bird's-eye view space from a unified overhead perspective, based on the known camera parameters and the geometric relationship of the unmanned vehicle. This allows the front lane and rear lane lines collected from different perspectives to be integrated and complemented in the bird's-eye view space.

[0056] S400: In the bird's-eye view space, the front lane lines and the rear lane lines are merged and spliced ​​to construct the global lane lines.

[0057] In this embodiment, the computing device can merge and stitch together the front lane lines and rear lane lines that have been spatially aligned in a unified bird's-eye view space, and finally generate a global lane line with a wider coverage and better continuity, thereby improving the driving reliability and operation accuracy of unmanned vehicles in tasks such as reversing and high-precision alignment.

[0058] Below, each step will be explained in detail.

[0059] (a) Obtain front-end image data and back-end image data.

[0060] Next, the steps for acquiring front-end image data and back-end image data will be introduced (i.e., step S100).

[0061] In one embodiment of the present invention, the unmanned vehicle is equipped with a front-end camera and a rear-end camera, respectively, for real-time acquisition of raw image data within their respective fields of view. The front-end and rear-end cameras can be industrial-grade high-resolution cameras, such as cameras with wide dynamic range capabilities, to adapt to complex lighting conditions in port operations, including strong light, backlighting, and alternating shadows. The front-end and rear-end cameras can synchronously acquire images at a preset frequency, such as 15 frames per second, 30 frames per second, or other configurable frame rates, thereby ensuring the temporal correspondence between the front-end and rear-end image data.

[0062] When acquiring front-end and back-end image data, the computing device can first synchronize the front-end and back-end image data in time to reduce the impact of time errors on subsequent fusion processing. Furthermore, after acquiring the front-end and back-end image data, the computing device can perform preprocessing on each data separately. This preprocessing includes at least one of distortion correction, image enhancement, and denoising. Distortion correction can be performed on the original image based on pre-calibrated intrinsic parameter information from the camera to correct geometric distortion, eliminating the influence of radial and tangential distortion on the lane line geometry in the image data and improving the geometric accuracy of lane line positions. In the image enhancement stage, brightness adjustment, contrast enhancement, and gamma correction can be applied to the image according to the current ambient lighting conditions to improve the overall visual effect. In the presence of environmental factors such as fog, dust, or low light, defogging or denoising processing can also be performed to improve image clarity and signal-to-noise ratio, thereby enhancing the stability of subsequent lane line feature extraction and recognition. After preprocessing, the computing device will use the preprocessed front-end image data and back-end image data as input for lane line detection in subsequent steps, thereby providing a reliable image basis for constructing global lane lines in a unified bird's-eye view space.

[0063] (ii) Lane line detection of front-end image data and back-end image data.

[0064] The following describes the lane line detection steps (i.e., step S200) for the front-end image data and the back-end image data, respectively. Figure 3 As shown, in one embodiment of the present invention, step S200 (i.e., performing lane line detection on the front-end image data and the back-end image data respectively, and obtaining the front-end lane line detection result and the back-end lane line detection result) may include the following steps S210-S230:

[0065] S210. Perform semantic segmentation on the front-end image data and the back-end image data respectively, and identify the lane line pixel regions corresponding to the front-end lane lines and the back-end lane lines.

[0066] In this embodiment, the computing device can use a pre-trained semantic segmentation network model to process the front-end image data. and backend image data Semantic segmentation is performed separately to identify the lane line pixel regions corresponding to the front and rear lane lines. Specifically, the semantic segmentation network model can output the front image data... and backend image data Probability plots of the same size Used to represent pixels The probability of belonging to class K, where class K includes at least lane line class and non-lane line class, can be represented as follows:

[0067]

[0068] Then, a threshold T can be set for the probability of lane line categories to filter those that meet the criteria. The semantic segmentation network identifies the pixels, thus forming lane line pixel regions corresponding to the front and rear lane lines. Preferably, the semantic segmentation network can also identify different types of lane lines, such as solid lines, dashed lines, single yellow lines, or boundary lines, thereby obtaining lane line category information during the detection phase.

[0069] S220. Perform post-processing on the lane line pixel areas corresponding to the front lane line and the rear lane line to obtain the front lane line instance structure and the rear lane line instance structure.

[0070] In this embodiment, after obtaining the lane line pixel regions corresponding to the front and rear lane lines, the computing device can further post-process the lane line pixel regions corresponding to the front and rear lane lines to obtain lane line instance structures with continuous geometric meaning. Specifically, the computing device can divide spatially continuous lane line pixels into different candidate regions through connected component analysis, and process the set of pixels within each candidate region {( Curve fitting can be performed. For example, a quadratic polynomial model can be used to parametrically represent lane lines.

[0071]

[0072] in, The solution can be obtained using the least squares method, which minimizes the error function, thus yielding a continuous and smooth geometric representation of the lane lines. Through this process, the pixel regions corresponding to the original discrete front and rear lane lines can be transformed into structured front and rear lane line instance structures, such as pixel sets or parametric curves.

[0073] S230. Generate front-end lane line detection results based on the front-end lane line instance structure, and generate back-end lane line detection results based on the back-end lane line instance structure.

[0074] In this embodiment, the front-end lane line detection result includes the pixel coordinates and confidence information corresponding to the front-end lane line, and the back-end lane line detection result includes the pixel coordinates and confidence information corresponding to the back-end lane line. The computing device can generate the front-end lane line detection result based on the front-end lane line instance structure, and generate the back-end lane line detection result based on the back-end lane line instance structure. Specifically, when the front-end and back-end lane line instance structures are represented by pixel sets, the pixel sets can be directly determined as the pixel coordinates of the lane line in the image coordinate system; when the front-end and back-end lane line instance structures are represented by parameterized curves, the curves can be discretely sampled within a preset vertical sampling interval, and the corresponding pixel coordinate points can be calculated based on the parameterized curves to obtain the pixel coordinate set of the lane line in the image coordinate system.

[0075] In other embodiments of the present invention, the front-end lane line detection results and the back-end lane line detection results may also include the lane line category information and confidence information obtained in step S210, respectively. The confidence information represents the reliability of the front-end and back-end lane line detection results. In subsequent fusion steps, the category information can be used for lane line matching constraints to ensure that only lane lines of the same type are fused, avoiding mismatches between different types of lane lines. The confidence information can be used to determine the weight allocation or priority strategy during the fusion process, such as prioritizing lane line detection results with higher confidence in overlapping areas, or assigning different weights to lane lines from different sources based on their confidence levels, thereby improving the stability and accuracy of the fusion results.

[0076] Specifically, confidence information can be obtained through probability graph-based methods. Statistical measures can also be used to evaluate the quality of parametric curve fitting. In probability-based graphs... When performing statistical measurements, since the semantic segmentation network outputs the probability that each pixel belongs to the lane line category in step S210... After forming the lane line pixel regions corresponding to the front and rear lane lines, the probability of all pixels within the lane line pixel region can be determined. Aggregate statistics are performed, such as calculating the average, minimum, or specific quantiles, as the overall confidence score for the front-end and back-end lane line instance structures, i.e., confidence information. When evaluating the quality of parametric curve fitting, the fitting residuals can be calculated in step S220 when the front-end and back-end lane line instance structures are represented using parametric curves. This residuals reflect the degree of fit between the fitted curve and the original pixels, thereby determining the confidence information. Therefore, by introducing category information and confidence information, the probability of misfusion can be effectively reduced and the reliability of global lane line construction can be improved during multi-view lane line fusion.

[0077] (iii) Project the front and rear lane lines into the bird's-eye view space.

[0078] In other words, after acquiring the front lane line detection results and the rear lane line detection results, the computing device can project the front lane lines and the rear lane lines onto the bird's-eye view space based on these results (i.e., step S300). For example... Figure 4 As shown, in one embodiment of the present invention, projecting the front lane lines and rear lane lines onto the bird's-eye view space includes steps S310-S320:

[0079] S310. Based on the front lane detection results and the rear lane detection results, and using the calibration parameters of the front and rear cameras, construct the front lane 3D points and the rear lane 3D points in the vehicle coordinate system of the unmanned vehicle.

[0080] In this embodiment, after obtaining the front lane detection results and the rear lane detection results, the computing device can construct the 3D points of the front lane and the rear lane in the vehicle coordinate system of the unmanned vehicle based on the correspondence between the image coordinate system and the vehicle coordinate system established by the calibration parameters. Through the above processing, the spatial representation of the front lane and the rear lane in a unified vehicle coordinate system is realized, providing a foundation for subsequent spatial transformation. Specifically, as shown... Figure 5 As shown, step S310 may include steps S311-S312:

[0081] S311. Based on the intrinsic parameter information of the front-end camera and the rear-end camera, convert the pixel coordinates corresponding to the front-end lane line into a three-dimensional point of the front-end lane in the camera coordinate system of the front-end camera, and convert the pixel coordinates corresponding to the rear-end lane line into a three-dimensional point of the rear-end lane in the camera coordinate system of the rear-end camera.

[0082] In this embodiment, the calibration parameters include intrinsic and extrinsic parameters. The intrinsic parameters describe the mapping relationship between the image coordinate system and the camera coordinate system. The computing device obtains the pixel coordinates of the front and rear lane lines in the image coordinate system. After that, the pixel coordinates can be transformed to the camera coordinate system based on the intrinsic parameter information of the corresponding camera. Specifically, the relationship between pixel coordinates and camera coordinates can be established through an intrinsic parameter matrix:

[0083]

[0084] in, This indicates the camera's intrinsic parameter information. , () represents the coordinates of a point in the camera coordinate system. This is the scaling factor. Given the ground constraints or the lane lines being located on the ground plane, the scaling factor can be determined by combining the constraint conditions. The pixel coordinates corresponding to the front lane lines are converted into 3D points of the front lane in the coordinate system of the front camera, and the pixel coordinates corresponding to the rear lane lines are converted into 3D points of the rear lane in the coordinate system of the rear camera.

[0085] S312. Based on the extrinsic information of the front-end camera and the rear-end camera, convert the three-dimensional points of the front lane and the three-dimensional points of the rear lane to the vehicle coordinate system of the unmanned vehicle.

[0086] In this embodiment, after obtaining the 3D points of the front and rear lanes in the camera coordinate system, the computing device can further transform the 3D points of the front and rear lanes to the vehicle coordinate system of the unmanned vehicle based on the extrinsic parameter information of the corresponding cameras. The extrinsic parameter information describes the spatial transformation relationship between the camera coordinate system and the vehicle coordinate system, typically including rotation and translation relationships. The 3D points in the camera coordinate system (…) , ) and three-dimensional points in the vehicle coordinate system ( , The relationship between them can be expressed as:

[0087]

[0088] in, This indicates the rotation relationship from the camera coordinate system to the vehicle coordinate system. This indicates a translation relationship. Through the above transformation, the 3D points of the front and rear lanes can be uniformly converted into the vehicle coordinate system of the unmanned vehicle. This allows for the expression of lane line spatial information from different cameras under a unified vehicle reference system, providing a consistent spatial reference for subsequent projection processing in the bird's-eye view space.

[0089] S320: Based on the real-time pose information of the unmanned vehicle, the three-dimensional points of the front lane and the rear lane are transformed into the bird's-eye view spatial coordinate system to construct the spatial projection of the front lane line and the rear lane line in the bird's-eye view space.

[0090] In this embodiment, real-time pose information may include the position of the unmanned vehicle in the global coordinate system ( , ) and posture ( ),in These represent the roll angle, pitch angle, and yaw angle, respectively. The transformation from the vehicle coordinate system to the bird's-eye view coordinate system can be represented by a homogeneous transformation matrix:

[0091]

[0092] in,( , () is a three-dimensional point in the vehicle coordinate system. , () represents the corresponding spatial coordinates of the bird's-eye view. This is the homogeneous transformation matrix from the vehicle coordinate system to the bird's-eye view coordinate system. It can be further decomposed into rotation matrices ( Translation vector .

[0093] (iv) Constructing global lane lines.

[0094] In other words, in the bird's-eye view space, the front lane lines and the rear lane lines are merged and spliced ​​to construct the global lane lines (i.e., step S400).

[0095] like Figure 6 As shown, in one embodiment of the present invention, step S400 may include steps S410-S430:

[0096] S410. Based on the spatial positional relationship between the three-dimensional points of the front lane and the three-dimensional points of the rear lane in the bird's-eye view spatial coordinate system, determine the overlapping area of ​​the front lane line and the rear lane line.

[0097] In this embodiment, after the computing device projects the front and rear lane lines onto a unified bird's-eye view space using the vehicle's real-time pose information, it can further use a distance threshold... Determine which points overlap in the spatial coordinate system of the bird's-eye view. Specifically, for the set of 3D points of the front lane... and rear lane 3D points If the following formula is true, then the corresponding 3D point is determined to be a point in the overlapping region. The formula is:

[0098]

[0099] Therefore, all overlapping points of the front and rear lane lines can be obtained, and the area where the overlapping points are located can be set as the overlapping area.

[0100] S420. Based on a preset weighted fusion strategy, the overlapping areas of the front lane lines and the rear lane lines are fused using confidence information to obtain fused lane segments.

[0101] In the fusion processing of overlapping areas, for the three-dimensional points of the front lane located in the bird's-eye view spatial coordinate system and rear lane 3D points When the spatial distance between the two lane lines meets the aforementioned preset overlap determination condition, they can be considered to correspond to the observation results of the same lane line under different fields of view. In one embodiment of the present invention, a weighted average strategy can be used to fuse the overlapping areas of the front lane line and the rear lane line. Specifically, the confidence information contained in the front lane line detection results and the rear lane line detection results can be used. and Construct weight coefficients respectively and Weighting coefficient and The calculation formula is:

[0102]

[0103] At this point, the merged lane line points can be represented as:

[0104]

[0105] In other embodiments of the present invention, the distance between the lane line and the autonomous vehicle can also be considered simultaneously. For example, the distances from the lane line points to the autonomous vehicle are respectively... and Then a comprehensive weight can be constructed:

[0106]

[0107] This allows observations with high confidence and closer proximity to have higher weight in the fusion process, improving fusion accuracy and suppressing the influence of long-distance noise. Furthermore, before performing weighted fusion, a consistency check can be performed based on lane line category information. Weighted fusion is only performed when the category information of lane lines on both sides is consistent or meets a preset matching relationship; otherwise, fusion is abandoned to avoid erroneous fusion between different types of lane lines.

[0108] In another embodiment of the invention, the computing device may also employ a Kalman filtering strategy to fuse lane lines in overlapping areas. Specifically, the position parameters of the lane lines in the bird's-eye view space can be used as state vectors. , the three-dimensional points of the front lane and rear lane 3D points As observations respectively and And construct the observation noise covariance matrix based on their respective confidence information, so that the observation noise covariance is inversely proportional to the confidence level:

[0109]

[0110] in, A baseline noise variance is preset. Through the prediction and update process of Kalman filtering, the observation information from both sides is recursively fused to obtain the state estimate as the fused lane line point. Since observations with higher confidence correspond to smaller observation noise covariance, they occupy a higher weight in the filtering update process, thus achieving a fusion effect based on confidence-adaptive adjustment.

[0111] In another embodiment of the invention, the computing device may also employ a high-confidence priority strategy for fusion. Specifically, when and When the difference exceeds a preset threshold, the lane line points on the side with higher confidence are directly selected as the fusion result, i.e.:

[0112]

[0113] in, By setting a preset threshold, this strategy can prevent low-confidence detection results from interfering with the fusion results. Simultaneously, it can also incorporate category information for constraint, prioritizing selection only when categories match.

[0114] S430, based on spatial continuity, splices the front and rear lane lines of the merged lane segments with the non-overlapping areas to construct a global lane line.

[0115] In this embodiment, the computing device can, based on the principle of spatial continuity, splice the fused lane segments with the front and rear lane lines of the non-overlapping areas to construct a global lane line. Specifically, the set of points for the fused lane segments can be set as { The set of lane line points in the non-overlapping region is { The set of lane line points in the non-overlapping region is { Since all point sets are located in a unified bird's-eye view spatial coordinate system, the computing device can determine the points that satisfy distance constraints based on spatial adjacency and tangential direction consistency. Connect line segments whose directional difference is less than a preset angle threshold to obtain an initial set of splicing points. Initial splicing point set It can be represented as:

[0116]

[0117] Based on this, curve fitting and smoothing can be performed on the spliced ​​point set, for example, using quadratic polynomials or spline functions. By performing continuous fitting, a global lane line that satisfies curvature continuity is obtained. This ensures the smoothness and consistency of the lane lines in terms of geometry.

[0118] In one embodiment of the present invention, the construction method further includes: when there is a gap between the front lane line and the rear lane line in the non-overlapping area and the fused lane line segment, interpolating to fill the gap.

[0119] Specifically, during the stitching process, if the computing device detects a spatial gap or interruption smaller than a preset upper threshold between adjacent lane segments in the bird's-eye view spatial coordinate system, it can be determined as a lane line discontinuity. In this case, the historical driving trajectory of the autonomous vehicle and the geometric features of the lane lines can be combined for completion processing. The historical driving trajectory of the autonomous vehicle can be represented as a time series curve. Lane lines can be represented as parametric curves. When two adjacent lane lines are close in position but discontinuous at their endpoints, an interpolation function can be constructed based on curvature continuity constraints to ensure that the completed curve satisfies both positional and curvature continuity conditions at the connection point, for example, satisfying:

[0120]

[0121] in For curvature. The interpolation process can also refer to historical driving trajectories. Within the corresponding area, the completed curve conforms to both the geometry of the lane lines and the actual driving path of the autonomous vehicle. This method effectively eliminates breakpoints caused by missing local detections, improving the continuity and stability of the overall lane lines.

[0122] In one embodiment of the present invention, the construction method further includes: when the unmanned vehicle performs a forward or backward reversal operation, using global lane lines as prior information to guide or constrain the unmanned vehicle. Specifically, when the unmanned vehicle performs a reversal operation, its driving direction changes, the original front-end camera becomes a rear-end camera, and the original rear-end camera becomes a front-end camera. To avoid interruption of lane line detection due to perspective switching, before the reversal operation occurs, the global lane lines constructed at the current moment can be stored as historical memory information. This global lane line includes the geometric shape, category information, and confidence information of the lane lines. After the reversal is completed, the new front-end camera begins to dominate lane line detection. At this time, the historical memory information can be used as prior information to participate in the lane line detection process of the current frame. For example, the region of interest for lane line search in the current frame can be limited based on the predicted position of the global lane lines in the historical memory information under the current vehicle pose, thereby narrowing the detection range and improving detection stability; or the global lane lines in the historical memory information can be used as matching targets to track and correct the current detection results to avoid detection drift caused by perspective changes during the reversal. As the unmanned vehicle continues to move in the new driving direction, the lane line information detected in the current frame will be merged and updated with the historical memory information, thereby realizing the dynamic correction and improvement of the global lane line, so that the perception process maintains continuity and consistency before and after the change, and improves the operational stability and safety of the unmanned vehicle in the complex working conditions of the port area.

[0123] In summary, the global lane line construction method for unmanned vehicles based on dual-view field fusion of the present invention acquires front-end image data collected by a front-end camera and back-end image data collected by a back-end camera, respectively, and performs lane line detection on the front-end and back-end image data to obtain the front-end lane line detection results and back-end lane line detection results. This effectively expands the physical range of lane line perception, overcomes the limitation of the limited field of view of a single camera, and provides a data foundation for constructing longer and more complete global lane lines. Simultaneously, by fusing and stitching the front-end and back-end lane lines in the bird's-eye view space to construct the global lane line, it is possible to integrate and complement lane lines from different perspectives in the bird's-eye view space, generating a more continuous and complete global lane line, thereby improving the driving reliability and operational accuracy of unmanned vehicles in tasks such as direction changing and high-precision alignment.

[0124] like Figure 7 As shown, a second aspect of the present invention also provides a global lane line construction system for unmanned vehicles based on dual-end field-of-view fusion. The unmanned vehicle has a front-end camera and a rear-end camera respectively installed at its front and rear ends. The construction system 1000 includes:

[0125] The image data acquisition module 1100 is used to acquire front-end image data captured by the front-end camera and back-end image data captured by the back-end camera, respectively.

[0126] The lane line detection module 1200 is used to perform lane line detection on front-end image data and back-end image data, and to obtain front-end lane line detection results and back-end lane line detection results.

[0127] The projection module 1300 is used to project the front lane lines and the rear lane lines onto the bird's-eye view space based on the front lane line detection results and the rear lane line detection results.

[0128] Module 1400 is used to merge and stitch together the front and rear lane lines in the bird's-eye view space to construct the global lane lines.

[0129] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "an" or "a" and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "connected" or "linked" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up," "down," "left," "right," etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0130] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for constructing global lane lines for unmanned vehicles based on dual-end field-of-view fusion, characterized in that, The unmanned vehicle is equipped with a front-end camera and a rear-end camera, respectively, and the construction method includes: The front-end image data captured by the front-end camera and the back-end image data captured by the back-end camera are acquired respectively. Lane line detection is performed on the front-end image data and the back-end image data respectively to obtain the front-end lane line detection result and the back-end lane line detection result; Based on the front lane line detection results and the rear lane line detection results, the front lane lines and the rear lane lines are projected onto the bird's-eye view space; In the bird's-eye view space, the front lane lines and the rear lane lines are merged and spliced ​​to construct a global lane line.

2. The construction method according to claim 1, characterized in that, The lane line detection is performed after preprocessing the front-end image data and the back-end image data respectively. The preprocessing includes at least one of distortion correction, image enhancement and noise reduction.

3. The construction method according to claim 1, characterized in that, Lane line detection is performed on the front-end image data and the back-end image data respectively to obtain the front-end lane line detection results and the back-end lane line detection results, including: Semantic segmentation is performed on the front-end image data and the back-end image data respectively to identify the lane line pixel regions corresponding to the front-end lane lines and the back-end lane lines; Post-processing is performed on the lane line pixel regions corresponding to the front lane line and the rear lane line to obtain the front lane line instance structure and the rear lane line instance structure. The front-end lane line detection result is generated based on the front-end lane line instance structure, and the back-end lane line detection result is generated based on the back-end lane line instance structure.

4. The construction method according to claim 3, characterized in that, Based on the front lane line detection results and the rear lane line detection results, the front lane lines and the rear lane lines are projected onto the bird's-eye view space, including: Based on the front lane detection results and the rear lane detection results, and using the calibration parameters of the front camera and the rear camera, three-dimensional points of the front lane and the rear lane are constructed in the vehicle coordinate system of the unmanned vehicle. Based on the real-time pose information of the unmanned vehicle, the three-dimensional points of the front lane and the rear lane are transformed into the bird's-eye view spatial coordinate system to construct the spatial projection of the front lane line and the rear lane line in the bird's-eye view space.

5. The construction method according to claim 4, characterized in that, The front lane line detection result includes the pixel coordinates and confidence information corresponding to the front lane line, and the rear lane line detection result includes the pixel coordinates and confidence information corresponding to the rear lane line.

6. The construction method according to claim 5, characterized in that, The calibration parameters include intrinsic and extrinsic information; Based on the front lane detection results and the rear lane detection results, and using the calibration parameters of the front and rear cameras, three-dimensional points of the front lane and the rear lane are constructed in the vehicle coordinate system of the unmanned vehicle, including: Based on the intrinsic parameter information of the front-end camera and the rear-end camera, the pixel coordinates corresponding to the front-end lane line are converted into the front-end lane three-dimensional points in the front-end camera coordinate system, and the pixel coordinates corresponding to the rear-end lane line are converted into the rear-end lane three-dimensional points in the rear-end camera coordinate system. Based on the extrinsic information of the front-end camera and the rear-end camera, the three-dimensional points of the front-end lane and the three-dimensional points of the rear-end lane are transformed into the vehicle coordinate system of the unmanned vehicle.

7. The construction method according to claim 5, characterized in that, In the bird's-eye view space, the front lane lines and the rear lane lines are merged and stitched together to construct a global lane line, including: Based on the spatial positional relationship between the three-dimensional points of the front lane and the three-dimensional points of the rear lane in the bird's-eye view spatial coordinate system, the overlapping area of ​​the front lane line and the rear lane line is determined. Based on a preset weighted fusion strategy, the overlapping areas of the front lane line and the rear lane line are fused using the confidence information to obtain a fused lane line segment. Based on spatial continuity, the fused lane segment is spliced ​​with the front lane line and the rear lane line in the non-overlapping area to construct the global lane line.

8. The construction method according to claim 7, characterized in that, Also includes: When there are gaps between the front lane line and the rear lane line and the fused lane line segment in the non-overlapping area, the gaps are filled by interpolation.

9. The construction method according to claim 1, characterized in that, Also includes: When the unmanned vehicle performs forward or backward direction changes, the global lane lines are used as prior information to guide or constrain the unmanned vehicle.

10. A global lane line construction system for unmanned vehicles based on dual-end field-of-view fusion, characterized in that, The unmanned vehicle is equipped with a front-end camera and a rear-end camera, respectively, and the construction system includes: The image data acquisition module is used to acquire the front-end image data captured by the front-end camera and the back-end image data captured by the back-end camera, respectively. The lane line detection module is used to perform lane line detection on the front-end image data and the back-end image data, and obtain the front-end lane line detection result and the back-end lane line detection result. The projection module is used to project the front lane line and the rear lane line onto the bird's-eye view space based on the front lane line detection results and the rear lane line detection results. A construction module is used to merge and stitch the front lane lines and the rear lane lines in the bird's-eye view space to construct a global lane line.

Images