Lane line generation method, electrical equipment, storage medium and computer program product

Abstract

The application discloses a lane line generation method. The method comprises the following steps: acquiring first historical traffic flow information, wherein the first historical traffic flow information comprises collected historical data of multiple frames of traffic flow instances of a target road; and fusing the first historical traffic flow information and current visual perception information to generate a target lane line. In this way, the lane line is generated by fusing the historical traffic flow information and the current visual perception information, the driving track and distribution law contained in the historical traffic flow information can provide reliable auxiliary basis for current lane line perception, the accuracy of lane line generation is improved, reliable lane line data is provided for a vehicle-mounted system, and therefore, the path planning accuracy and driving control safety of the vehicle in a complex traffic environment are improved to a certain extent, and the driving risk of the vehicle is effectively reduced.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a lane line generation method, electrical equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Lane line data is crucial perception data supporting vehicle path planning and driving control in autonomous driving. Its quality significantly impacts the effectiveness of vehicle system functions. Related technologies typically employ visual perception to collect lane line data. However, in complex driving environments, interference such as large-scale obstructions in congested areas makes it difficult to effectively perceive lane lines visually, posing safety risks to the vehicle. Summary of the Invention

[0003] This application provides a lane line generation method, an electrical device, a computer-readable storage medium, and a computer program product.

[0004] This application provides a lane line generation method, the method comprising: Acquire first historical traffic flow information, wherein the first historical traffic flow information includes multi-frame historical traffic flow instance data of the target road; The first historical traffic flow information and the current visual perception information are fused together to generate the target lane line.

[0005] In this way, by integrating historical traffic flow information and current visual perception information to generate lane lines, the driving trajectory and distribution patterns contained in the historical traffic flow information can provide reliable auxiliary basis for current lane line perception, improve the accuracy of lane line generation, provide reliable lane line data for the vehicle system, and thus improve the accuracy of vehicle path planning and driving control safety in complex traffic environments to a certain extent, effectively reducing vehicle driving risks.

[0006] In some implementations, the step of fusing the first historical traffic flow information and the current visual perception information to generate the target lane line includes: The coordinate system of the target traffic flow instance in the current visual perception information is determined as the target coordinate system; Based on the pose of the target traffic flow instance and the pose of each traffic flow instance in the first historical traffic flow information, the first historical traffic flow information is transformed to the target coordinate system; The second historical traffic flow information and the current visual perception information are fused to generate the target lane line, wherein the second historical traffic flow information is the traffic flow information transformed from the first historical traffic flow information to the target coordinate system.

[0007] In this way, by using a unified coordinate system, the consistency between historical traffic flow information and current visual perception information in space is ensured, eliminating the fusion interference caused by coordinate differences. This allows the features after the fusion of the second historical traffic flow information and the current visual perception information to accurately reflect the driving patterns of traffic flow and the spatial structure of the road, providing reliable basic data for the generation of target lane lines, improving the accuracy and reliability of target lane line generation, and ensuring the safety of vehicle driving.

[0008] In some implementations, the step of fusing the second historical traffic flow information and the current visual perception information to generate the target lane line includes: The second historical traffic flow information is subjected to a first encoding process to obtain a rasterized feature vector of historical traffic flow; The second historical traffic flow information is subjected to a second encoding process to obtain the spatiotemporal feature vector of historical traffic flow; The target lane line is generated by fusing the historical traffic flow rasterized feature vector, the historical traffic flow spatiotemporal feature vector, and the current visual perception information.

[0009] Thus, by using two different encoding methods, features of the second historical traffic flow information are extracted from two dimensions: spatial distribution and spatiotemporal evolution. This yields two types of feature vectors that comprehensively reflect the key information of the traffic flow. These vectors are then fused with the current visual perception information, achieving complementarity and integration of multi-dimensional features. This allows the generated target lane lines to comprehensively consider the spatial distribution and spatiotemporal variation patterns of the traffic flow, improving the accuracy of lane line detection. This enables vehicles to accurately infer lane line positions based on traffic flow features in complex scenarios, providing reliable lane line data support for the vehicle system and ensuring vehicle driving safety.

[0010] In some embodiments, the second encoding process of the second historical traffic flow information to obtain the historical traffic flow spatiotemporal feature vector includes: A traffic flow matrix is ​​constructed based on the number of frames of historical data of traffic flow instances in the historical traffic flow information, the number of dimensions of historical data of traffic flow instances, and a preset threshold for the number of traffic flow instances, wherein the dimensions represent the feature information category of the traffic flow instance; The traffic flow matrix is ​​input into a pre-established spatiotemporal information feature encoder to obtain the historical traffic flow spatiotemporal feature vector.

[0011] Thus, by constructing a traffic flow matrix, the scattered and complex historical traffic flow data is transformed into a structured matrix form, laying the foundation for feature extraction by the encoder. Furthermore, the spatiotemporal information feature encoder can effectively capture the temporal variation patterns and spatial correlation features of traffic flow, improving the accuracy and effectiveness of spatiotemporal feature extraction. This enables the generated target lane lines to accurately reflect the actual road conditions, improving the accuracy of lane line detection and providing reliable lane line data support for the vehicle system.

[0012] In some implementations, the step of fusing the historical traffic flow rasterized feature vector, the historical traffic flow spatiotemporal feature vector, and the current visual perception information to generate the target lane line includes: The current visual perception information is subjected to a first encoding process to obtain the current visual perception rasterized feature vector; The target lane line is generated by fusing the current visual perception rasterized feature vector, the historical traffic flow rasterized feature vector, and the historical traffic flow spatiotemporal feature vector.

[0013] Thus, by performing the first encoding process on the current visual perception information, a current visual perception rasterized feature vector with the same format as the historical traffic flow rasterized feature vector is generated, which effectively solves the alignment problem of traffic flow instance features, lays the foundation for the deep fusion of the three types of features, and realizes the efficient interaction of real-time spatial features, historical spatial features and spatiotemporal evolution features through the cross-attention mechanism. The generated fused features can comprehensively reflect the current road conditions, historical driving patterns and dynamic changes in traffic flow, improve the accuracy and reliability of target lane line generation, enable lane line detection to adapt to complex and ever-changing traffic environments, and ensure the safety and stability of vehicle driving.

[0014] In some implementations, the step of fusing the current visual perception rasterized feature vector, the historical traffic flow rasterized feature vector, and the historical traffic flow spatiotemporal feature vector to generate the target lane line includes: The historical traffic flow rasterized feature vector and the current visual perception rasterized feature vector are fused to generate a fused traffic flow rasterized feature vector; The target lane line is generated by fusing the fused traffic flow rasterized feature vector and the historical traffic flow spatiotemporal feature vector.

[0015] Thus, the step-by-step fusion approach effectively avoids information interference and redundancy issues caused by direct fusion of multi-dimensional features. Furthermore, the target lane lines generated by step-by-step fusion can more accurately match the actual road conditions. This allows the vehicle system to accurately infer lane line positions based on the integrated multi-dimensional features in scenarios such as congestion, occlusion, and complex and variable traffic flow, effectively improving the accuracy of lane line detection and providing reliable lane line data support for the vehicle system's path planning and driving control.

[0016] In some implementations, generating the target lane line based on the fused traffic flow rasterized feature vector and the historical traffic flow spatiotemporal feature vector includes: Based on the fused traffic flow rasterized feature vector, extract the fused lane line feature vector; The fused lane line feature vector and the historical traffic flow spatiotemporal feature vector are fused to generate the target lane line feature vector. The target lane line is generated based on the target lane line feature vector.

[0017] Thus, by extracting and fusing lane line feature vectors, redundant information in the fusing traffic flow rasterized feature vectors is effectively filtered out, spatial features directly related to lane line generation are strengthened, and the relevance and effectiveness of the features are improved. Furthermore, the subsequent fusion with historical traffic flow spatiotemporal feature vectors achieves a deep integration of key spatial features and spatiotemporal evolution patterns. The generated target lane line feature vectors can accurately characterize the attributes of lane lines, effectively improving the accuracy of lane line detection and providing solid lane line data support for path planning and driving control of the vehicle system.

[0018] This application also provides an electrical device, including: a processor and a memory for storing processor-executable instructions, wherein the processor is configured to execute the instructions to implement the above-described method.

[0019] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0020] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0021] The electrical equipment, computer-readable storage medium, and computer program product provided in this application, when implementing the above method, first acquire first historical traffic flow information, wherein the first historical traffic flow information includes multi-frame historical traffic flow instance data of the target road; finally, the first historical traffic flow information and the current visual perception information are fused to generate the target lane line. In this way, by fusing historical traffic flow information and current visual perception information to generate lane lines, the driving trajectory and distribution patterns contained in the historical traffic flow information can provide reliable auxiliary basis for current lane line perception, improving the accuracy of lane line generation, providing reliable lane line data for the vehicle system, thereby improving the accuracy of vehicle path planning and driving control safety in complex traffic environments to a certain extent, and effectively reducing vehicle driving risks.

[0022] Additional aspects and advantages of embodiments of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of embodiments of this application. Attached Figure Description

[0023] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein: Figure 1 This is one of the flowcharts illustrating a lane line generation method according to certain embodiments of this application; Figure 2 This is a second schematic flowchart of a lane line generation method according to certain embodiments of this application; Figure 3 This is the third flowchart of a lane line generation method according to certain embodiments of this application; Figure 4 This is a schematic diagram of the traffic flow processing flow in the network part of certain embodiments of this application; Figure 5 This is the fourth flowchart of a lane line generation method according to certain embodiments of this application; Figure 6 This is the fifth flowchart of a lane line generation method according to certain embodiments of this application; Figure 7 This is a schematic flowchart of a lane line generation method according to certain embodiments of this application; Figure 8 This is the seventh flowchart of a lane line generation method according to certain embodiments of this application. Detailed Implementation

[0024] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0025] In the fields of autonomous driving and map reconstruction, road structure perception is one of the core technologies supporting autonomous driving path planning and high-precision map reconstruction, while lane lines and lane center lines are key elements in road structure detection. Among related technologies, most lane line perception methods rely on visual features as their information source, specifically including single-view RGB images, multi-view RGB images, and point cloud information. These methods depend on the direct extraction and analysis of visual features. However, in practical applications, when there are large areas of occlusion or numerous interfering factors in the visible environment, the effectiveness of visual features decreases, thus increasing the difficulty of lane line detection.

[0026] To address these issues, some perception solutions incorporate maps as prior knowledge into the network training process. By leveraging existing road structure information in the map, they compensate for information gaps in scenarios where visual features fail, thereby improving the stability of lane line detection. However, the effectiveness of lane line detection decreases when encountering issues such as map interface glitches, changes in map information, or positioning errors.

[0027] Besides the methods mentioned above, lane line perception schemes based on pure vision can be divided into two categories: single-frame architecture and temporal architecture. However, in complex scenarios such as traffic congestion and large-area occlusion, the single-frame image of the single-frame architecture cannot provide sufficient lane line features, increasing the difficulty and uncertainty of lane line detection.

[0028] Temporal architecture, on the other hand, supplements the visual features of a single frame with the spatiotemporal features of consecutive frames, improving detection capabilities in complex scenes by mining the correlation information between different frames. However, the effective range of this approach is limited by the coverage duration of temporal feature extraction. When the duration of traffic congestion or occlusion exceeds the coverage range, temporal architecture still struggles to solve the lane line detection problem in long-term congestion scenarios.

[0029] In summary, lane perception solutions still face many technical bottlenecks when dealing with complex and dynamically changing traffic environments, making it difficult to reliably meet the accuracy requirements of autonomous driving and map reconstruction.

[0030] Based on the above issues, please refer to Figure 1 This application provides a lane line generation method, the method comprising: 01: Obtain the first historical traffic flow information, which includes the collected historical data of multiple frames of traffic flow instances on the target road; 02: The first historical traffic flow information and the current visual perception information are fused and processed to generate the target lane line.

[0031] This application provides a lane line generation device. The lane line generation method of this application can be implemented by the lane line generation device of this application. Specifically, the lane line generation device includes an acquisition module and a fusion module. The acquisition module is used to acquire first historical traffic flow information, wherein the first historical traffic flow information includes multi-frame historical traffic flow instance data of the target road. The fusion module is used to fuse the first historical traffic flow information and the current visual perception information to generate the target lane line.

[0032] This application also provides a server, which includes a memory and a processor. The lane line generation method of this application can be implemented by the server of this application. Specifically, the memory stores a computer program, and the processor is used to acquire first historical traffic flow information, wherein the first historical traffic flow information includes multi-frame historical traffic flow instance data of the target road. The processor is also used to perform fusion processing on the first historical traffic flow information and current visual perception information to generate target lane lines.

[0033] Specifically, the target road is a specific road segment that the autonomous vehicle is currently driving or is about to drive on, and it is a specific area for traffic flow information collection and lane line generation.

[0034] Traffic flow instances are individual traffic participants involved in traffic operations on a target road, including various moving entities such as vehicles and pedestrians. Each instance includes its own characteristic information, such as location, speed, and category.

[0035] The first historical traffic flow information is a collection of multiple frames of continuous traffic flow instance data accumulated over a period of time, which can reflect the change process of traffic flow instances over a period of time.

[0036] The current visual perception information is road environment data collected in real time by the vehicle through onboard vision sensors. It is usually monocular or multi-view RGB images and point clouds, which can reflect the visual characteristics of the current road scene.

[0037] Fusion processing is a process of extracting, integrating, and interacting features from first-historical traffic flow information and current visual perception information through a specific algorithm model, mining effective correlations related to lane lines in the two types of information, and removing redundant and interfering information.

[0038] The target lane lines are lane line data obtained through fusion processing, which can accurately reflect the actual lane line position, direction and geometry of the target road, and are used to support the path planning and driving control of autonomous vehicles.

[0039] When acquiring first-hand historical traffic flow information, one can flexibly choose offline processing or vehicle-side subscription tracking nodes to obtain multi-frame continuous traffic flow instance historical data of the target road. This ensures that the historical data can cover the traffic flow change process within a certain time range, providing a foundation for subsequent analysis of the spatiotemporal patterns of traffic flow.

[0040] At the same time, real-time road environment data is collected to obtain the instantaneous status data of each traffic flow instance on the target road at the current moment, including feature information such as location, speed, and category, to ensure that the current visual perception information can reflect the real-time distribution of each traffic flow instance on the target road.

[0041] After obtaining the first historical traffic flow information and the current road environment data, a specific algorithm model is used to extract the effective features from the two types of information. For example, key features related to lane lines, such as the driving trajectory, speed changes, and spatial distribution of traffic flow instances, are extracted.

[0042] Subsequently, by integrating and correlating these features, and utilizing the long-term driving patterns contained in the first historical traffic flow information and the real-time road conditions reflected by the current road environment data, a comprehensive feature that can characterize the distribution of road lane lines is constructed, and finally, the target lane line is generated based on this comprehensive feature.

[0043] In this way, by integrating historical traffic flow information and current visual perception information to generate lane lines, the driving trajectory and distribution patterns contained in the historical traffic flow information can provide reliable auxiliary basis for current lane line perception, improve the accuracy of lane line generation, provide reliable lane line data for the vehicle system, and thus improve the accuracy of vehicle path planning and driving control safety in complex traffic environments to a certain extent, effectively reducing vehicle driving risks.

[0044] Please see Figure 2 In some implementations, step 02 includes: 021: Determine the coordinate system of the target traffic flow instance in the current visual perception information as the target coordinate system; 022: Based on the pose of the target traffic flow instance and the poses of each traffic flow instance in the first historical traffic flow information, transform the first historical traffic flow information to the target coordinate system; 023: The second historical traffic flow information and the current visual perception information are fused to generate the target lane line. The second historical traffic flow information is the traffic flow information of the first historical traffic flow information transformed into the target coordinate system.

[0045] In some implementations, the fusion module is further configured to determine the coordinate system of the target traffic flow instance in the current visual perception information as the target coordinate system. The fusion module is also configured to transform the first historical traffic flow information to the target coordinate system based on the pose of the target traffic flow instance and the poses of each traffic flow instance in the first historical traffic flow information. The fusion module is further configured to perform fusion processing on the second historical traffic flow information and the current visual perception information to generate target lane lines, wherein the second historical traffic flow information is the traffic flow information transformed from the first historical traffic flow information to the target coordinate system.

[0046] In some implementations, the processor is further configured to determine the coordinate system of the target traffic flow instance in the current visual perception information as the target coordinate system. The processor is further configured to transform the first historical traffic flow information to the target coordinate system based on the pose of the target traffic flow instance and the poses of each traffic flow instance in the first historical traffic flow information. The processor is further configured to perform fusion processing on the second historical traffic flow information and the current visual perception information to generate target lane lines, wherein the second historical traffic flow information is traffic flow information transformed from the first historical traffic flow information to the target coordinate system.

[0047] Specifically, a target traffic flow instance is a specific traffic participant with a clear location and stable state selected from the current visual perception information, and is a traffic flow instance that can be used as a reference for the coordinate system.

[0048] The target coordinate system is a spatial reference system established based on the location and orientation of the target traffic flow instance, used to unify the coordinate standards of historical traffic flow information and current visual perception information.

[0049] Pose is a comprehensive information of the location and orientation of a traffic flow instance in space. Location is described by coordinate values, and orientation is described by parameters such as angles, which indicate the direction of the traffic flow instance.

[0050] The second historical traffic flow information is a set of historical traffic flow data that, after coordinate system transformation, is in the same target coordinate system as the current visual perception information.

[0051] After acquiring the first historical traffic flow information and the current visual perception information, the target traffic flow instances are first selected from the current visual perception information. The selection of target traffic flow instances must meet the requirements of accurate positioning and stable state. Usually, traffic participants with smooth driving trajectories and no obvious attitude changes are selected to ensure the stability and reliability of their coordinate system.

[0052] Subsequently, the coordinate system in which the target traffic flow instance is located is determined as the target coordinate system, which will serve as a unified reference standard for all subsequent traffic flow information.

[0053] Next, the pose data of the target traffic flow instance and the pose data of each traffic flow instance in the first historical traffic flow information are obtained. Using the pose of the target traffic flow instance as a reference, the pose of each traffic flow instance in the first historical traffic flow information is calculated and adjusted, and the spatial information of all historical traffic flow instances is uniformly transformed to the target coordinate system.

[0054] In addition, by indexing the traffic flow tracking numbers of consecutive frames, traffic flow instances after coordinate transformation can be stored from front to back in frame order. The speed, category, and location information corresponding to each frame's traffic flow instance can be stored, and the mask information for whether the frame is valid can be saved simultaneously for subsequent traffic flow coding.

[0055] During the coordinate transformation process, it is necessary to accurately calculate the spatial positional relationship and orientation difference between historical traffic flow instances and target traffic flow instances to ensure that the transformed historical traffic flow information can accurately correspond to the spatial scale and orientation reference of the target coordinate system, and finally form the second historical traffic flow information.

[0056] Finally, by integrating the effective features of the second historical traffic flow information and the current visual perception information through a specific fusion algorithm, redundancy and interference caused by coordinate differences are removed, and a target lane line that can accurately reflect the actual situation of the road lane line is generated.

[0057] In this way, by using a unified coordinate system, the consistency between historical traffic flow information and current visual perception information in space is ensured, eliminating the fusion interference caused by coordinate differences. This allows the features after the fusion of the second historical traffic flow information and the current visual perception information to accurately reflect the driving patterns of traffic flow and the spatial structure of the road, providing reliable basic data for the generation of target lane lines, improving the accuracy and reliability of target lane line generation, and ensuring the safety of vehicle driving.

[0058] Please refer to the following: Figure 3 and Figure 4 In some implementations, step 023 includes: 0231: Perform the first encoding process on the second historical traffic flow information to obtain the rasterized feature vector of historical traffic flow; 0232: Perform a second encoding process on the second historical traffic flow information to obtain the spatiotemporal feature vector of historical traffic flow; 0233: The target lane line is generated by fusing the historical traffic flow rasterized feature vector, the historical traffic flow spatiotemporal feature vector, and the current visual perception information.

[0059] In some implementations, the fusion module is further configured to perform a first encoding process on the second historical traffic flow information to obtain a rasterized feature vector of the historical traffic flow. The fusion module is also configured to perform a second encoding process on the second historical traffic flow information to obtain a spatiotemporal feature vector of the historical traffic flow. The fusion module is further configured to fuse the rasterized feature vector of the historical traffic flow, the spatiotemporal feature vector of the historical traffic flow, and the current visual perception information to generate the target lane line.

[0060] In some embodiments, the processor is further configured to perform a first encoding process on the second historical traffic flow information to obtain a rasterized feature vector of the historical traffic flow. The processor is also configured to perform a second encoding process on the second historical traffic flow information to obtain a spatiotemporal feature vector of the historical traffic flow. The processor is further configured to perform a fusion process on the rasterized feature vector of the historical traffic flow, the spatiotemporal feature vector of the historical traffic flow, and the current visual perception information to generate a target lane line.

[0061] Specifically, the first encoding process is an encoding method that extracts the spatial distribution features of the bird's eye view (BEV) of traffic flow information. The core of this method is to map the traffic flow information into the grid structure of the bird's eye view through rasterization.

[0062] The rasterized feature vector of historical traffic flow is a feature vector obtained by processing the second historical traffic flow information through the first encoding process. It can intuitively reflect the distribution pattern and location correlation of the second historical traffic flow in the BEV space.

[0063] The second encoding process is an encoding method that jointly extracts the temporal and spatial dimensions of traffic flow information to capture the dynamic characteristics of traffic flow evolution over time.

[0064] The historical traffic flow spatiotemporal feature vector is a feature vector obtained by processing the second historical traffic flow information through the second encoding process. It can characterize the change pattern of the second historical traffic flow between different time frames and the spatial mutual influence relationship.

[0065] After completing the coordinate system transformation of the first historical traffic flow information to obtain the second historical traffic flow information, from the perspective of BEV, the category attributes of each traffic flow instance in the second historical traffic flow information are first encoded and transformed so that they can be recognized and processed by the pre-built traffic flow raster feature encoder. Then, a raster structure is constructed according to the preset BEV spatial size, and the position, speed and other information of each traffic flow instance are accurately mapped to the corresponding raster to form the initial raster data.

[0066] Subsequently, a traffic flow rasterization feature encoder, including a two-dimensional convolutional layer and a multi-layer perceptron, is used to extract features from the initial rasterized data, enhancing the spatial distribution characteristics of traffic flow, and finally generating historical traffic flow rasterization feature vectors.

[0067] For example, assuming the BEV space has a size of 256×256, there are four attribute categories for the obtained traffic flow instances. In practical applications, the attribute categories of traffic flow instances are based on the types of traffic participants that the vehicle-mounted sensors can reliably identify.

[0068] First, one-hot encoding is used to convert the category attributes of traffic flow instances into one-hot encoding form. Then, a rasterized encoding is constructed in the BEV space according to the set size, and finally an initial encoding result with a size of 256×256×4 is obtained.

[0069] Finally, the initial encoding result is input into a pre-built traffic flow rasterized feature encoder. Through a combination of two-dimensional convolutional layers and a multi-layer perceptron (MLP) in a deep learning network architecture, the corresponding output features are extracted and generated. This encoder can capture the distribution characteristics of traffic flow in the BEV space, providing data support for subsequent fusion and analysis tasks.

[0070] Meanwhile, the number of frames, the number of traffic flow instances, and the feature dimensions of each instance in the second historical traffic flow information are sorted out to construct a structured traffic flow data matrix, which includes the complete feature information of all traffic flow instances in different historical frames.

[0071] For traffic flow instance data missing in some frames, zero padding is used to ensure the integrity of the matrix, and time domain masks and spatial domain masks are constructed simultaneously to mark the valid data area in order to avoid interference from invalid information to lane line detection.

[0072] Finally, the data matrix and mask are input together into a pre-built spatiotemporal information feature encoder. This encoder captures the temporal correlation of traffic flow between different frames through a self-attention mechanism, and combines a multilayer perceptron to perform in-depth feature mining and nonlinear transformation on the features, extracting the spatiotemporal dynamic features of traffic flow and generating historical traffic flow spatiotemporal feature vectors.

[0073] Finally, the rasterized feature vectors of historical traffic flow, the spatiotemporal feature vectors of historical traffic flow, and the current visual perception information are fused. In practical applications, a cross-attention mechanism can be used to achieve deep interaction among the three types of information, allowing spatial distribution features, spatiotemporal evolution features, and the real-time state of the current visual perception information to fully correlate, uncover the intrinsic connections between different features, and then optimize and integrate the fused features through a multilayer perceptron to ultimately generate target lane lines that accurately reflect the actual situation of road lane lines.

[0074] In this way, by using two different encoding methods, features are extracted from the second historical traffic flow information from two dimensions: spatial distribution and spatiotemporal evolution. Two types of feature vectors that can comprehensively reflect the key information of traffic flow are obtained. These vectors are then fused with the current visual perception information, achieving the complementarity and integration of multi-dimensional features. This allows the generated target lane lines to comprehensively consider the spatial distribution and spatiotemporal change patterns of traffic flow, improving the accuracy of lane line detection. This enables vehicles to accurately infer the position of lane lines based on traffic flow features in complex scenarios, providing reliable lane line data support for the vehicle system and ensuring vehicle driving safety.

[0075] Please see Figure 5 In some implementations, step 0232 includes: 02321: Construct a traffic flow matrix based on the number of frames, the number of dimensions, and a preset threshold for the number of traffic flow instances in the historical traffic flow information. Here, the dimension represents the feature information category of the traffic flow instance. 02322: Input the traffic flow matrix into the pre-established spatiotemporal information feature encoder to obtain the historical traffic flow spatiotemporal feature vector.

[0076] In some implementations, the fusion module is further configured to construct a traffic flow matrix based on the number of frames, the number of dimensions, and a preset threshold for the number of traffic flow instances in the historical traffic flow information. The dimensions represent the feature information categories of the traffic flow instances. The fusion module is also configured to input the traffic flow matrix into a pre-established spatiotemporal information feature encoder to obtain historical traffic flow spatiotemporal feature vectors.

[0077] In some implementations, the processor is further configured to construct a traffic flow matrix based on the number of frames of historical traffic flow instance data, the number of dimensions of historical traffic flow instance data, and a preset threshold for the number of traffic flow instances, wherein the dimensions represent the feature information category of the traffic flow instance. The processor is further configured to input the traffic flow matrix into a pre-established spatiotemporal information feature encoder to obtain a historical traffic flow spatiotemporal feature vector.

[0078] Specifically, the number of frames is the total number of continuous data frames included in the second historical traffic flow information, reflecting the coverage of traffic flow data in the time dimension. The number of frames can be determined according to the collection duration and collection frequency of the second historical traffic flow information.

[0079] The dimensions of historical traffic flow instance data represent the categories of feature information included in each traffic flow instance. Each dimension corresponds to a specific feature, such as location, speed, or category. The number of dimensions is equal to the number of feature information categories. The dimensions of historical traffic flow instance data can be determined according to actual needs, including key feature categories of traffic flow instances to ensure the completeness of feature information.

[0080] The preset threshold for the number of traffic flow instances is a maximum number of instances that can be accommodated in the current scenario, pre-set based on the actual application scenario and the processing capacity of the onboard system. For example, the preset threshold for the number of traffic flow instances can be 40.

[0081] A traffic flow matrix is ​​a structured data format that organizes historical data of scattered traffic flow instances according to specific dimensions, so as to facilitate the encoding and processing of traffic flow data by a spatiotemporal information feature encoder.

[0082] The spatiotemporal information feature encoder is a pre-trained encoding model based on a deep learning network architecture that can extract the temporal evolution patterns and spatial correlation features of traffic flow from the traffic flow matrix. Its core structure includes a self-attention mechanism and a multilayer perceptron.

[0083] After obtaining the second historical traffic flow information, a traffic flow matrix M×N×C is constructed based on the preset threshold M for the number of traffic flow instances, the number of frames N and the number of dimensions C of the historical data of the traffic flow instances in the second historical traffic flow information. This matrix contains the C-dimensional feature information of M traffic flow instances in N frames of historical data.

[0084] If a three-dimensional matrix is ​​used, the dimensions are the preset number of traffic flow instances, the number of frames, and the number of dimensions, so that each traffic flow instance can find its corresponding position in the matrix for each feature information of each frame. If a two-dimensional matrix is ​​used, the number of rows of the matrix corresponds to the preset threshold for the number of traffic flow instances, and the number of columns corresponds to the product of the number of frames and the number of dimensions.

[0085] In practical applications, some traffic flow instances may be missing in certain frames. For these missing traffic flow instance information, the corresponding positions in the matrix are padded with 0 values ​​to ensure the integrity of the matrix and the consistency of the data structure. Simultaneously, to mark the validity of the data, a temporal mask M×N and a spatial mask M×N are constructed. The temporal mask marks the validity of each traffic flow instance in each frame, while the spatial mask marks the valid existence of each traffic flow instance in each frame. These two sets of masks allow for more precise control and processing of valid information in the input data, preventing invalid data from introducing noise that interferes with feature extraction.

[0086] In some implementations, the valid existence and validity of information of each traffic flow instance can be determined based on the completeness of the feature information of each traffic flow instance and the lane line perception performance of vehicles in actual applications.

[0087] Then, the constructed traffic flow matrix, along with the corresponding temporal and spatial domain masks, are input into a pre-established spatiotemporal information feature encoder. This encoder employs a deep learning network architecture, including a self-attention mechanism and a multilayer perceptron, which can efficiently capture the dynamic changes in traffic flow in time and space.

[0088] The self-attention mechanism enables deep learning models to focus on the correlation information between different frames and instances in the traffic flow matrix, automatically assign weights, and explore the time-series dependencies and spatial interactions of traffic flow. The multilayer perceptron then performs nonlinear transformations and deep processing on the correlation features extracted by the self-attention mechanism to further enhance feature representation, remove redundant information, and finally output a historical traffic flow spatiotemporal feature vector that can comprehensively reflect the spatiotemporal dynamics of traffic flow.

[0089] Finally, the spatiotemporal feature vector of the historical traffic flow is fused with the rasterized feature vector of the historical traffic flow and the current visual perception information to generate the target lane line.

[0090] Understandably, the spatiotemporal information feature encoder first fuses the features of each traffic flow instance in the input matrix across different frames using a self-attention mechanism, extracting spatiotemporal features with high-order correlation. Then, these spatiotemporal features are passed to a multilayer perceptron for deep feature learning and abstraction, ultimately outputting a feature vector. This feature vector comprehensively represents the dynamic changes and spatial distribution of each traffic flow instance across historical frames, providing a solid data foundation for subsequent lane line perception tasks.

[0091] Thus, by constructing a traffic flow matrix, the scattered and complex historical traffic flow data is transformed into a structured matrix form, laying the foundation for feature extraction by the encoder. Furthermore, the spatiotemporal information feature encoder can effectively capture the temporal variation patterns and spatial correlation features of traffic flow, improving the accuracy and effectiveness of spatiotemporal feature extraction. This enables the generated target lane lines to accurately reflect the actual road conditions, improving the accuracy of lane line detection and providing reliable lane line data support for the vehicle system.

[0092] Please see Figure 6 In some implementations, step 0233 includes: 02331: Perform the first encoding process on the current visual perception information to obtain the visual perception rasterized feature vector; 02332: The target lane line is generated by fusing the visual perception rasterized feature vector, the historical traffic flow rasterized feature vector, and the historical traffic flow spatiotemporal feature vector.

[0093] In some implementations, the fusion module is further configured to perform a first encoding process on the current visual perception information to obtain a visual perception rasterized feature vector. The fusion module is also configured to perform a fusion process on the visual perception rasterized feature vector, the historical traffic flow rasterized feature vector, and the historical traffic flow spatiotemporal feature vector to generate the target lane line.

[0094] In some implementations, the processor is further configured to perform a first encoding process on the current visual perception information to obtain a visual perception rasterized feature vector. The processor is also configured to perform a fusion process on the visual perception rasterized feature vector, the historical traffic flow rasterized feature vector, and the historical traffic flow spatiotemporal feature vector to generate a target lane line.

[0095] Specifically, the visual perception rasterized feature vector is a feature vector obtained after performing the first encoding process on the current visual perception information, which can accurately reflect the spatial distribution and positional relationship of traffic flow in the BEV space at the current moment.

[0096] The specific method of the first encoding process is consistent with the first encoding of the second historical traffic flow information to ensure the uniformity of the output feature vector.

[0097] When encoding the current visual perception information, the category attributes of each traffic flow instance in the current visual perception information are first encoded and converted to adapt to the recognition and processing requirements of the deep learning model. Then, a standardized raster structure is constructed according to the preset BEV space size, and the key information such as the position and speed of each traffic flow instance is accurately mapped to the corresponding raster unit to form the initial raster data.

[0098] Subsequently, the initial rasterized data is input into a traffic flow rasterized feature encoder that includes a two-dimensional convolutional layer and a multilayer perceptron. Local spatial features are extracted through convolution operations, and then nonlinear transformation and feature enhancement are performed by the multilayer perceptron to finally generate a visual perception rasterized feature vector.

[0099] Next, the visual perception rasterized feature vector, the historical traffic flow rasterized feature vector, and the historical traffic flow spatiotemporal feature vector are input into the preset fusion model.

[0100] Fusion models typically employ a cross-attention mechanism as their core interaction module. This mechanism enables bidirectional information exchange among the three types of feature vectors, automatically learning the correlation weights between different features. For example, it correlates real-time spatial distribution features in the visual perception rasterized feature vectors with historical spatial patterns in the historical traffic flow rasterized feature vectors, while also incorporating the dynamic evolution patterns in the historical traffic flow spatiotemporal feature vectors, achieving deep integration of multi-dimensional features. After the cross-attention mechanism completes the information exchange, the multilayer perceptron further mines and optimizes the integrated features, filtering redundant information and strengthening effective features related to lane line generation. Finally, the target lane line is generated based on the optimized fusion features.

[0101] Thus, by performing the first encoding process on the current visual perception information, a visual perception rasterized feature vector with the same format as the historical traffic flow rasterized feature vector is generated, which effectively solves the alignment problem of traffic flow instance features, lays the foundation for the deep fusion of the three types of features, and realizes the efficient interaction of real-time spatial features, historical spatial features and spatiotemporal evolution features through the cross-attention mechanism. The generated fusion features can comprehensively reflect the current road conditions, historical driving patterns and dynamic changes in traffic flow, improve the accuracy and reliability of target lane line generation, enable lane line detection to adapt to complex and ever-changing traffic environments, and ensure the safety and stability of vehicle driving.

[0102] Please see Figure 7 In some implementations, step 02332 includes: 023321: The historical traffic flow rasterized feature vector and the visual perception rasterized feature vector are fused to generate a fused traffic flow rasterized feature vector; 023322: The target lane line is generated by fusing the rasterized feature vector of the fused traffic flow and the spatiotemporal feature vector of the historical traffic flow.

[0103] In some implementations, the fusion module is further used to fuse the historical traffic flow rasterized feature vector and the visually perceived rasterized feature vector to generate a fused traffic flow rasterized feature vector. The fusion module is also used to fuse the fused traffic flow rasterized feature vector and the historical traffic flow spatiotemporal feature vector to generate the target lane line.

[0104] In some implementations, the processor is further configured to fuse the historical traffic flow rasterized feature vector and the visually perceived rasterized feature vector to generate a fused traffic flow rasterized feature vector. The processor is also configured to fuse the fused traffic flow rasterized feature vector and the historical traffic flow spatiotemporal feature vector to generate a target lane line.

[0105] Specifically, the fused traffic flow rasterized feature vector is a feature vector obtained by fusing the historical traffic flow rasterized feature vector and the visual perception rasterized feature vector. It can comprehensively reflect the spatial distribution pattern and correlation characteristics of traffic flow in the BEV space at different time periods.

[0106] Since both historical traffic flow rasterized feature vectors and visually perceived rasterized feature vectors are rasterized features and can reflect the spatial distribution of traffic flow, they can be fused using methods such as element-wise addition, weighted averaging, or feature integration through shallow neural networks. This fusion method combines the spatial distribution patterns of historical traffic flow with the spatial distribution status in current visually perceived information to generate a fused traffic flow rasterized feature vector. This vector reflects both the empirical patterns of historical traffic flow spatial distribution and the real-time spatial characteristics of current visually perceived information, achieving the enhancement and complementarity of spatial dimension features.

[0107] Subsequently, the rasterized feature vectors of the fused traffic flow and the spatiotemporal feature vectors of the historical traffic flow are fused to form a fused feature that can comprehensively reflect the spatial distribution and spatiotemporal evolution of traffic flow. Finally, the target lane line is generated based on the fused feature.

[0108] Thus, the step-by-step fusion approach effectively avoids information interference and redundancy issues caused by direct fusion of multi-dimensional features. Furthermore, the target lane lines generated by step-by-step fusion can more accurately match the actual road conditions. This allows the vehicle system to accurately infer lane line positions based on the integrated multi-dimensional features in scenarios such as congestion, occlusion, and complex and variable traffic flow, effectively improving the accuracy of lane line detection and providing reliable lane line data support for the vehicle system's path planning and driving control.

[0109] Please see Figure 8 In some implementations, step 023322 includes: 0233221: Extract the fused lane line feature vector based on the fused traffic flow rasterized feature vector; 0233222: The fusion process is performed on the fused lane line feature vector and the historical traffic flow spatiotemporal feature vector to generate the target lane line feature vector; 0233223: Generate the target lane line based on the target lane line feature vector.

[0110] In some implementations, the fusion module is further configured to extract fused lane line feature vectors based on the fused traffic flow rasterized feature vectors. The fusion module is also configured to fuse the fused lane line feature vectors and historical traffic flow spatiotemporal feature vectors to generate target lane line feature vectors. The fusion module is further configured to generate target lane lines based on the target lane line feature vectors.

[0111] In some implementations, the processor is further configured to extract a fused lane line feature vector based on the fused traffic flow rasterized feature vector. The processor is also configured to fuse the fused lane line feature vector and the historical traffic flow spatiotemporal feature vector to generate a target lane line feature vector. The processor is further configured to generate a target lane line based on the target lane line feature vector.

[0112] Specifically, the fused lane line feature vector is a feature vector extracted from the fused traffic flow rasterized feature vector, which is directly related to lane line generation and can reflect key spatial information such as lane line position and direction.

[0113] The target lane line feature vector is a feature vector obtained by fusing the lane line feature vector and the historical traffic flow spatiotemporal feature vector. It can comprehensively reflect the key spatial features and spatiotemporal evolution patterns related to lane lines.

[0114] The fused traffic flow rasterized feature vector includes the overall spatial distribution characteristics of historical traffic flow information and current visual perception information. It includes not only the key features of lane lines, but also other redundant information. Through a pre-set feature extraction network, it can accurately identify and filter the features in the fused traffic flow rasterized feature vector that are closely related to the position and direction of lane lines, such as the concentrated areas of traffic flow trajectories and the boundary areas of traffic flows in different directions.

[0115] During the extraction process, operations such as convolution and pooling can be used to enhance the expression of key lane line features, filter out noise information that is irrelevant to lane lines, and finally generate a fused lane line feature vector, which can directly provide reliable spatial feature support for lane line generation.

[0116] Next, a fusion architecture combining a cross-attention mechanism and a multilayer perceptron, namely a global feature encoder, is employed to fuse the fused lane line feature vector and the historical traffic flow spatiotemporal feature vector. The cross-attention mechanism interacts and assigns weights to the spatial key features in the fused lane line feature vector and the dynamic evolution patterns in the historical traffic flow spatiotemporal feature vector. This allows the spatial key features in the fused lane line feature vector to be supported by the dynamic patterns in the traffic flow spatiotemporal features, supplementing the spatial information of occluded areas. Simultaneously, it allows the traffic flow spatiotemporal features to be constrained by the range of the spatial key features, effectively avoiding the ineffective application of dynamic patterns. Ultimately, this achieves a precise correlation between the spatial location of lane lines and the dynamic patterns of traffic flow, providing reliable fused features for subsequent multilayer perceptron processing and lane line instantiation, and ensuring the accuracy of lane line perception in complex scenarios to a certain extent.

[0117] The multilayer perceptron performs further nonlinear transformation and deep abstraction on the interactive features, and integrates them to form a target lane line feature vector. This vector has both spatial specificity and spatiotemporal correlation, and can fully support lane line generation.

[0118] Finally, the target lane line feature vector is input to the terminal encoder, which includes network components such as convolutional layers. It can map the high-dimensional target lane line feature vector back to the actual BEV space through convolution operations, restoring the geometric shape, position coordinates and direction information of the lane line.

[0119] During the mapping process, the terminal encoder accurately generates lane line instances that conform to the actual road conditions based on the distribution pattern of feature vectors, so as to ensure the continuity and accuracy of lane lines and provide reliable lane line data for the vehicle system.

[0120] Thus, by extracting and fusing lane line feature vectors, redundant information in the fusing traffic flow rasterized feature vectors is effectively filtered out, spatial features directly related to lane line generation are strengthened, and the relevance and effectiveness of the features are improved. Furthermore, the subsequent fusion with historical traffic flow spatiotemporal feature vectors achieves a deep integration of key spatial features and spatiotemporal evolution patterns. The generated target lane line feature vectors can accurately characterize the attributes of lane lines, effectively improving the accuracy of lane line detection and providing solid lane line data support for path planning and driving control of the vehicle system.

[0121] This application also provides an electrical device, including: a processor and a memory for storing processor-executable instructions, wherein the processor is configured to execute instructions to implement the methods of some of the above embodiments.

[0122] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods described in some of the above embodiments.

[0123] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the methods described in some of the above embodiments.

[0124] It is understood that a computer program includes computer program code. Computer program code can be in the form of source code, object code, executable files, or some intermediate form. Computer-readable storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc.

[0125] In this specification, the terms "specifically," "furthermore," "particularly," "understandably," etc., refer to specific features, structures, materials, or characteristics described in connection with embodiments or examples that are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0126] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of executable request code comprising one or more steps for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0127] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for generating lane lines, characterized in that, The method includes: Acquire first historical traffic flow information, wherein the first historical traffic flow information includes multi-frame historical traffic flow instance data of the target road; The first historical traffic flow information and the current visual perception information are fused together to generate the target lane line.

2. The method according to claim 1, characterized in that, The step of fusing the first historical traffic flow information and the current visual perception information to generate the target lane line includes: The coordinate system of the target traffic flow instance in the current visual perception information is determined as the target coordinate system; Based on the pose of the target traffic flow instance and the pose of each traffic flow instance in the first historical traffic flow information, the first historical traffic flow information is transformed to the target coordinate system; The second historical traffic flow information and the current visual perception information are fused to generate the target lane line, wherein the second historical traffic flow information is the traffic flow information transformed from the first historical traffic flow information to the target coordinate system.

3. The method according to claim 2, characterized in that, The process of fusing the second historical traffic flow information and the current visual perception information to generate the target lane line includes: The second historical traffic flow information is subjected to a first encoding process to obtain a rasterized feature vector of historical traffic flow; The second historical traffic flow information is subjected to a second encoding process to obtain the spatiotemporal feature vector of historical traffic flow; The target lane line is generated by fusing the historical traffic flow rasterized feature vector, the historical traffic flow spatiotemporal feature vector, and the current visual perception information.

4. The method according to claim 3, characterized in that, The second encoding process of the second historical traffic flow information to obtain the historical traffic flow spatiotemporal feature vector includes: A traffic flow matrix is ​​constructed based on the number of frames of historical data of traffic flow instances in the historical traffic flow information, the number of dimensions of historical data of traffic flow instances, and a preset threshold for the number of traffic flow instances, wherein the dimensions represent the feature information category of the traffic flow instance; The traffic flow matrix is ​​input into a pre-established spatiotemporal information feature encoder to obtain the historical traffic flow spatiotemporal feature vector.

5. The method according to claim 3, characterized in that, The process of fusing the historical traffic flow rasterized feature vector, the historical traffic flow spatiotemporal feature vector, and the current visual perception information to generate the target lane line includes: The current visual perception information is subjected to a first encoding process to obtain the current visual perception rasterized feature vector; The target lane line is generated by fusing the current visual perception rasterized feature vector, the historical traffic flow rasterized feature vector, and the historical traffic flow spatiotemporal feature vector.

6. The method according to claim 5, characterized in that, The process of fusing the current visual perception rasterized feature vector, the historical traffic flow rasterized feature vector, and the historical traffic flow spatiotemporal feature vector to generate the target lane line includes: The historical traffic flow rasterized feature vector and the current visual perception rasterized feature vector are fused to generate a fused traffic flow rasterized feature vector; The target lane line is generated by fusing the fused traffic flow rasterized feature vector and the historical traffic flow spatiotemporal feature vector.

7. The method according to claim 6, characterized in that, The step of generating the target lane line based on the fused traffic flow rasterized feature vector and the historical traffic flow spatiotemporal feature vector includes: Based on the fused traffic flow rasterized feature vector, extract the fused lane line feature vector; The fused lane line feature vector and the historical traffic flow spatiotemporal feature vector are fused to generate the target lane line feature vector. The target lane line is generated based on the target lane line feature vector.

8. An electrical appliance, characterized in that, include: A processor and a memory for storing processor-executable instructions, wherein the processor is configured to execute the instructions to implement the method as claimed in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-7.

Images