Road information determination method and device, electronic equipment and medium

Abstract

The invention provides a road information determination method and device, electronic equipment and a medium, and relates to the technical field of intelligent driving, and the method comprises the steps: obtaining a plurality of sensor data of a current frame of a vehicle, the sensor data comprising image data and geographic space data; based on the image data and the geographic space data, a perception lane feature map corresponding to the image data and a non-perception lane feature map corresponding to the geographic space data are determined, the perception lane feature map comprises surrounding environment features of the vehicle, and the non-perception lane feature map comprises beyond-visual-range map features and lane attribute features of the vehicle; fusing the non-perceptual lane feature map and the perceptual lane feature map to obtain a fused lane feature map; and decoding the fused lane feature map to obtain the road information of the fused lane feature map, thereby effectively solving the limitation of processing a long tail problem by depending on a manual setting rule in a post-fusion scheme, and realizing supplement and fusion of beyond-visual-range map information and lane attribute information.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent driving technology, and in particular to a method, apparatus, electronic device and medium for determining road information. Background Technology

[0002] Autonomous driving technology relies on a vehicle's ability to perceive its surroundings, especially in urban and highway scenarios. Vehicles can accurately acquire road information within a certain range using neural network technology, but limitations in camera line-of-sight and network capabilities prevent them from effectively acquiring information beyond line-of-sight and specific lane attributes. Therefore, high-precision maps are used to supplement this information and improve the environmental perception capabilities of autonomous driving systems.

[0003] Currently, related technologies employ a "post-fusion" approach, combining map information with vehicle perception results. This involves precisely locating the vehicle's position, transforming the map to the vehicle's coordinate system, and then using manually defined rules for fusion to fill in blind spots and lane attributes. However, this method heavily relies on high-precision absolute positioning and is limited by the timeliness of map information and potential cartographic errors. This results in limitations in practical applications, particularly in handling abnormal scenarios, leaving long-tail problems that are difficult to effectively address through manually defined rules. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and medium for determining road information to solve the problems caused by the "post-fusion" method in related technologies. By fusing the image data and geospatial data of the current frame of the vehicle, a perceptual lane feature map containing environmental features and a non-perceptual lane feature map containing beyond-line-of-sight map features and lane attribute features are constructed respectively. The two are then fused and decoded to directly output road information. This effectively solves the limitation of relying on manually set rules to handle long-tail problems in post-fusion schemes and realizes intelligent supplementation and fusion of beyond-line-of-sight map information and lane attribute information.

[0005] A first aspect of this disclosure provides a method for determining road information. The method includes: acquiring multiple sensor data of a vehicle in the current frame, the sensor data including image data and geospatial data; determining a perceived lane feature map corresponding to the image data and a non-perceived lane feature map corresponding to the geospatial data based on the image data and the geospatial data, respectively, the perceived lane feature map including features of the vehicle's surrounding environment, and the non-perceived lane feature map including the vehicle's beyond-line-of-sight map features and lane attribute features; fusing the non-perceived lane feature map and the perceived lane feature map to obtain a fused lane feature map; and decoding the fused lane feature map to obtain road information from the fused lane feature map.

[0006] In some embodiments of this disclosure, multiple sensor data of a vehicle are acquired, and then the method includes: distorting the image data to obtain distorted image data.

[0007] In some embodiments of this disclosure, obtaining a perceived lane feature map corresponding to image data includes: extracting an initial feature map corresponding to the image data using the backbone network in a convolutional neural network; and aggregating the surrounding environment features in the initial feature map using first feature query information and a first attention mechanism to obtain the perceived lane feature map.

[0008] In some embodiments of this disclosure, the geospatial data includes navigation data and local map data. Obtaining the non-perceptive lane feature map corresponding to the geospatial data includes: encoding the navigation data and local map data through a vector coding network to obtain the non-perceptive lane feature map. The non-perceptive lane feature map includes beyond-line-of-sight map features and lane attribute features, which are obtained by converting the navigation data and local map data.

[0009] In some embodiments of this disclosure, the non-perceived lane feature map and the perceived lane feature map are fused to obtain a fused lane feature map, including: using a second attention mechanism to fuse the non-perceived lane feature map and the perceived lane feature map to obtain an initial fused lane feature map; and iteratively updating the initial fused lane feature map according to a preset number of iterations to obtain the fused lane feature map.

[0010] In some embodiments of this disclosure, after obtaining a perceived lane feature map corresponding to image data and a non-perceived lane feature map corresponding to geospatial data, the method includes: obtaining a fused lane feature map of the vehicle in the previous frame.

[0011] In some embodiments of this disclosure, the non-perceptive lane feature map and the perceived lane feature map are fused to obtain a fused lane feature map, including: using a second attention mechanism to fuse the non-perceptive lane feature map, the perceived lane feature map, and the fused lane feature map of the vehicle in the previous frame to obtain an initial fused lane feature map; and iteratively updating the initial fused lane feature map according to a preset number of iterations to obtain the fused lane feature map.

[0012] A second aspect of this disclosure provides a road information determination apparatus, the apparatus comprising:

[0013] The first acquisition unit is used to acquire multiple sensor data of the vehicle in the current frame, including image data and geospatial data.

[0014] The second acquisition unit is used to determine the perceived lane feature map corresponding to the image data and the non-perceived lane feature map corresponding to the geospatial data based on the image data and geospatial data, respectively. The perceived lane feature map includes the surrounding environment features of the vehicle, and the non-perceived lane feature map includes the vehicle's beyond-line-of-sight map features and lane attribute features.

[0015] The fusion unit is used to fuse the non-perceived lane feature map and the perceived lane feature map to obtain the fused lane feature map;

[0016] The decoding unit is used to decode the fused lane feature map to obtain the road information of the fused lane feature map.

[0017] In some embodiments of this disclosure, the apparatus further includes a preprocessing unit, configured to perform distortion processing on the image data after acquiring multiple sensor data of the vehicle, to obtain distorted image data.

[0018] In some embodiments of this disclosure, the second acquisition unit is configured to: extract an initial feature map corresponding to the image data using the backbone network in the convolutional neural network; and aggregate the surrounding environment features in the initial feature map using the first feature query information and the first attention mechanism to obtain a perceived lane feature map.

[0019] In some embodiments of this disclosure, the geospatial data includes navigation data and local map data. The second acquisition unit is configured to: encode the navigation data and local map data through a vector coding network to obtain a non-perceptive lane feature map. The non-perceptive lane feature map includes beyond-line-of-sight map features and lane attribute features, which are obtained by converting the navigation data and local map data.

[0020] In some embodiments of this disclosure, the fusion unit is configured to: fuse the non-perceived lane feature map and the perceived lane feature map using a second attention mechanism to obtain an initial fused lane feature map; and iteratively update the initial fused lane feature map according to a preset number of iterations to obtain the fused lane feature map.

[0021] In some embodiments of this disclosure, the second acquisition unit is further configured to: after acquiring the perceived lane feature map corresponding to the image data and the non-perceived lane feature map corresponding to the geospatial data, acquire the fused lane feature map of the vehicle in the previous frame.

[0022] In some embodiments of this disclosure, the fusion unit is further configured to: use a second attention mechanism to fuse the non-perceived lane feature map, the perceived lane feature map, and the fused lane feature map of the vehicle in the previous frame to obtain an initial fused lane feature map; and iteratively update the initial fused lane feature map according to a preset number of iterations to obtain the fused lane feature map.

[0023] A third aspect of this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described in the first aspect of this disclosure.

[0024] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the first aspect of this disclosure.

[0025] In summary, the road information determination method proposed in this disclosure acquires multiple sensor data from the current frame of the vehicle, including image data and geospatial data. Based on the image data and geospatial data, a perceptual lane feature map corresponding to the image data and a non-perceptual lane feature map corresponding to the geospatial data are determined, respectively. The perceptual lane feature map includes the vehicle's surrounding environment features, while the non-perceptual lane feature map includes the vehicle's beyond-line-of-sight map features and lane attribute features. The non-perceptual lane feature map and the perceptual lane feature map are fused to obtain a fused lane feature map. The fused lane feature map is then decoded to obtain the road information of the fused lane feature map. This effectively solves the limitation of relying on manually set rules to handle long-tail problems in post-fusion schemes and realizes the supplementation and fusion of beyond-line-of-sight map information and lane attribute information.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0028] Figure 1 A flowchart of a road information determination method provided in this embodiment of the disclosure;

[0029] Figure 2 A flowchart of a road information determination method provided in this embodiment of the disclosure;

[0030] Figure 3 A flowchart of a road information determination method provided in this embodiment of the disclosure;

[0031] Figure 4 A schematic diagram illustrating a specific road information determination method provided in this embodiment of the disclosure;

[0032] Figure 5A schematic diagram of local map data obtained by a road information determination method provided in an embodiment of this disclosure;

[0033] Figure 6 This is a schematic diagram of a local map data projected onto an image obtained by a road information determination method provided in this embodiment of the present disclosure.

[0034] Figure 7 This is a schematic diagram of the structure of a road information determination device provided in an embodiment of the present disclosure;

[0035] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure;

[0036] Figure 9 This is a schematic diagram of the chip structure provided in an embodiment of this disclosure. Detailed Implementation

[0037] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments are described below with reference to the accompanying drawings.

[0038] In autonomous driving technology, especially in urban and highway scenarios, accurate perception of the surrounding environment is fundamental to planning decisions and driving behavior. With advancements in neural network technology, vehicle-mounted perception networks (such as BEV perception) can accurately acquire the location and structural information of road elements such as lanes and lane lines within a certain range (e.g., 100 meters) around the vehicle. However, these perception networks still have limitations, particularly in acquiring beyond-line-of-sight information (such as lane and lane line information at opposite intersections) and information on special lane attributes (such as tidal flow lanes, bus lanes, and non-motorized vehicle lanes).

[0039] To supplement this information, related technologies rely on post-fusion of maps. Post-fusion methods use perception results and other sensors (such as GPS, IMU, etc.) to pinpoint the vehicle's exact location on the map, convert the map information to the vehicle's own coordinate system, and fuse it with the perception results. This completes the parts that are not visible to the perception and binds lane attributes from the map to the perception results.

[0040] However, traditional post-fusion methods have several limitations. First, they require high accuracy and stability in absolute positioning, while sensor performance (such as GPS signals) may be limited in real-world scenarios, leading to unstable positioning accuracy. Second, maps are not entirely accurate and may differ from reality due to timeliness or mapping errors. These issues mean that post-fusion methods may require extensive manual rule-based processing when dealing with complex scenarios, leaving many long-tail problems that are difficult to resolve.

[0041] To address the problems existing in related technologies, this disclosure proposes a road information determination method. By directly fusing image data, map data, and navigation data in a network, without relying on visual positioning, it utilizes the encoder-decoder framework, multi-scale feature fusion, and attention mechanisms to achieve all functions of static perception, visual positioning, and post-fusion. This not only improves the perception accuracy and positioning stability of autonomous driving systems but also optimizes the fusion effect and reduces computational complexity.

[0042] The road information determination method proposed in this disclosure can be widely applied in the field of intelligent driving, especially in autonomous driving scenarios in complex road environments, urban intersections, highways, and at night or in adverse weather conditions. By accurately fusing image, map, and navigation data, it can improve the vehicle's environmental perception, positioning accuracy, and decision-making safety.

[0043] The method for determining road information provided in this application will be described in detail below with reference to the accompanying drawings.

[0044] Figure 1 This is a flowchart illustrating a road information determination method provided in an embodiment of this disclosure. Figure 1 As shown, the road information determination method includes steps 101-104.

[0045] Step 101: Obtain multiple sensor data of the vehicle in the current frame. The sensor data includes image data and geospatial data.

[0046] In this embodiment of the disclosure, the multiple sensor data are collected from multiple sensors installed on the vehicle. Since the road information determination method of this disclosure is applied to a neural network architecture, this disclosure can directly receive multiple sensor data sent by multiple sensors. The multiple sensors installed on the vehicle include, but are not limited to, cameras (for capturing image data), Global Positioning System (GPS, for providing geospatial data such as geographic coordinates and speed), LiDAR (for providing high-precision three-dimensional environmental data), and possibly an Inertial Navigation System (INS).

[0047] Image data is primarily collected by in-vehicle cameras. These cameras are typically mounted in key locations on the vehicle, such as the front and rear bumpers, near the rearview mirrors, or on the roof, to capture real-time images of the vehicle's surroundings. Geospatial data is mainly collected by the vehicle's positioning system (such as GPS) and other sensors (such as radar, lidar, etc.).

[0048] Image data is used to capture real-time visual information about the vehicle's surroundings, while geospatial data provides key information such as the vehicle's precise location under the Global Positioning System, beyond-line-of-sight location data, and lane attribute data.

[0049] Step 102: Based on image data and geospatial data, determine the perceived lane feature map corresponding to the image data and the non-perceived lane feature map corresponding to the geospatial data, respectively. The perceived lane feature map includes the vehicle's surrounding environment features, and the non-perceived lane feature map includes the vehicle's beyond-line-of-sight map features and lane attribute features.

[0050] In embodiments of this disclosure, the acquired image data and geospatial data can be processed separately in a neural network architecture to obtain a perceived lane feature map and a non-perceived lane feature map.

[0051] Specifically, this disclosure can analyze image data using machine learning algorithms (such as convolutional neural networks CNN) to extract features of the vehicle's surrounding environment, such as lane lines, traffic signs, and obstacles. Based on these features, a perceptual lane feature map is obtained, which includes direct visual information from the vehicle's current driving environment.

[0052] This disclosure can also encode geospatial data (such as navigation data, local map data, etc.) to obtain a non-perceptive lane feature map. The non-perceptive lane feature map can include not only the vehicle's current location and surrounding map information (such as road shape, intersections, service areas, etc.), but also beyond-line-of-sight map information (i.e., information that the vehicle cannot directly see but is known from the map data) and lane attribute information (such as the number of lanes, lane direction, speed limit, etc.).

[0053] Step 103: Fuse the non-perceived lane feature map and the perceived lane feature map to obtain the fused lane feature map.

[0054] In embodiments of this disclosure, after obtaining the non-perceived lane feature map and the perceived lane feature map, this disclosure can directly fuse the non-perceived lane feature map and the perceived lane feature map in a neural network architecture.

[0055] This disclosure provides a method for fusing non-perceptive lane feature maps and perceived lane feature maps using various techniques, such as feature stitching, weighted summation, and feature fusion layers in deep learning models. The purpose of this fusion is to combine real-time visual information from the perceived lane feature map with map information and beyond-line-of-sight information from the non-perceptive lane feature map, thereby obtaining a more comprehensive and accurate description of the vehicle's surrounding environment.

[0056] Step 104: Decode the fused lane feature map to obtain the road information of the fused lane feature map.

[0057] In embodiments of this disclosure, after obtaining the fused lane feature map, the fused lane feature map can be decoded in a neural network architecture to extract useful road information and output the road information. The road information may include road elements and road attributes corresponding to the road elements, such as the specific location of lane lines, the vehicle's current lane, traffic conditions ahead, possible turns or intersections, etc.

[0058] The decoding process disclosed herein can be achieved through semantic segmentation, object detection, and other methods. Ultimately, the decoded road information can be used for downstream tasks such as intelligent driving decisions and path planning for vehicles.

[0059] In summary, the road information determination method proposed in this disclosure acquires multiple sensor data from the current frame of the vehicle, including image data and geospatial data. Based on the image data and geospatial data, a perceptual lane feature map corresponding to the image data and a non-perceptual lane feature map corresponding to the geospatial data are determined, respectively. The perceptual lane feature map includes the vehicle's surrounding environment features, while the non-perceptual lane feature map includes the vehicle's beyond-line-of-sight map features and lane attribute features. The non-perceptual lane feature map and the perceptual lane feature map are fused to obtain a fused lane feature map. The fused lane feature map is then decoded to obtain the road information of the fused lane feature map. This effectively solves the limitation of relying on manually set rules to handle long-tail problems in post-fusion schemes and realizes the supplementation and fusion of beyond-line-of-sight map information and lane attribute information.

[0060] based on Figure 1 The embodiment shown, Figure 2 A flowchart of a road information determination method proposed in this disclosure is further shown. Figure 2 based on Figure 1 The illustrated embodiment further defines steps 101, 102, and 103. Figure 2 In the illustrated embodiment, step 101 is followed by step 202, step 102 includes steps 203 and 204, step 102 includes step 205, and step 103 includes steps 205 and 207. For example... Figure 2 As shown, the method includes the following steps:

[0061] Step 201: Obtain multiple sensor data of the vehicle in the current frame. The sensor data includes image data and geospatial data.

[0062] In embodiments of this disclosure, the disclosure may receive multiple sensor data from the current frame of the vehicle. In other words, the disclosure may use multiple sensor data from the current frame of the vehicle as input to a neural network, so that the disclosure may use the neural network to process the multiple sensor data.

[0063] Since the image data disclosed herein is acquired through vehicle-mounted cameras, and these cameras can be arranged facing different directions outside the vehicle (e.g., vehicle-mounted cameras may include front-view cameras, rear-view cameras, and side cameras), the image data disclosed herein may include surround view image data and circumferential view image data.

[0064] Step 202: Perform distortion processing on the image data to obtain distorted image data.

[0065] In embodiments of this disclosure, before feature extraction from image data, the image data may be preprocessed to improve image quality or make it more suitable for subsequent image processing tasks. Specifically, the preprocessing may include distortion correction.

[0066] The distortion correction disclosed herein can include various types of image transformations, such as perspective transformations, affine transformations, and radial distortion correction. These transformations are typically based on mathematical models and can adjust the shape, size, orientation, or position of an image to eliminate or mitigate image distortion caused by lens distortion, shooting angle, or other factors. For example, radial distortion is one of the common types of lens distortion, which causes straight lines in an image to appear curved. Through radial distortion correction, this disclosure can adjust the pixel positions in an image, restoring straight lines to their original straight-line state.

[0067] In the embodiments of this disclosure, the purpose of distorting the image data is to optimize the image data and make it more suitable for subsequent feature extraction steps. This disclosure improves the accuracy and recognizability of key features in an image by distorting the image data, thereby enhancing the performance and accuracy of the entire image processing system.

[0068] It is understood that the specific methods and parameters for distortion processing can be determined according to the actual application scenario and the needs of the image processing task, and are not limited in the embodiments disclosed herein.

[0069] Step 203: Use the backbone network in the convolutional neural network to extract the initial feature map corresponding to the image data.

[0070] In embodiments of this disclosure, feature extraction can be performed on the distorted image data, that is, the distorted image data can be input into the backbone network of a convolutional neural network.

[0071] The backbone network is a deep neural network composed of multiple convolutional layers, pooling layers (and sometimes activation function layers). Through the stacking of multiple convolutional layers, pooling layers, and activation function layers, the backbone network can progressively extract multi-level and multi-scale feature information from images.

[0072] This disclosure obtains an initial feature map corresponding to the input image through backbone network processing. The initial feature map includes multi-level and multi-scale feature information from the image data.

[0073] Step 204: Using the first feature query information and the first attention mechanism, the surrounding environment features in the initial feature map are aggregated to obtain the perceived lane feature map.

[0074] In embodiments of this disclosure, after extracting the initial feature map, a first feature query (also referred to as the initial query, which may be preset, experience-based, or learned through pre-training) may be introduced. The first feature query guides the extraction of specific, lane-recognition-related surrounding environmental features from the initial feature map. The first feature query may be a set of vectors representing a "template" or "expectation" of the features to be extracted from the initial feature map. In practical applications, the number and dimensions of queries in the first query will be determined according to the specific task requirements and are not limited in embodiments of this disclosure.

[0075] Meanwhile, this disclosure can utilize a first attention mechanism to dynamically select and aggregate surrounding environmental features on the initial feature map based on the first feature query information.

[0076] The first attention mechanism disclosed herein can be a soft attention mechanism, a hard attention mechanism, a self-attention mechanism, a deformable attention mechanism, etc. In order to better capture the complex structures and changes in the image, this disclosure may choose a deformable attention mechanism as the first attention mechanism.

[0077] Deformable attention mechanisms offer greater flexibility and adaptability compared to traditional attention mechanisms (such as soft and hard attention mechanisms), making them suitable for handling irregular shapes or deformable objects, as well as tasks requiring precise capture of image details. In a deformable attention mechanism, each piece of query information in the initial feature query is associated with one or more "key points." These key points are pre-determined on the initial feature map and represent the locations where the query information should focus. The deformable attention mechanism allows for subtle offsets of these key points on the initial feature map to more accurately capture the features required by the query information.

[0078] Specifically, for each query, this disclosure can calculate its similarity (or relevance) to each position on the initial feature map, and then perform a weighted summation of the features on the initial feature map based on the similarity score to obtain the image query features corresponding to the query. In the deformable attention mechanism, the weighted summation process is dynamic because the positions of key points are adjusted according to the needs of the query.

[0079] Through the above process, this disclosure can extract image query features (i.e., surrounding environment features) related to the query information from the initial feature map using the first query information and the first attention mechanism, thereby generating a perception lane feature map (image query feature).

[0080] Step 205: Encode the navigation data and local map data using a vector coding network to obtain a non-perceptive lane feature map.

[0081] In this disclosure, the non-perceptual lane feature map includes beyond-line-of-sight map features and lane attribute features, which are obtained by converting navigation data and local map data.

[0082] In embodiments of this disclosure, geospatial data includes navigation data and local map data.

[0083] Navigation data can include the vehicle’s current location (such as latitude and longitude), destination information, route planning, and other beyond-line-of-sight map data.

[0084] Local map data can include detailed map information about the vehicle's surrounding environment, such as lane structure, lane lines, traffic signs, obstacles, and other lane attribute data.

[0085] Navigation data and local map data can be used as input data into a vector coding network. The task of the vector coding network is to transform this raw navigation data and local map data into a more compact and easier-to-process feature representation, namely a non-aware lane feature map (map query feature). The vector coding network can include multiple layers, such as convolutional layers, pooling layers, and fully connected layers, to extract and transform features from the input data.

[0086] The non-perceptual lane feature map includes useful information extracted from the input navigation data and local map data, such as beyond-line-of-sight map features and lane attribute features. Beyond-line-of-sight map features provide information about the road structure far ahead of the vehicle, helping it understand road changes it will encounter in the future, such as turns, merges, and forks. Lane attribute features provide specific attributes of the current lane, such as lane width, lane direction, and lane type (e.g., highway, city street).

[0087] Step 206: Using the second attention mechanism, the non-perceptual lane feature map and the perceived lane feature map are fused to obtain the initial lane feature map.

[0088] In embodiments of this disclosure, a second attention mechanism can be used to combine the obtained non-perceptive lane feature map and perceived lane feature map. The second attention mechanism in this disclosure can be a cross-attention mechanism. Cross-attention is a special type of attention mechanism that allows the model to focus on relevant information in another sequence (or feature map) while processing one sequence (or feature map).

[0089] Specifically, this disclosure allows both the non-perceptive lane feature map and the perceived lane feature map to be simultaneously input into a second attention mechanism. The second attention mechanism then calculates the correlation score between the two feature maps. Based on these correlation scores, a weighted sum is applied to the features in one of the feature maps (typically the perceived lane feature map) to obtain a feature representation that incorporates non-perceptive information. This process is bidirectional, and similar processing can be performed on the non-perceptive lane feature map to incorporate perceived information.

[0090] Step 207: Based on the preset number of iterations, iteratively update the initial lane feature map to obtain the fused lane feature map.

[0091] In embodiments of this disclosure, after obtaining the initial lane feature map, the initial lane feature map can be further fused and optimized through an iterative update process.

[0092] The preset iteration count is a pre-defined parameter that determines the number of iterations. In this disclosure, the preset iteration count can be 0 (i.e., no iteration update) or multiple times. This disclosure uses a preset iteration count of 6 as an example, meaning that the initial lane feature map will be updated 6 times.

[0093] In each iteration, this disclosure can use the initial lane feature map as the perceived lane feature map and use a second attention mechanism to further fuse and optimize the perceived lane feature map and the non-perceived lane feature map.

[0094] After a predetermined number of iterations, this disclosure yields the final fused lane feature map. The fused lane feature map integrates non-perceptual and perceptual information and has undergone multiple optimizations, thus more accurately reflecting the vehicle's current lane information.

[0095] In addition, after obtaining the fused lane feature map, this disclosure may also store the fused lane feature map of the current frame in a database.

[0096] Step 208: Decode the fused lane feature map to obtain the road information of the fused lane feature map.

[0097] In summary, the method provided in this disclosure, employing an end-to-end network architecture, achieves post-fusion functionality, enabling the neural network to uniformly handle absolute localization and fusion tasks. It efficiently integrates beyond-line-of-sight map information (such as intersection sight distance) and lane attributes into the perception results. Compared to traditional post-fusion methods, using neural networks to fuse map and perception information effectively adopts a data-driven approach. By allowing the neural network to learn the characteristics of the data, it effectively solves the long-tail problem in post-fusion schemes that is difficult to cover due to manually set rules, achieving a more accurate and comprehensive fusion effect.

[0098] based on Figure 1 The embodiment shown, Figure 3 A flowchart of a road information determination method proposed in this disclosure is further shown. Figure 3 based on Figure 1 The illustrated embodiment further defines steps 101, 102, 103, and 104. Figure 3 In the illustrated embodiment, step 101 is followed by step 302, step 102 includes steps 303 and 304, step 102 includes step 305, step 102 is followed by step 306, and step 103 includes steps 307 and 308. For example... Figure 3 As shown, the method includes the following steps:

[0099] Step 301: Obtain multiple sensor data of the vehicle in the current frame. The sensor data includes image data and geospatial data.

[0100] Step 302: Perform distortion processing on the image data to obtain distorted image data.

[0101] Step 303: Use the backbone network in the convolutional neural network to extract the initial feature map corresponding to the image data.

[0102] Step 304: Using the first feature query information and the first attention mechanism, the surrounding environment features in the initial feature map are aggregated to obtain the perceived lane feature map.

[0103] Step 305: Encode the navigation data and local map data using a vector coding network to obtain a non-perceptive lane feature map.

[0104] In this disclosure, geospatial data includes navigation data and local map data. The non-perceptual lane feature map includes beyond-line-of-sight map features and lane attribute features, which are derived from the navigation data and local map data.

[0105] It is understandable that the implementation methods of steps 301 to 305 above can be referred to Figure 2 Steps 201 to 204 in the illustrated embodiment will be performed, and will not be described again here.

[0106] Step 306: Obtain the fused lane feature map of the vehicle in the previous frame.

[0107] In embodiments of this disclosure, the fused lane feature map of the vehicle in the previous frame can also be obtained, that is, the fused lane feature map of the vehicle in the previous frame is determined from the database and transmitted to the current frame through relative positioning.

[0108] In this disclosure, relying solely on the fused lane feature map of the vehicle's current frame may be insufficient to handle all situations. For example, when the vehicle is moving rapidly or encountering a complex environment, the information from the current frame may not provide enough context to make the correct decision. To address this issue, this disclosure may acquire the fused lane feature map of the vehicle's previous frame.

[0109] Relative positioning uses vehicle motion models (such as wheel speed sensors, IMUs, etc.) and map information (such as GPS, local maps, etc.) to estimate the vehicle's position change in the current frame relative to the previous frame. This technique can provide the relative positional relationship of the vehicle between different frames, thereby allowing the present disclosure to transfer information from the previous frame to the current frame.

[0110] In embodiments of this disclosure, the present disclosure can utilize relative positioning technology to transfer the fused lane feature map (pre-query feature) of the previous frame to the current frame, that is, to use relative positioning technology to obtain the fused lane feature map of the vehicle in the previous frame.

[0111] Step 307: Using the second attention mechanism, the non-perceptive lane feature map, the perceived lane feature map, and the fused lane feature map of the vehicle in the previous frame are fused to obtain the initial lane feature map.

[0112] In the embodiments of this disclosure, the same method as step 206 can be used to fuse the non-perceptive lane feature map, the perceived lane feature map, and the fused lane feature map of the vehicle in the previous frame using the second attention mechanism to obtain an initial lane feature map. The specific process will not be described here.

[0113] Step 308: Based on the preset number of iterations, iteratively update the initial lane feature map to obtain the fused lane feature map.

[0114] Step 309: Decode the fused lane feature map to obtain the road information of the fused lane feature map.

[0115] It is understandable that the implementation methods of steps 308 to 309 above can refer to... Figure 2 Steps 207 to 208 in the illustrated embodiment will be performed, and will not be described again here.

[0116] In summary, the method provided in this disclosure, employing an end-to-end network architecture, achieves post-fusion functionality. This enables the neural network to simultaneously handle absolute localization and fusion tasks, and effectively integrates the fused lane feature map from the previous frame. It efficiently integrates beyond-line-of-sight map information (such as intersection sight distance) and lane attributes, along with the contextual information from the previous frame, into the current perception result. Furthermore, by effectively employing a data-driven approach, allowing the neural network to learn the characteristics of the data, it not only solves the long-tail problem in post-fusion schemes where manually set rules are difficult to cover, but also further improves the continuity and accuracy of the fusion effect.

[0117] based on Figures 1 to 3 The illustrated embodiment provides a schematic diagram of a specific method for determining road information, as shown below. Figure 4 As shown.

[0118] In embodiments of this disclosure, reference is made to Figure 4 This disclosure allows for the acquisition of sensor data from the vehicle's current frame, including image data and geospatial data. The image data may include surround view image data and perimeter view image data, while the geospatial data may include navigation data and local map data. Specifically, the local map data may be as follows: Figure 5 , Figure 6 As shown, Figure 5 This is a schematic diagram of local map data obtained using a road information determination method. Figure 6 This is a schematic diagram of a local map data projected onto an image obtained from a road information determination method. Different line segments represent different lane attributes, such as lane lines, ground markings, stop lines, etc.

[0119] After obtaining the image data, this disclosure performs distortion processing on the panoramic and circumferential image data to obtain distorted panoramic and circumferential image data. Then, deep learning techniques (such as the backbone network in a convolutional neural network CNN) are used to process the distorted panoramic and circumferential image data to generate an initial feature map. Next, using the first feature query information (initial query) and a first attention mechanism (deformable attention), the surrounding environment features in the initial feature map are aggregated to obtain a perceptual lane feature map (image query feature) containing the surrounding environment features.

[0120] Meanwhile, navigation data and local map data can be processed by a vector coding network to generate a non-perceptual lane feature map (map query feature) containing beyond-line-of-sight map features and lane attribute features.

[0121] In addition, this disclosure can also obtain the fused lane feature map (pre-query feature) of the vehicle in the previous frame, which is transmitted to the current frame through relative positioning technology, to enhance the continuity of features and contextual information.

[0122] Next, this disclosure can fuse these three feature maps through a cross attention mechanism to generate an initial lane feature map.

[0123] To further optimize the feature representation, this disclosure can take a preset iteration count of 6 as an example, performing the above fusion process iteratively 6 times. Each iteration generates a more accurate fused lane feature map (refined query feature) based on the lane feature map obtained in the previous fusion. Finally, the final fused lane feature map (refined query feature) is decoded by a decoder to extract road information including road elements such as lane lines, lane center lines, and ground markings, as well as their corresponding attributes.

[0124] Figure 7 This is a schematic diagram of the structure of a road information determination device 700 provided in an embodiment of this disclosure. Figure 7 As shown, the road information determination device includes:

[0125] The first acquisition unit 710 is used to acquire multiple sensor data of the vehicle in the current frame, including image data and geospatial data.

[0126] The second acquisition unit 720 is used to determine the perceived lane feature map corresponding to the image data and the non-perceived lane feature map corresponding to the geospatial data based on the image data and geospatial data, respectively. The perceived lane feature map includes the surrounding environment features of the vehicle, and the non-perceived lane feature map includes the vehicle's beyond-line-of-sight map features and lane attribute features.

[0127] The fusion unit 730 is used to fuse the non-perceived lane feature map and the perceived lane feature map to obtain a fused lane feature map;

[0128] Decoding unit 740 is used to decode the fused lane feature map to obtain the road information of the fused lane feature map.

[0129] In some embodiments of this disclosure, the apparatus further includes a preprocessing unit, configured to perform distortion processing on the image data after acquiring multiple sensor data of the vehicle, to obtain distorted image data.

[0130] In some embodiments of this disclosure, the second acquisition unit 720 is configured to: extract an initial feature map corresponding to the image data using the backbone network in the convolutional neural network; and aggregate the surrounding environment features in the initial feature map using the first feature query information and the first attention mechanism to obtain a perceived lane feature map.

[0131] In some embodiments of this disclosure, geospatial data includes navigation data and local map data. The second acquisition unit 720 is configured to: encode the navigation data and local map data through a vector coding network to obtain a non-perceptive lane feature map. The non-perceptive lane feature map includes beyond-line-of-sight map features and lane attribute features, which are obtained by converting the navigation data and local map data.

[0132] In some embodiments of this disclosure, the fusion unit 730 is configured to: use a second attention mechanism to fuse the non-perceived lane feature map and the perceived lane feature map to obtain an initial fused lane feature map; and iteratively update the initial fused lane feature map according to a preset number of iterations to obtain the fused lane feature map.

[0133] In some embodiments of this disclosure, the second acquisition unit 720 is further configured to: after acquiring the perceived lane feature map corresponding to the image data and the non-perceived lane feature map corresponding to the geospatial data, acquire the fused lane feature map of the vehicle in the previous frame.

[0134] In some embodiments of this disclosure, the fusion unit 730 is further configured to: use a second attention mechanism to fuse the non-perceived lane feature map, the perceived lane feature map, and the fused lane feature map of the vehicle in the previous frame to obtain an initial fused lane feature map; and iteratively update the initial fused lane feature map according to a preset number of iterations to obtain the fused lane feature map.

[0135] In summary, the road information determination device acquires multiple sensor data from the vehicle's current frame, including image data and geospatial data. Based on the image data and geospatial data, it determines the perceived lane feature map corresponding to the image data and the non-perceived lane feature map corresponding to the geospatial data, respectively. The perceived lane feature map includes the vehicle's surrounding environment features, while the non-perceived lane feature map includes the vehicle's beyond-line-of-sight map features and lane attribute features. The non-perceived lane feature map and the perceived lane feature map are then fused to obtain a fused lane feature map. The fused lane feature map is then decoded to obtain the road information. This effectively solves the limitation of relying on manually set rules to handle long-tail problems in post-fusion schemes, and realizes the supplementation and fusion of beyond-line-of-sight map information and lane attribute information.

[0136] Corresponding to the methods provided in the above embodiments, this disclosure also provides a road information determination device. Since the device provided in this disclosure corresponds to the methods provided in the above embodiments, the implementation of the methods is also applicable to the device provided in this embodiment, and will not be described in detail in this embodiment.

[0137] The methods and apparatus provided in the embodiments of this application have been described above. To implement the functions of the methods provided in the embodiments of this application, the electronic device may include a hardware structure and software modules, and may implement the above functions in the form of a hardware structure, software modules, or a hardware structure plus software modules. One of the above functions may be executed in the form of a hardware structure, software modules, or a hardware structure plus software modules.

[0138] Figure 8 This is a block diagram illustrating an electronic device 800 for implementing the road information determination method described above, according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, computer, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0139] Reference Figure 8 The electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0140] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0141] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0142] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0143] Multimedia component 808 includes a screen that provides an output interface between electronic device 800 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When electronic device 800 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0144] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0145] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0146] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 can detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0147] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio), or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0148] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0149] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0150] Embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the road information determination method described in the above embodiments of this disclosure.

[0151] Embodiments of this disclosure also provide a computer program product, including a computer program that is executed by a processor using the road information determination method described in the above embodiments of this disclosure.

[0152] Embodiments of this disclosure also propose a chip, such as Figure 9 As shown, the chip includes one or more interface circuits 901 and one or more processors 902; the interface circuits are used to receive signals and send signals to the processors, the signals including computer instructions stored in the memory, and when the processor executes the computer instructions, the chip causes the chip to execute the road information determination method described in the above embodiments of this disclosure.

[0153] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0154] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0155] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0156] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (control method), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0157] It should be understood that various parts of the embodiments of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0158] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0159] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0160] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for determining road information, characterized in that, The method includes: Acquire multiple sensor data for the current frame of the vehicle, including image data and geospatial data; Based on the image data and the geospatial data, a perceptual lane feature map corresponding to the image data and a non-perceptual lane feature map corresponding to the geospatial data are determined respectively. The perceptual lane feature map includes the surrounding environment features of the vehicle, and the non-perceptual lane feature map includes the beyond-line-of-sight map features and lane attribute features of the vehicle. The non-perceived lane feature map and the perceived lane feature map are fused to obtain a fused lane feature map; The fused lane feature map is decoded to obtain the road information of the fused lane feature map.

2. The method according to claim 1, characterized in that, The method involves acquiring data from multiple vehicle sensors, followed by: The image data is subjected to distortion processing to obtain distorted image data.

3. The method according to any one of claims 1 to 2, characterized in that, The step of obtaining the perceived lane feature map corresponding to the image data includes: The initial feature map corresponding to the image data is extracted using the backbone network in the convolutional neural network; Using the first feature query information and the first attention mechanism, the surrounding environment features in the initial feature map are aggregated to obtain the perceived lane feature map.

4. The method according to claim 1, characterized in that, The geospatial data includes navigation data and local map data. The step of obtaining the non-perceptive lane feature map corresponding to the geospatial data includes: The navigation data and the local map data are encoded by a vector coding network to obtain the non-perceptual lane feature map. The non-perceptual lane feature map includes beyond-line-of-sight map features and lane attribute features, which are obtained by converting the navigation data and the local map data.

5. The method according to claim 1, characterized in that, The step of fusing the non-perceived lane feature map and the perceived lane feature map to obtain a fused lane feature map includes: Using a second attention mechanism, the non-perceptual lane feature map and the perceived lane feature map are fused to obtain an initial lane feature map; The initial lane feature map is iteratively updated according to a preset number of iterations to obtain the fused lane feature map.

6. The method according to claim 1, characterized in that, After acquiring the perceived lane feature map corresponding to the image data and the non-perceived lane feature map corresponding to the geospatial data, the method includes: Obtain the fused lane feature map of the vehicle in the previous frame.

7. The method according to claim 6, characterized in that, The step of fusing the non-perceived lane feature map and the perceived lane feature map to obtain a fused lane feature map includes: Using a second attention mechanism, the non-perceptual lane feature map, the perceived lane feature map, and the fused lane feature map of the vehicle in the previous frame are fused to obtain an initial lane feature map. The initial lane feature map is iteratively updated according to a preset number of iterations to obtain the fused lane feature map.

8. A road information determination device, characterized in that, The device includes: The first acquisition unit is used to acquire multiple sensor data of the vehicle in the current frame, the sensor data including image data and geospatial data; The second acquisition unit is used to determine, based on the image data and the geospatial data, a perceived lane feature map corresponding to the image data and a non-perceived lane feature map corresponding to the geospatial data, respectively. The perceived lane feature map includes the surrounding environment features of the vehicle, and the non-perceived lane feature map includes the vehicle's beyond-line-of-sight map features and lane attribute features. The fusion unit is used to fuse the non-perceived lane feature map and the perceived lane feature map to obtain a fused lane feature map; The decoding unit is used to decode the fused lane feature map to obtain the road information of the fused lane feature map.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 7.

Images