Road topology detection method, and related apparatus

Abstract

A road topology detection method, which can be applied to the field of artificial intelligence and the field of autonomous driving. The method comprises: acquiring target information, wherein the target information is data collected regarding a road condition of a target road; on the basis of the target information, obtaining sequence information by means of a deep learning model, wherein the sequence information comprises point information of feature points in a topology of the target road and edge information of feature lines in the topology of the target road, the point information comprises physical positions and categories corresponding the feature points, and the edge information comprises a connection relationship and connection direction between the feature points; and determining the topology of the target road on the basis of the sequence information. In the present application, by means of directly predicting information of nodes (feature points) and information of edges (feature lines) in a stage, accumulative errors of an overall road structure caused by an error of each stage are avoided, and by means of fusing data of the nodes and data of the edges, mutual promotion is achieved in a network, thereby increasing the precision of a final result.

Description

A road topology detection method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 21, 2023, with application number 202310911900.0 and invention name “A road topology detection method and related device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence, and in particular to a road topology detection method and related devices. Background Art

[0003] Artificial intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0004] The first step in autonomous driving is the collection and processing of environmental and in-vehicle information. Obtaining environmental information around the vehicle is the foundation and prerequisite for autonomous driving for intelligent vehicles. This involves technologies such as road structure recognition, vehicle detection, and pedestrian detection. As one of the most crucial pieces of road surface information, road structure effectively guides intelligent vehicles along compliant routes. The road network, consisting of lane centerlines, key road points, and centerline connections, fully represents road structure information and is a crucial form of representation. Real-time road network detection is a crucial component of intelligent vehicle assisted driving systems. This technology facilitates route planning, provides lane deviation warnings, and provides a reference for precise navigation.

[0005] Road network prediction is a complex task, requiring holistic scene perception and reasoning to extract three pieces of information: lane centerlines, key road points, and centerline connectivity. This information includes Euclidean features such as the shape and position of centerlines and key points, as well as non-Euclidean features such as road topology. Accurately predicting both types of information presents significant challenges.

[0006] Existing technologies first perform centerline detection and then predict the connectivity between centerlines. These two predictions are separate and unconnected. This two-stage prediction process can lead to information loss and error accumulation. For example, deviations in centerline detection and prediction can lead to inaccurate connectivity predictions, resulting in poor road topology prediction accuracy.

[0007] Summary of the Invention

[0008] This application provides a road topology detection method that can improve the prediction accuracy of road topology.

[0009] In the first aspect, the present application provides a road topology detection method, comprising: obtaining data collected on the road conditions of a target road (that is, target information), processing the target information through a deep learning model, and obtaining sequence information that can uniquely determine the topology of the target road, wherein the sequence information is represented as point information of feature points in the topology of the target road, and edge information of feature lines in the topology of the target road; the point information includes the physical location and category corresponding to the feature points, and the edge information includes the connection relationship and connection direction between the feature points, and the topology of the target road is determined based on the sequence information.

[0010] Among them, the deep learning model can be a language model or other deep learning models, as well as models formed by combinations of these models such as series, parallel, and cascade.

[0011] Existing technologies typically perform centerline detection first, followed by prediction or post-processing of centerline connectivity. These two predictions are separate and unconnected. On the one hand, this two-stage prediction process can lead to information loss and error accumulation. For example, deviations in centerline detection predictions can lead to even more inaccurate connectivity predictions. On the other hand, this relatively separate two-stage prediction process fails to integrate the two types of data, which would allow the model to better perceive the overall road network. The position and shape of centerlines contribute to the prediction of centerline connectivity, while the connectivity between centerlines also contributes to the prediction of position and shape. Existing technologies fail to implement the interaction between these two types of information, limiting model performance.

[0012] In the embodiment of the present application, by directly predicting nodes (feature points) and edges (feature lines) in a single stage, the accumulated errors in the overall road structure caused by errors in each stage are avoided. Moreover, the fusion of the two data can promote each other in the network and increase the accuracy of the final result. And due to the change in the prediction mechanism, the embodiment of the present application can use the road topology alone for pre-training without the original data, which is also impossible for previous solutions.

[0013] In one possible implementation, the feature point is a road key point, a lane line key point, a lane centerline key point, or a topological derivative point of the road in the target road, and the feature line is a lane line segment, a lane centerline segment, a road segment, or a traffic entity.

[0014] In a possible implementation, the side information further includes: shape features of the feature line.

[0015] In a possible implementation, the topology information further includes: an arrangement order of the feature points, where the arrangement order is consistent with an arrangement obtained by performing a topological sorting method specified in graph theory on the feature points.

[0016] In a possible implementation, the topological sorting method is a depth-first traversal, a breadth-first traversal, or a topological sorting method obtained through network learning.

[0017] In one possible implementation, the category is one of the following:

[0018] Starting node, relay node, fork node and merge node.

[0019] In one possible implementation, obtaining sequence information through a deep learning model based on the target information includes: obtaining first sequence information and the confidence of each basic unit in the first sequence information through a deep learning model based on the target information and a first mask result; the first mask result is a mask result of an output of an intermediate iterative process of the language model; masking the basic units in the first sequence information whose confidence is less than a threshold to obtain a second mask result; obtaining second sequence information through a deep learning model based on the target information and the second mask result; the second sequence information is used to obtain the sequence information.

[0020] During the inference phase, the initial sequence with all the masked padded characters can be input into the language model, allowing the language model to predict the tokens covered by the mask. Of course, during the first prediction, the entire sequence faced by the language model is masked, and it may not be realistic to predict all tokens at once. Therefore, after the first prediction, a screening is performed to re-mask the low-confidence tokens while retaining the high-confidence tokens. In this way, after multiple predictions, the new sequence will have increasingly higher accuracy relative to the obtained accuracy. The number of times here is actually much smaller than the entire sequence length. In short, using this mechanism, only a few (less than 0.1 times the sequence length) are needed to fully predict the sequence, thereby significantly reducing inference time.

[0021] In a possible implementation, the target information is image data or radar data, and pose information of collecting the image data or radar data.

[0022] In a second aspect, the present application provides a model training method, the method comprising:

[0023] Acquire sequence information, the sequence information including point information of a first feature point and a second feature point in the topology of a target road, and side information of a first feature line and a second feature line in the topology of the target road; the point information including physical locations and categories corresponding to the first feature point and the second feature point, and the side information including a connection relationship and a connection direction between the first feature point and the second feature point;

[0024] Determine, by a deep learning model, a predicted value of the point information of the second feature point and the side information of the second feature line based on the point information of the first feature point and the side information of the first feature line;

[0025] The language model is updated according to the predicted value, the point information of the second feature point, and the side information of the second feature line.

[0026] In a possible implementation, the sequence information is determined through image data, radar data, or map data.

[0027] In a third aspect, the present application provides a data processing device, comprising:

[0028] An acquisition module, configured to acquire target information, wherein the target information is data collected on the road condition of a target road;

[0029] a processing module configured to obtain sequence information using a deep learning model based on the target information, the sequence information including point information of feature points in the topology of the target road and side information of feature lines in the topology of the target road; the point information including physical locations and categories corresponding to the feature points, and the side information including connection relationships and connection directions between the feature points;

[0030] The topology of the target road is determined according to the sequence information.

[0031] In one possible implementation, the feature point is a road key point, a lane line key point, a lane centerline key point, or a topological derivative point of the road in the target road, and the feature line is a lane line segment, a lane centerline segment, a road segment, or a traffic entity.

[0032] In a possible implementation, the side information further includes: shape features of the feature line.

[0033] In a possible implementation, the topology information further includes: an arrangement order of the feature points, where the arrangement order is consistent with an arrangement obtained by performing a topological sorting method specified in graph theory on the feature points.

[0034] In a possible implementation, the topological sorting method is a depth-first traversal, a breadth-first traversal, or a topological sorting method obtained through network learning.

[0035] In one possible implementation, the category is one of the following:

[0036] Starting node, relay node, fork node and merge node.

[0037] In a possible implementation, the processing module is specifically configured to:

[0038] Obtaining, through a deep learning model, first sequence information and the confidence of each basic unit in the first sequence information based on the target information and the first mask result; the first mask result is a mask result outputted from an intermediate iterative process of the language model;

[0039] Masking the basic units in the first sequence information whose confidence is less than a threshold to obtain a second masking result;

[0040] According to the target information and the second mask result, second sequence information is obtained through a deep learning model; the second sequence information is used to obtain the sequence information.

[0041] In a fourth aspect, the present application provides a model training device, the device comprising:

[0042] an acquisition module, configured to acquire sequence information, the sequence information including point information of a first feature point and a second feature point in the topology of a target road, and side information of a first feature line and a second feature line in the topology of the target road; the point information including the physical locations and categories corresponding to the first feature point and the second feature point, and the side information including the connection relationship and connection direction between the first feature point and the second feature point;

[0043] A processing module, configured to determine, based on the point information of the first feature point and the side information of the first feature line, a predicted value of the point information of the second feature point and the side information of the second feature line through a deep learning model;

[0044] An updating module is used to update the language model according to the predicted value, the point information of the second feature point and the side information of the second feature line.

[0045] In a possible implementation, the sequence information is determined through image data, radar data, or map data.

[0046] In the fifth aspect, an embodiment of the present application provides a data processing device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to execute any optional method such as the first aspect above, and any optional method such as the second aspect above.

[0047] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect and any optional method, as well as any optional method of the above-mentioned second aspect.

[0048] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising code, which, when executed, is used to implement the above-mentioned first aspect and any optional method, as well as any optional method of the above-mentioned second aspect.

[0049] In an eighth aspect, the present application provides a chip system comprising a processor configured to support a data processing device in implementing the functions described in the aforementioned aspects, such as transmitting or processing data or information described in the aforementioned methods. In one possible design, the chip system further comprises a memory configured to store program instructions and data necessary for executing or training the device. The chip system may consist solely of a chip or may include a chip and other discrete components. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a schematic diagram of the structure of the artificial intelligence main framework;

[0051] FIG2 is a schematic diagram of a road topology provided in an embodiment of the present application;

[0052] FIG3 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0053] FIG4 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0054] FIG5 is a schematic diagram of a system architecture provided in an embodiment of the present application;

[0055] FIG6 is a schematic diagram of a system architecture provided in an embodiment of the present application;

[0056] FIG7 is a schematic diagram of a cloud service provided in an embodiment of the present application;

[0057] FIG8 is a flow chart of a data processing method provided in an embodiment of the present application;

[0058] FIG9 is a flow chart of a data processing method provided in an embodiment of the present application;

[0059] 10A to 10D are schematic diagrams of an encoding process provided in an embodiment of the present application;

[0060] FIG11 is a schematic diagram of a method flow chart provided in an embodiment of the present application;

[0061] FIG12 is a schematic diagram of a network structure provided in an embodiment of the present application;

[0062] FIG13 is a schematic diagram of a network structure provided in an embodiment of the present application;

[0063] FIG14 is a schematic diagram of a road topology prediction result provided by an embodiment of the present application;

[0064] FIG15 is a schematic diagram of a network structure provided in an embodiment of the present application;

[0065] FIG16 is a schematic diagram of a method flow chart provided in an embodiment of the present application;

[0066] FIG17 is a schematic diagram of a method flow chart provided in an embodiment of the present application;

[0067] FIG18 is a schematic diagram of a method flow chart provided in an embodiment of the present application;

[0068] FIG19 is a diagram illustrating the beneficial effects of an embodiment of the present application;

[0069] FIG20 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0070] FIG21 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;

[0071] FIG22 is a schematic diagram of a terminal device provided in an embodiment of the present application;

[0072] FIG23 is a schematic diagram of a server provided in an embodiment of the present application;

[0073] Figure 24 is a schematic diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0075] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0076] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0077] First, let's describe the overall workflow of an AI system. See Figure 1, which shows a schematic diagram of the main AI framework. This framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0078] (1) Infrastructure

[0079] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0080] (2) Data

[0081] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0082] (3) Data processing

[0083] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0084] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0085] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0086] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0087] (4) General ability

[0088] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0089] (5) Smart products and industry applications

[0090] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0091] The embodiments of the present application can be applied to the training optimization of the model, which is specifically reflected in the construction of training sample data. When the embodiments of the present application perform true value annotation on the training sample data, the training sample data is annotated with the true value set, and then the training samples annotated with the true value set are used to train the model. The model optimized and trained by this application can be specifically applied to road topology generation in the field of artificial intelligence.

[0092] Road topology generation, also known as road network prediction, is a complex task requiring holistic scene perception and reasoning to extract three pieces of information: lane centerlines, key road points, and centerline connectivity. This information includes Euclidean information, such as the shape and position of centerlines and key points, as well as non-Euclidean information, such as the road topology. Accurately predicting both types of information presents significant challenges. Limited work has been conducted in this area. Early work used aerial images as input, and the predicted road network structure was relatively crude. This aerial image-based approach cannot be directly applied to on-board cameras and does not meet the requirements for accurate online mapping using on-board cameras for autonomous driving. Recent work has used RGB images captured by a front-mounted camera as input to predict road networks. While this meets the basic requirements of autonomous driving, the prediction accuracy leaves much room for improvement. The road network detection algorithm of the present invention uses images captured by on-board cameras as visual input and obtains prediction results through a neural network. The input and output visualization is shown in Figure 2. Compared to previous work, the algorithm in the present embodiment not only meets the requirements for online mapping using on-board cameras for autonomous driving, but also further improves the accuracy of road network prediction.

[0093] For example, the embodiments of the present application can be applied to scenarios such as autonomous driving or AR navigation.

[0094] Scenario 1: Autonomous Driving

[0095] With economic development, the number of cars worldwide continues to increase, and the incidence of traffic accidents has also increased significantly, posing a huge threat to people's lives and property. The field of autonomous driving is gaining increasing attention. Human error is the main cause of traffic accidents, and how to reduce human error is a key issue in improving driving safety.

[0096] As shown in Figure 3, in autonomous driving, road topology detection (including lane detection, for example) is crucial for accurate positioning of autonomous vehicles and its impact on driving decisions. As one of the most crucial pieces of road surface information, road structure effectively guides intelligent vehicles along compliant routes. The road network, consisting of lane centerlines, key road points, and centerline connections, fully represents road structure information and is a crucial form of road structure representation. Real-time road network detection is a crucial component of intelligent vehicle assisted driving systems. This technology facilitates route planning, provides lane deviation warnings, and provides a reference for precise navigation.

[0097] Scenario 2: AR Navigation

[0098] As the number of cars continues to grow, people's mobility is also expanding. Relying solely on experience to reach their destination is becoming less common, and people are increasingly relying on navigation for travel. Traditional navigation systems project the vehicle's location onto a map based on GPS signals. The navigation software then plans a route based on the set destination, guiding the user to the destination through screen displays and voice announcements. This presentation method requires the driver to connect the map guidance and voice announcements with the real world around them to understand the specific meaning of the guidance. If a driver encounters a fork in the road and fails to listen carefully to the navigation announcement, they can easily miss the opportunity to change lanes. [The following appears to be unrelated and likely a separate topic:]

[0099] In AR navigation (as shown in Figure 4), the integration of navigation information and road network detection results enables the system to plan the vehicle's route based on the vehicle's location and road network detection results. This optimizes the presentation of guidance information and reduces the user's reception and comprehension costs. What you see is what you get; users only need to act according to the navigation information. Accurate road network information, combined with reasonable navigation information, can effectively guide intelligent vehicles within restricted road areas, improving the user experience of current in-vehicle navigation software.

[0100] It should be noted that in the embodiments of the present application, there is no limitation on the specific type of the model used for road topology generation (which may be referred to as a language model in this application). For example, the target model described in this application may be a transformer-based model or a model based on other architectures, which is not specifically limited here.

[0101] After determining the target model for road topology generation, the target model needs to be trained. Only after the trained target model is applied to the specific road topology generation task can it be applied. Specifically, the entire target model training and road topology generation process can be introduced based on the image processing system 200 shown in Figure 5. In Figure 5, the image processing system 200 includes an execution device 210, a training device 220, a database 230, a client device 240, a data storage system 250, and a data acquisition device 260. The execution device 210 includes a computing module 211, which is essentially the trained target model 201 provided in the embodiments of the present application.

[0102] Among them, the data acquisition device 260 (such as a camera mounted on a wheeled mobile device) is used to obtain the open source large-scale data set (i.e., training set) required by the user, and store the data set in the database 230. The training device 220 trains the target model 201 based on the data set maintained in the database 230. It should be noted here that in the embodiment of the present application, it is necessary to perform multiple types of true value annotations on the data set maintained in the database 230. Specifically, each training sample used for the training device 220 must be annotated with a true value. The true value includes the point information of each feature point in the road topology and the line information of the feature line. Based on the annotated true value, the corresponding road topology can be uniquely determined. Conversely, the unique annotated true value can also be determined based on the road topology. Afterwards, these training samples annotated with true values ​​are used to train the target model 201.

[0103] The execution device 210 can call data, code, etc. from the data storage system 250, or store data, instructions, etc. in the data storage system 250. The data storage system 250 can be located in the execution device 210, or the data storage system 250 can be an external memory relative to the execution device 210.

[0104] The target model 201 trained by the training device 220 can be applied to different systems or devices (i.e., the execution device 210). For example, the execution device 210 can be various wheeled mobile devices (wheeled construction equipment, autonomous driving vehicles, assisted driving vehicles, etc.). Autonomous driving vehicles can also be cars, trucks, motorcycles, buses, ships, airplanes, helicopters, lawn mowers, recreational vehicles, amusement park vehicles, construction equipment, trams, golf carts, trains, and carts, etc.

[0105] Optionally, the execution device 210 may also be a device with AR and MR capabilities, such as AR glasses.

[0106] In FIG. 5 , the execution device 210 is configured with an I / O interface 212 for data interaction with external devices. A “user” can input data into the I / O interface 212 through a client device 240 . For example, the client device 240 can be a camera or radar mounted on an autonomous vehicle (or the aforementioned device with AR or MR capabilities). The image captured by the camera or the data collected by the radar is input as input data to the computing module 211 of the execution device 210. The target model 201 in the computing module 211 processes the input image (optionally, radar data) to obtain the road topology information, and then outputs the road topology information to the downstream module (such as the autonomous driving device) or directly displays it on the display interface of the execution device 210 (if any). In addition, in some embodiments of the present application, the client device 240 can also be integrated into the execution device 210. For example, when the execution device 210 is an autonomous vehicle, it can directly capture images through the camera of the autonomous vehicle or receive images sent by other devices (such as mobile phones). The computing module 211 in the autonomous vehicle then determines the road topology of the image and obtains the result, and directly presents the result on the vehicle display interface or transmits it to the downstream task module. The product form of the execution device 210 and the client device 240 is not limited here.

[0107] It is worth noting that Figure 5 is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 5, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 can also be placed in the execution device 210; in Figure 5, the client device 240 is an external device relative to the execution device 210. In other cases, the client device 240 can also be integrated into the execution device 210.

[0108] It should also be noted that, in some embodiments of the present application, the target model 201 can also be split into multiple sub-modules / sub-units to jointly implement the solution provided by the embodiments of the present application, which is not specifically limited here.

[0109] It should also be noted that the training of the target model 201 described in the above embodiment can be implemented on the cloud side. For example, the training device 220 on the cloud side (the training device 220 can be set on one or more servers or virtual machines) can obtain a training set (the training samples included in the training set have been annotated with true values), and train the target model based on the training samples in the training set to obtain a trained target model 201. After that, the trained target model 201 is sent to the execution device 210 for application, for example, to the relevant systems of the autonomous driving vehicle (such as ADAS, ADS, etc.). For example, as described in the system architecture corresponding to Figure 5, the training device 22 0 The target model 201 is trained as a whole, and the trained target model 201 is then sent to the execution device 210 for use; the training of the target model 201 described in the above embodiment can also be implemented on the terminal side, that is, the training device 220 can be located on the terminal side. For example, a wheeled mobile device (such as an autonomous driving vehicle, an assisted driving vehicle, etc.) can obtain a training set (the training samples included in the training set have been annotated with true values), and train the target model 201 based on the training samples in the training set to obtain the trained target model 201. The trained target model 201 can be used directly on the terminal device, or it can be sent by the terminal device to other devices for use. The specific embodiment of the present application does not limit on which device (cloud side or terminal side) the target model 201 is trained or applied.

[0110] Next, the following description will be made by taking the case where the execution device 210 is deployed in a wheeled mobile device as an example:

[0111] It should be noted that since the trained target model 201 described in FIG5 (or the execution device 210 described in FIG5) can be deployed in various wheeled mobile devices, it is used to process the relevant perception information (such as video, image or radar data) on the road surface captured by the camera or radar carried by the wheeled mobile device and output the road topology. The road topology is input to the downstream module of the wheeled mobile device for further processing. The deployment location of the trained target model and the overall architecture of the autonomous driving vehicle are described below using the wheeled mobile device as an example of an autonomous driving vehicle. Please refer to FIG6 for details. FIG6 illustrates a top-down hierarchical system architecture. There may be defined interfaces between the systems for transmitting data between the systems to ensure the real-time and integrity of the data. The following is a brief introduction to each system:

[0112] (1) Environmental Perception System

[0113] Environmental perception is the most basic part of autonomous vehicles. Whether it is making driving behavior decisions or global path planning, it needs to be based on environmental perception. According to the real-time perception results of the road traffic environment, corresponding judgments, decisions and plans are made to enable the vehicle to achieve intelligent driving.

[0114] The environmental perception system mainly uses various sensors to obtain relevant environmental information, thereby completing the construction of the environmental model and the expression of knowledge about the traffic scene. The sensors used include one or more cameras, single-line radar (SICK), four-line radar (IBEO), three-dimensional lidar (HDL-64E), etc. Among them, the video or image collected by the camera is mainly responsible for traffic light detection, lane line detection, road sign detection, vehicle recognition, etc.; other lidar sensors are mainly responsible for the detection, identification and tracking of dynamic and static key obstacles, as well as the detection and extraction of non-key obstacles such as road boundaries, shrub belts, and surrounding buildings. In the embodiment of the present application, the perception data related to the road surface captured by the camera is used, that is, the image to be detected of the road surface. The image to be detected (it can be an image directly captured by the camera or an image extracted from a video captured by the camera, and the image to be detected is not limited here) is input into the trained target model constructed in the embodiment of the present application. After detection by the trained target model, the prediction result of the road topology in the image to be detected is output. The prediction result can be directly used as a downstream module of the autonomous driving vehicle (such as an autonomous decision-making system) for further processing, or it can be input into other models in the environmental perception system (such as a map construction model, not shown in Figure 5) for jointly constructing an occupancy grid map (OGM) that can express the road environment with other perception information, and finally the OGM is sent to the autonomous decision-making system for further decision-making and planning.

[0115] (2) Autonomous decision-making system

[0116] The autonomous decision-making system is a key component of autonomous vehicles. The system is mainly divided into two core subsystems: behavioral decision-making and motion planning. Among them, the behavioral decision-making subsystem mainly obtains the global optimal driving route by running the global planning layer to clarify the specific driving task, and then decides on reasonable driving behavior based on the current real-time road information sent by the environmental perception system, road traffic rules and driving experience, and sends the driving behavior instructions to the motion planning subsystem; the motion planning subsystem plans a feasible driving trajectory based on the received driving behavior instructions and the current local environmental perception information, based on indicators such as safety and stability, and sends it to the control system.

[0117] (3) Control system

[0118] Specifically, the control system is divided into two parts: the control subsystem and the execution subsystem. The control subsystem is used to convert the feasible driving trajectory generated by the autonomous decision-making system into specific execution instructions for each execution module and pass them to the execution subsystem; the execution subsystem receives the execution instructions from the control subsystem and sends them to each control object to reasonably control the vehicle's steering, braking, throttle, gear, etc., so that the vehicle can drive automatically to complete the corresponding driving operations.

[0119] It should be noted that during the driving process of an autonomous vehicle, the accuracy of the driving operation of the autonomous vehicle mainly depends on whether the specific execution instructions generated by the control system for each execution module are accurate, and the accuracy depends on the autonomous decision-making system. The autonomous decision-making system is faced with uncertainty. The uncertainty factors mainly include the following aspects: 1) The uncertainty caused by the characteristics and calibration errors of each sensor in the environmental perception system. The perception mechanism, perception range and corresponding error mode of different sensors are different, and the calibration error caused by their installation on the autonomous vehicle will eventually be reflected in the uncertainty of the perception information; 2) The uncertainty caused by the data processing delay of the environmental perception system. This is because the road environment is complex and the road environment is complex. The environment is complex and the amount of data information is huge, which makes the computational complexity of data processing in the environmental perception system large. The entire environment is constantly changing, which will inevitably lead to data information delays, thereby affecting the correct judgment of the autonomous decision-making system; 3) Different ways of processing perception information will also bring uncertainty. Taking the embodiments of the present application as an example, if the traditional regression-based or segmentation-based lane line detection method is adopted, the lane line detection accuracy will be low. The existing lane line detection method cannot meet the increasingly demanding application scenarios of autonomous driving vehicles. If the lane line detection accuracy can be improved, the uncertainty of the autonomous decision-making system can be reduced accordingly, thereby improving the accuracy of the specific execution instructions generated by the control system for each execution module.

[0120] It should also be noted that the overall architecture of the autonomous driving vehicle shown in Figure 5 is for illustration only. In actual applications, it may include more or fewer systems / subsystems or modules, and each system / subsystem or module may include multiple components, which are not specifically limited here.

[0121] In addition, a product form of the embodiment of the present application may also be a road topology determination cloud service provided by a server:

[0122] In a possible implementation, the server may provide a road topology determination service to the terminal side through an application programming interface (API).

[0123] Among them, the terminal device can send relevant parameters (such as images (for example, surround view images), radar data, etc.) to the server through the API provided by the cloud. The server can obtain processing results based on the received parameters, etc., and return the processing results to the terminal.

[0124] The description of the terminal and the server can be the same as that of the above embodiments, and will not be repeated here.

[0125] FIG7 shows a process of using a road topology determination function cloud service provided by a cloud platform.

[0126] 1. Activate and purchase content review services.

[0127] 2. Users can download the software development kit (SDK) corresponding to the content review service. Usually, the cloud platform provides multiple development versions of the SDK for users to choose according to the requirements of the development environment, such as JAVA version SDK, Python version SDK, PHP version SDK, Android version SDK, etc.

[0128] 3. After the user downloads the corresponding version of the SDK to the local computer as needed, they import the SDK project into the local development environment, configure and debug it in the local development environment, and develop other functions in the local development environment, forming an application that integrates road topology determination functional capabilities.

[0129] 4. When a road topology determination application is used and needs to perform road topology determination, it can trigger an API call for this function. When the application triggers this function, it initiates an API request to a running instance of the road topology determination service in the cloud environment. The API request includes an image, which is then processed by the running instance in the cloud environment to obtain a result.

[0130] 5. The cloud environment returns the processing results to the application, thereby completing a road topology determination function call.

[0131] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0132] (1) Neural Network

[0133] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input. The output of the operation unit can be:

[0134] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0135] (2) Natural language processing (NLP)

[0136] Natural language refers to human language, and natural language processing (NLP) is the processing of human language. Natural language processing is the process of systematically analyzing, understanding, and extracting information from text data in an intelligent and efficient manner. By using NLP and its components, we can manage very large amounts of text data, perform a large number of automated tasks, and solve a wide variety of problems, such as automatic summarization, machine translation (MT), named entity recognition (NER), relation extraction (RE), information extraction (IE), sentiment analysis, speech recognition, question answering, and topic segmentation.

[0137] (3) Deep Neural Networks

[0138] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. Based on the position of different layers in DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0139] (4) Prompt

[0140] A natural language term that can be categorized into hard templates and soft templates. Hard templates are generally natural language words or sentences with a specific meaning; soft templates are generally parameterized representation vectors without any meaning.

[0141] The first step in autonomous driving is the collection and processing of environmental and in-vehicle information. Obtaining environmental information around the vehicle is the foundation and prerequisite for autonomous driving for intelligent vehicles. This involves technologies such as road structure recognition, vehicle detection, and pedestrian detection. As one of the most crucial pieces of road surface information, road structure effectively guides intelligent vehicles along compliant routes. The road network, consisting of lane centerlines, key road points, and centerline connections, fully represents road structure information and is a crucial form of representation. Real-time road network detection is a crucial component of intelligent vehicle assisted driving systems. This technology facilitates route planning, provides lane deviation warnings, and provides a reference for precise navigation.

[0142] Road network prediction is a complex task, requiring holistic scene perception and reasoning to extract three pieces of information: lane centerlines, key road points, and centerline connectivity. This information includes Euclidean features such as the shape and position of centerlines and key points, as well as non-Euclidean features such as road topology. Accurately predicting both types of information presents significant challenges.

[0143] In existing technologies, centerline detection is performed first, followed by prediction of the connections between centerlines. These two predictions are performed separately and not linked. This two-stage prediction process can lead to information loss and error accumulation. For example, deviations in centerline detection predictions can lead to inaccurate predictions of connectivity relationships, resulting in poor road topology prediction accuracy.

[0144] In combination with the above description, the embodiments of the present application provide a model training method and a road topology prediction method, which can be applied to the training stage and reasoning stage of the model, respectively, and are described below.

[0145] 1. Training Phase

[0146] In the embodiment of the present application, the training phase is the process in which the training device 220 in FIG5 performs a training operation on the target model 201 using the training samples in the training set. Specifically, please refer to FIG8 , which is a flow chart of a model training method provided in the embodiment of the present application, which may specifically include:

[0147] 801. Obtain sequence information, where the sequence information includes point information of a first feature point and a second feature point in the topology of a target road, and side information of a first feature line and a second feature line in the topology of the target road; the point information includes the physical locations and categories corresponding to the first feature point and the second feature point, and the side information includes the connection relationship and connection direction between the first feature point and the second feature point.

[0148] Among them, the training phase is the stage in which the model (for example, the language model) adjusts its own weights through backpropagation based on the labeled data. The training quality in the training phase determines the inference accuracy in the subsequent inference phase.

[0149] The embodiment corresponding to FIG8 may be a pre-training process of the model, or a real training process (that is, a process of fine-tuning the model based on real collected data).

[0150] In an embodiment of the present application, the sequence information may be a training sample. That is, the training sample for the language model may include sequence information obtained by encoding sensory data collected by the sensor about road conditions. By training the language model, the language model can be enabled to generate accurate road topology information based on the sensory data collected. The so-called accurate road topology information here can be understood as the ability to uniquely determine the corresponding accurate road topology based on the road topology information, and vice versa.

[0151] In order to enable the model to have the above capabilities, it is necessary to design the true value in the training sample (that is, sequence information, that is, road topology information, or encoded information of road topology) accordingly.

[0152] Next, the sequence information in the examples of this application is introduced.

[0153] A road network can be composed of road feature points (for example, stop lines, bifurcations, and merge points) and feature lines. By using feature lines to connect the feature points in an orderly manner, a road network can be represented (that is, road topology. It should be understood that the network here is based on the concept in computer graph theory, where the network refers to a dense graph). Considering that each lane is always one-way, the road network can be further abstracted into a directed acylic graph (DAG), which can be represented by the following formula G = (V, E), where the vertex set V is the set of all road key points, and the edge set E is the set of center lines connecting all key points. Among them, the points in the directed graph (or cloned points based on the points in the directed graph) can be feature points in the embodiments of the present application, and the lines in the directed graph can be feature lines in the embodiments of the present application.

[0154] Therefore, the road topology can be accurately represented by road feature points and feature lines.

[0155] Among them, the feature points can be key points on the road, and the key points can be physical points, such as stop lines, bifurcation points, and merging points, or they can be virtual markers, such as lane line segments, lane centerline segments, and road segments.

[0156] The feature line may be a connecting line between feature points, or may be a physical lane line, a center line, or a traffic entity (such as a traffic light).

[0157] In one possible implementation, the sequence information may include point information of multiple feature points and line information of multiple feature lines in the topology of the target road. In the embodiment of the present application, the sequence information can uniquely represent the road topology through the point information of the feature points and the line information of the feature lines.

[0158] The point information includes the physical location and category of the feature point, and the edge information may include the connection relationship and connection direction between the feature points.

[0159] For example, the physical position of the feature point may be the coordinates of the feature point (e.g., the geometric position in the BEV plane);

[0160] The connection relationship between the feature points can indicate which two endpoints of the corresponding feature line are, and the point information of the two feature points based on the endpoints can indicate the physical position of the feature line. Usually, lane lines are directional, so the line information can also include the connection direction between the feature points. The direction can be represented by which of the two endpoints is the starting point and which is the end point (the direction is from the starting point to the end point).

[0161] For example, the category of the feature point can be one of the following:

[0162] A parent node (also called an Ancestor), a child node of the parent node and a node adjacent to the parent node (also called a Lineal), a child node of the parent node and a node not adjacent to the parent node (also called an Offshoot), and a clone node.

[0163] Vertices are categorized into four types: Ancestor, Lineal, Offshoot, and Clone. The distinction between these two types is that when a parent node has multiple children, the first child traversed during topological sorting is marked as Lineal, while the remaining children are marked as Offshoot. This classification facilitates recovery from language sequences to RoadNet.

[0164] For example, the category can also be: starting point, end point, separation point or merging point.

[0165] In one possible implementation, each vertex V = (vx, vy, vc)∈v contains two properties:

[0166] 1) Physical position, for example, can be represented by physical coordinates (vx∈R, vy∈R), which optionally specifically represents the geometric position of the feature point in the BEV plane,

[0167] 2) Category, i.e. Vc. The partition vertices can be divided into four categories: {starting point, end point, separation point, and merging point}, each category represents the specific road topology represented by this point.

[0168] Each edge e = (es, et, epx, epy)∈e can contain two properties:

[0169] 1) The connection relationship between feature points, for example, can be represented by source nodes and target nodes, the vertices of the edge between source es∈V and target et∈V; that is, which two feature points are connected by the edge;

[0170] 2) Shape features, for example, the shape of the feature line can be represented by the middle control points (epx∈R, epy∈R) of the Bezier curve.

[0171] Among them, the Euclidean G data can include vx, vy, epx, epy, and non-Euclidean data can include vc, es, etc. The RoadNet sequence proposed in the embodiment of the present application is a unified representation method for Euclidean data and non-Euclidean data. After making a compromise between accuracy and speed, the sequence encoding method (that is, the method of obtaining sequence information) in the embodiment of the present application can be, but is not limited to, the following two schematic diagrams:

[0172] 1. AR encoding method:

[0173] 10A , the construction from road topology to sequence information may include three steps: 1. From DAG to directed forest; 2. Topological sorting of directed forest; 3. Topological sequence construction.

[0174] Among them, from DAG to directed forest. A typical DAG has a quantitative fuzzy relationship between vertices and edges. This makes it possible to retain all edges and vertices without redundant edge directions caused by reusing vertices when indicating them. In contrast, the directed forest structure has clear relationships between them. This avoids this redundancy. Therefore, the DAG can be transformed into a directed forest Gf = (Vf, Ef) by adding clone nodes. This transformation has the advantage of being able to find bijections between Ef and Vf {v∈Vf|id(v)=0} where id(v) represents the vertex's degree of introduction.

[0175] Among them, the topological sorting of a directed forest. For a single directed tree in a directed forest, graph theory sorting methods, such as depth-first (or breadth-first) traversal, are used to obtain the node sequence in the language. For simplicity, the cloned node is often placed after the original node during traversal, which is very convenient when the code has been error-checked. At the same time, the topological sequence is restored to the original state.

[0176] Among them, the construction of the topological sequence. The final sequence is composed of ordered point-edge pairs. Each point-edge pair consists of 6 integers (of course, the various variants of this algorithm are not limited to 6 integers, it can be 4 integers or more), where these 6 integers represent: {the x-coordinate of the point, the y-coordinate of the point, the type of the point, the position of the parent node of this point in the topological sequence, the Bezier parameter 1 of the edge, the Bezier parameter 2 of the edge}. The classification of points is divided into four types {Ancestor, Lineal, Offshoot, Clone}. The difference between the legitimate and illegitimate types is that when a parent node has multiple child nodes, the child node that is traversed first during topological sorting will be marked as Lineal, and the remaining child nodes will be marked as Offshoot. The reason for this classification is that such classification facilitates the recovery from the language sequence to RoadNet.

[0177] The sequence constructed in the above way is constructed in the order of depth-first traversal. Therefore, the RoadNet sequence has a forward dependency feature, which is well compatible with the language model.

[0178] 2. SAR encoding method:

[0179] The original autoregressive RoadNet model requires multiple iterations when generating sequences, each time predicting only a single token in the sequence, making the entire process extremely time-consuming. The use of autoregressive learning stems from an assumption in natural language processing that human language is highly coherent. This means that any attempt to generate text without this assumption will result in a significant drop in accuracy. See Figure 10B, which illustrates the data processing flow of the model. This is not entirely true for road structures. For example, in the road network generated in Figure 2, the locations of certain key points (starting points, forks, or merges) can be independent of previous vertices due to their topological characteristics. Beyond the locations of these key points, other tokens, because they are on the same line, have strong conditional dependencies, so using the original autoregressive approach can significantly improve performance. Based on these findings, the Semi-RoadNet-Transformer (SAR-RNTR) is proposed. This approach can significantly improve inference speed. To complement this approach, one form of representation for road topology information is the Semi-RoadNet sequence. The Semi-RoadNet sequence divides the trees in the directed forest into smaller subtrees so that each tree can be inferred in parallel. Due to the above measures, the autoregressive length can be reduced to as shown in Figure 12. Below we will describe this encoding process in detail.

[0180] Specifically, all key points in the directed acyclic graph (DAG) that meet the conditions of out-degree od(v)>1 or in-degree id(v)=1 or id(v)=0 can be determined. Then recursively traverse their original parent tree and child tree, and stop when the traversal encounters id(v)=0. Restore the edge key point v and its parent node, create a parent object, and designate it as a child object of v with a special label. Similarly, the clone will be traversed only after its original clone. As shown in Figure 10C, as in the autoregressive RoadNet sequence, an independent sequence can be created for each independent tree, so the SAR-RNTR sequence is a two-dimensional sequence. Where L is the maximum length of each subsequence and M is the number of subsequences. The filling rule is shown in the figure below. Note that the new data structure is also a directed forest, so the construction and recovery follow the autoregressive RoadNet.

[0181] SAR-RNTR is a two-dimensional sequence rather than the linear structure corresponding to the original language model, so the language model also needs to be adjusted accordingly. Specifically, SAR-RNTR can be divided into three components: (i) BEV feature encoder, (ii) key point transformer decoder, and (iii) parallel sequence transformer decoder. The BEV feature encoder does not need to be modified and is consistent with the encoder in AR-RNTR. The key point transformer decoder is a parallel transformer-decoder that takes a fixed set of position-embeddings as input and predicts key points. Its output is the key points mentioned above.

[0182] Based on the modified sequence obtained in the embodiment of the present application, special modifications are required when performing attention. Two types of attention are designed, tentatively referred to as inter-attention and intra-attention, as shown in Figure 10D. In this way, the memory complexity is reduced to O(M2+L2)<<O(M2×L2).

[0183] Through the above method, the sequence information in the training sample can include point information of feature points and line information of feature lines. In addition, the sequence information can also indicate the arrangement order of the feature points, which can be the prediction order of the subsequent language model. As introduced in the above embodiment, the arrangement order can be obtained by sorting the feature points using the sorting method in graph theory, for example, depth (or breadth) first traversal. Through the construction of the true value of the training sample introduced above, the language model can generate each feature point in the sequence information according to the order obtained by sorting the feature points using the sorting method in graph theory.

[0184] In one possible implementation, sequence information can be generated from image data, radar data, or map data (e.g., HD maps). Existing HD maps currently cover all roads and intersections in at least six cities, including complex road topologies. Optionally, surround camera images may not be required during pre-training. Generative pre-training can be performed directly using the GT synthetic road topology language of HD maps.

[0185] In an embodiment of the present application, the model can be pre-trained through self-supervised training. Specifically, in the training task, the above-mentioned sequence information can be used as the input of the language model. The language model can predict each unit in the sequence information (such as the information of the feature point) in turn. After each step of prediction, the next step of prediction can be made based on the historical prediction results (that is, the sequence information that has been predicted).

[0186] In an embodiment of the present application, point information of the first feature point and the second feature point, as well as side information of the first feature line and the second feature line in the topology of the target road can be obtained; wherein, the point information includes the physical position and category corresponding to the first feature point and the second feature point, and the side information includes the connection relationship and connection direction between the first feature point and the second feature point.

[0187] The point information of the first feature point and the line information of the first feature line can be information already predicted from the sequence information, while the point information of the second feature point and the line information of the second feature line can be information to be predicted from the sequence information. In other words, during the feedforward process, the language model can process the point information of the first feature point and the line information of the first feature line, and predict the point information of the second feature point and the line information of the second feature line.

[0188] To construct the input sequence, simply add SOS (a special integer value, a constant, used to indicate the beginning) to the original sequence. To construct the output sequence, simply add EOS (a special integer value, a constant, used to indicate the end) to the end of the original sequence. This gives us a pair of language sequences representing road structure that can be used for autoregressive supervision.

[0189] 802. Determine predicted values ​​of the point information of the second feature point and the side information of the second feature line through a deep learning model based on the point information of the first feature point and the side information of the first feature line.

[0190] In an embodiment of the present application, the model can be pre-trained through self-supervised training. Specifically, in the training task, only the road topology is pre-trained. According to the classic paradigm of self-supervision, a pre-task can be found, and then driven by the pre-task, the weights that need to be adjusted can be roughly adjusted through pre-training to accelerate the convergence of the network. At the same time, when designing the pre-training task, the dependence of the pre-training task on manual annotation will be specifically reduced. This can quickly increase the amount of pre-training data, thereby greatly increasing the pre-training effect and improving the generalization ability of the model. The structure of the pre-training model can be shown in Figure 11. The entire model is modeled as a high-precision map generation task. The input and output both use the original sequence, except that an SOS token is added to the head of the input sequence and an EOS token is added to the output sequence. According to the conditional dependency characteristics of the RoadNet sequence, the baseline is designed as an autoregressive RoadNet sequence generation model based on BEV features.

[0191] 803. Update the language model according to the predicted value, the point information of the second feature point, and the side information of the second feature line.

[0192] Label the length of the RoadNet sequence of length L as y and the predicted RoadNet sequence as Then the loss function of the autoregressive RoadNet Transformer is the maximum likelihood loss:

[0193] where yi is the i-th token of y, y<i represents all tokens before i, and wi is the class weight. In practice, since the linearity of label vc and the index 0 of vd appear most frequently, we set the wj of these classes to smaller values so as not to cause the classification surface to shift.

[0194] The embodiment of this application designs a self-supervised pre-training scheme. This training scheme does not require manual annotation and does not require perceptual data (such as surround-view camera pictures). During pre-training, the sequence information provided by the embodiment of this application is directly used to represent the road topology for generative pre-training.

[0195] In addition, the embodiment of this application proposes an extensible method for representing road topology structure, which uses the sequence prediction and logical learning ability of the language model itself to predict this sequence, and finally the sequence can be decoded into the representation form of the road network.

[0196] In addition, the above training process can also be a fine-tuning process for the language model after pre-training, that is, training based on real data.

[0197] The true value generation is responsible for the task of converting the complex road structure into a sequence. Since the road structure topology is relatively complex in the actual scenario, this part needs to be designed with sufficient redundancy to ensure that the corresponding topology can be safely predicted under the actual road conditions. The schematic flow diagram is shown in Figure 18.

[0198] Specifically, a dataset corresponding to the real scene can be collected by using image capture devices equipped on data collection vehicles, autonomous driving vehicles, etc. Optionally, when collecting data, attention needs to be paid to the multi-camera synchronization problem of the surround-view camera, and the internal and external parameters corresponding to each camera need to be retained.

[0199] The complex road structures in the corresponding dataset can be manually annotated. Road scenes can be roughly divided into three categories: straight roads, bifurcated and merged roads, and intersection roads. The situation of straight roads is relatively easy to handle, just directly annotate the corresponding lane lines and center lines; for bifurcated and merged roads, the topological connections need to be made at their bifurcated and merged positions; intersections are the most difficult part to annotate because there are no clearly visible markings on the ground. At this time, all possible virtual markings for passing on the ground can be supplemented during annotation.

[0200] The manually labeled results are converted into a special sequence using a language synthesizer. This special sequence must meet the requirements of the autoregressive model and fully cover the road topology in real-world scenarios. The specific encoding method can be referred to in the above embodiment and will not be repeated here.

[0201] During full training, we first load the pretrained weights and then perform full training using the ground truth generated in the previous step. The full training steps are similar to those of other deep learning models. Ground truth ground truth (GT) is used as supervision, and after a number-theoretic iteration, we ultimately obtain a valid network. Validity is defined as achieving a satisfactory evaluation result on a pre-set test set. We then proceed to the inference phase, which uses the weights from the training phase.

[0202] It should be noted that in some embodiments of the present application, there are multiple ways to determine the extent to which a language model should be trained using a loss function. The following provides some termination conditions for terminating further training of the language model, including but not limited to:

[0203] (1) The loss function reaches the preset threshold.

[0204] After configuring the loss function, a threshold value (e.g., 0.03) can be set for the loss function in advance. During the iterative training of the language model, after each training, it is determined whether the value of the loss function obtained in the current round of training reaches the threshold value. If not, the training is continued. If it reaches the preset threshold value, the training is terminated. The values ​​of the network parameters of the language model determined by the current round of training are used as the network parameter values ​​of the final trained language model.

[0205] (2) The loss function begins to converge.

[0206] After configuring the loss function, the language model can be iteratively trained. If the difference between the loss function value obtained in the current round of training and the loss function value obtained in the previous round of training is within a preset range (e.g., 0.01), the loss function is considered to have converged, and the training can be terminated. The values ​​of the network parameters of the language model determined by the current round of training are used as the network parameter values ​​of the final trained language model.

[0207] (3) The training reaches the preset number of times.

[0208] In this way, the number of iterations for training the language model can be pre-configured (for example, 1000 times). After the loss function is configured, the language model can be iteratively trained. After each round of training, the values ​​of the network parameters of the language model of that round will be stored until the number of training iterations reaches the preset number. Afterwards, the language models obtained in each round are verified using test data, and the value of the network parameter with the best performance is selected as the final network parameter value of the language model.

[0209] 2. Reasoning Stage

[0210] In the embodiment of the present application, the inference phase is the process by which the execution device 210 in Figure 5 performs road topology detection using the trained target model 201. For each image to be detected, the present application can use this trained model to predict lane lines, thereby obtaining the final prediction result. This process is explained in detail below. Please refer to Figure 9 for details. Figure 9 is a schematic flow chart of the road topology detection method provided in the embodiment of the present application, which may specifically include:

[0211] 901. Obtain target information, where the target information is data collected on the road condition of a target road.

[0212] In one possible implementation, the target information is image data or radar data, and pose information for collecting the image data or radar data. For example, the target information may be data collected by a vehicle-mounted surround view camera, a laser radar, or a millimeter-wave radar.

[0213] Since the downstream of autonomous driving perception, including regulation and prediction, is currently modeled on the bev plane, the perception results can be presented on the bev plane, that is, the target information can be converted into BEV features.

[0214] There are two methods for presenting perception results on the BEV plane: 1) designing the perception algorithm from the perspective of the vehicle (PV) and then projecting the perception structure from the PV onto the BEV plane; 2) mapping the PV image to the BEV plane based on camera parameters and then designing the perception algorithm on the BEV plane. Due to hardware limitations of autonomous vehicles, it is impossible to directly obtain a true BEV image using a camera. Instead, cameras are installed around the vehicle and an approximate BEV is obtained through projection. As shown in Figure 12, the BEV coverage area is located as shown. Generally speaking, the rear axle of an autonomous vehicle is more stable than the front axle, and its position better represents the vehicle's position. Therefore, the inertial navigation unit is typically mounted at the center of the rear axle. Based on this, when calibrating each camera in an autonomous vehicle, the calibration reference frame is selected as the inertial navigation coordinate system. In practice, the center of the BEV feature is generally set at the inertial navigation unit's installation location to facilitate BEV feature conversion.

[0215] The method of obtaining bev feature can be shown in Figure 12, including: collecting pictures through the surround-view camera, and converting them into features of the BEV plane through the model. The current method of converting pv input into bev plane feature map is LSS. One branch of LSS estimates the distribution of depth, and then takes the outer product with the image feature to project the feature under the pv perspective onto the depth predicted by the network to obtain the bev feature. The downstream uses the generated bev feature to perform static road structure recognition tasks. Since the depth used in LSS is generated by depth estimation, it has a certain robustness when projecting, and has good compatibility with different car models and different camera installation angles (of course, the camera parameters need to be as accurate as possible).

[0216] The mechanisms involved in LSS are relatively complex, so pre-training is required during actual training. This pre-training typically uses lane segmentation on the BEV plane as an auxiliary task. After several rounds of segmentation, the trained LSS model is connected to the backend language model to form a complete model, which is then trained end-to-end.

[0217] 902. Based on the target information, sequence information is obtained through a deep learning model, where the sequence information includes point information of feature points in the topology of the target road and edge information of feature lines in the topology of the target road; the point information includes the physical location and category corresponding to the feature points, and the edge information includes the connection relationship and connection direction between the feature points.

[0218] The language model may be a model trained based on the embodiment corresponding to FIG8 .

[0219] In one possible implementation, the feature point is a road key point, a lane line key point, a lane centerline key point, or a topological derivative point of the road in the target road, and the feature line is a lane line segment, a lane centerline segment, a road segment, or a traffic entity.

[0220] In a possible implementation, the side information further includes: shape features of the feature line.

[0221] In a possible implementation, the topology information further includes: an arrangement order of the feature points, where the arrangement order is consistent with an arrangement obtained by performing a topological sorting method specified in graph theory on the feature points.

[0222] In a possible implementation, the topological sorting method is a depth-first traversal, a breadth-first traversal, or a topological sorting method obtained through network learning.

[0223] Among them, the language model can be an autoregressive model, which uses the BEV feature to predict the topology of the road surface based on the generated bev feature. The bev feature output by the previous stage actually already contains the information of the road topology. What needs to be done in this step is to output the feature map containing the road topology information as a structured code representing the road topology according to the modeling method (that is, the sequence information in the embodiment of the present application). Optionally, this step can use an autoregressive method to predict the structured code of the road topology, and the specific structure used is transformer.

[0224] Referring to Figure 13, when this part is working, the first thing input into the network is a predetermined numerical value SOS as the beginning of the sequence. According to the beginning of the sequence, the first encoded token is output, and then the output encoded first token and SOS are put together and input into the transformer again to obtain the encoded second token. Then the encoded second token is put in parallel behind the previous input, and this cycle is repeated until the final output is EOS and then the cycle is stopped.

[0225] After deployment, the autonomous vehicle can capture real-time video streams. Using the model proposed in this embodiment, the model can output a topological language predicting the road structure. Decoding the model yields a graph representing the actual road structure. This graph, transmitted through some means, provides relevant information for vehicle control and braking. The prediction results from the nuscenes validation set are directly plotted on the BEV plane, as shown in Figure 14.

[0226] Existing technologies typically perform centerline detection first, followed by prediction or post-processing of the connectivity between centerlines. These two predictions are separate and unconnected. On the one hand, this two-stage prediction can lead to information loss and error accumulation. For example, deviations in centerline detection predictions can lead to even more inaccurate connectivity predictions. On the other hand, this relatively separate two-stage prediction fails to fuse the two types of data. Fusion of these two types of data allows the model to better perceive the overall road network. The position and shape information of the centerlines aids in predicting the connectivity between centerlines, while the connectivity between centerlines also aids in predicting the position and shape. Existing technologies fail to implement the interaction between these two types of information, limiting model performance.

[0227] In the embodiment of the present application, by directly predicting nodes (feature points) and edges (feature lines) in a single stage, the accumulated errors in the overall road structure caused by errors in each stage are avoided. Moreover, the fusion of the two data can promote each other in the network and increase the accuracy of the final result. And due to the change in the prediction mechanism, the embodiment of the present application can use the road topology alone for pre-training without the original data, which is also impossible for previous solutions.

[0228] In one possible implementation, first sequence information and the confidence of each basic unit in the first sequence information can be obtained through a deep learning model based on the target information and the first mask result; the first mask result is a mask result of an output of an intermediate iterative process of the language model; the basic units in the first sequence information whose confidence is less than a threshold are masked to obtain a second mask result; second sequence information is obtained through a deep learning model based on the target information and the second mask result; the second sequence information is used to obtain the sequence information.

[0229] Referring to Figure 15, during the training phase, a masked language modeling strategy can be used. Specifically, the input is randomly masked and fed into the language model, which then predicts the masked tokens. During the inference phase, the initial sequence, completely masked, can be fed into the language model, forcing it to predict the masked tokens. Of course, during the first prediction, the entire sequence is masked, making it impractical to predict all tokens at once. Therefore, after the first prediction, a filter is performed to re-mask low-confidence tokens while retaining high-confidence tokens. This approach yields increasingly higher accuracy over multiple predictions. The number of times here is actually much smaller than the total sequence length. In short, this mechanism allows a complete sequence prediction using only a few iterations (less than 0.1 times the sequence length), significantly reducing inference time.

[0230] 903. Determine the topology of the target road according to the sequence information.

[0231] After obtaining the topology of the target road, it can be passed to the downstream task module to perform services such as autonomous driving or AR navigation.

[0232] Next, the method in the embodiment of the present application is introduced with reference to a specific example.

[0233] The embodiment of the present application serves as a perception front end, and is mainly used to process snapshots of the surrounding environment captured by the surround-view camera onboard the autonomous driving vehicle, and then process the snapshots, and finally output the complex road structure information contained in the environment to the downstream in the form of a road network.

[0234] Figure 16 is a schematic diagram of reasoning in an embodiment of the present application, which is mainly divided into two parts: the BEV encoder and the transformer. The BEV encoder is mainly responsible for converting the surround view camera mounted on the autonomous vehicle into features representing the BEV plane; the transformer is mainly responsible for converting the BEV plane features into the target road structure.

[0235] Referring to Figure 17 , this embodiment of the present application includes two main phases: training and inference. The inference phase is the final operational state deployed on the autonomous vehicle, capturing images of the surrounding environment captured by the surround-view camera and then, after inference, outputting a structured representation of the road structure.

[0236] Next, the beneficial effects of the embodiments of the present application are described in conjunction with experiments:

[0237] Quantitative comparison of the results of the embodiment of this application with other solutions on the NuScenes dataset. The evaluation method is based on the TPLR paper. The comparison shows that the results of the embodiment of this application have a clear advantage over other algorithms. Compared with the latest work, as shown in Table 1, the embodiment of this application can surpass the SOTA algorithm on the NuScenes dataset.

[0238] Table 1

[0239] Visualization effects of this scheme and other schemes on the nuscenes dataset and other schemes, among which TR-RNN and PTLR are specific implementations of the TPLR paper.

[0240] Refer to Figure 19, which is a visualization of the embodiment of the present application and other methods on the nuscenes dataset. It can be seen from the boxes in Figure 19 that the effect of AR-RNTR in the embodiment of the present application is better than other algorithms in terms of detailed depiction.

[0241] Next, a data processing device provided by an embodiment of the present application will be described from the perspective of a device. Referring to FIG. 20 , FIG. 20 is a schematic diagram of the structure of a data processing device provided by an embodiment of the present application. As shown in FIG. 20 , a data processing device 2000 provided by an embodiment of the present application includes:

[0242] An acquisition module 2001 is used to acquire target information, wherein the target information is data collected on the road condition of a target road;

[0243] For a detailed introduction to the acquisition module 2001 , please refer to the description of step 901 in the above embodiment, which will not be repeated here.

[0244] Processing module 2002 is configured to obtain sequence information using a deep learning model based on the target information, wherein the sequence information includes point information of feature points in the topology of the target road and edge information of feature lines in the topology of the target road; the point information includes the physical location and category corresponding to the feature points, and the edge information includes the connection relationship and connection direction between the feature points;

[0245] The topology of the target road is determined according to the sequence information.

[0246] The detailed introduction of the processing module 2002 can refer to the description of step 902 and step 903 in the above embodiment, which will not be repeated here.

[0247] In one possible implementation, the feature point is a road key point, a lane line key point, a lane centerline key point, or a topological derivative point of the road in the target road, and the feature line is a lane line segment, a lane centerline segment, a road segment, or a traffic entity.

[0248] In a possible implementation, the side information further includes: shape features of the feature line.

[0249] In a possible implementation, the topology information further includes: an arrangement order of the feature points, where the arrangement order is consistent with an arrangement obtained by performing a topological sorting method specified in graph theory on the feature points.

[0250] In a possible implementation, the topological sorting method is a depth-first traversal, a breadth-first traversal, or a topological sorting method obtained through network learning.

[0251] In one possible implementation, the category is one of the following:

[0252] Starting node, relay node, fork node and merge node.

[0253] In a possible implementation, the processing module 2002 is specifically configured to:

[0254] Obtaining, through a deep learning model, first sequence information and the confidence of each basic unit in the first sequence information based on the target information and the first mask result; the first mask result is a mask result outputted from an intermediate iterative process of the language model;

[0255] Masking the basic units in the first sequence information whose confidence is less than a threshold to obtain a second masking result;

[0256] According to the target information and the second mask result, second sequence information is obtained through a deep learning model; the second sequence information is used to obtain the sequence information.

[0257] 21 , which is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. As shown in FIG21 , a data processing device 2100 provided in an embodiment of the present application includes:

[0258] An acquisition module 2101 is configured to acquire sequence information, the sequence information including point information of a first feature point and a second feature point in the topology of a target road, and side information of a first feature line and a second feature line in the topology of the target road; the point information including the physical location and category of the first feature point and the second feature point, and the side information including the connection relationship and connection direction between the first feature point and the second feature point;

[0259] For a detailed introduction to the acquisition module 2101 , please refer to the description of step 801 in the above embodiment, which will not be repeated here.

[0260] A processing module 2102 is configured to determine, based on the point information of the first feature point and the side information of the first feature line, a predicted value of the point information of the second feature point and the side information of the second feature line using a deep learning model;

[0261] For a detailed introduction to the processing module 2102 , please refer to the description of step 802 in the above embodiment, which will not be repeated here.

[0262] The updating module 2103 is configured to update the language model according to the predicted value, the point information of the second feature point, and the side information of the second feature line.

[0263] For a detailed introduction to the processing module 2102 , please refer to the description of step 803 in the above embodiment, which will not be repeated here.

[0264] Next, a terminal device provided by an embodiment of the present application is introduced. Please refer to Figure 22. Figure 22 is a structural schematic diagram of a terminal device provided by an embodiment of the present application. The terminal device 2200 can be specifically manifested as a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., which is not limited here. Among them, the terminal device 2200 implements the function of the data processing method in the embodiment corresponding to Figure 5. Specifically, the terminal device 2200 includes: a receiver 2201, a transmitter 2202, a processor 2203 and a memory 2204 (wherein the number of processors 2203 in the terminal device 2200 can be one or more), wherein the processor 2203 may include an application processor 22031 and a communication processor 22032. In some embodiments of the present application, the receiver 2201, the transmitter 2202, the processor 2203 and the memory 2204 may be connected via a bus or other means.

[0265] The memory 2204 may include a read-only memory and a random access memory, and provides instructions and data to the processor 2203. A portion of the memory 2204 may also include non-volatile random access memory (NVRAM). The memory 2204 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0266] Processor 2203 controls the operation of the terminal device. In specific applications, the various components of the terminal device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0267] The method disclosed in the above embodiment of the present application can be applied to the processor 2203, or implemented by the processor 2203. The processor 2203 can be an integrated circuit chip with signal processing capabilities. During the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 2203 or the instruction in the form of software. The above-mentioned processor 2203 can be a general-purpose processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and a vision processor (vision processing unit, VPU), a tensor processor (tensor processing unit, TPU) and other processors suitable for AI computing, and can further include an application specific integrated circuit (ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 2203 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of this application can be directly implemented as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 2204, and processor 2203 reads the information in memory 2204 and, in conjunction with its hardware, completes steps 501 to 504 in the above embodiment.

[0268] Receiver 2201 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the terminal device. Transmitter 2202 can be used to output digital or character information through the first interface. Transmitter 2202 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group. Transmitter 2202 can also include a display device such as a display screen.

[0269] The embodiment of the present application also provides a server. Please refer to Figure 23, which is a schematic diagram of the structure of a server provided by the embodiment of the present application. Specifically, the server 2300 is implemented by one or more servers. The server 2300 may have relatively large differences due to different configurations or performance. It may include one or more central processing units (CPUs) 2323 (for example, one or more processors) and memory 2332, and one or more storage media 2330 (for example, one or more mass storage devices) for storing application programs 2342 or data 2344. Among them, the memory 2332 and the storage medium 2330 can be temporary storage or permanent storage. The program stored in the storage medium 2330 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the central processing unit 2323 can be configured to communicate with the storage medium 2330 to execute a series of instruction operations in the storage medium 2330 on the server 2300.

[0270] The server 2300 may also include one or more power supplies 2326, one or more wired or wireless network interfaces 2350, one or more input and output interfaces 2358; or one or more operating systems 2341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0271] Specifically, the server may perform steps 501 to 504 in the above embodiment.

[0272] An embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0273] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0274] The execution device, training device or terminal device provided in the embodiments of the present application can specifically be a chip, and the chip includes: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute the computer execution instructions stored in the storage unit, so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0275] Specifically, see Figure 24, which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 2400. NPU 2400 is mounted on the host CPU (host CPU) as a coprocessor and is assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 2403, which is controlled by controller 2404 to extract matrix data from memory and perform multiplication operations.

[0276] The NPU 2400 can implement the data processing method provided in the embodiments described in FIG. 8 and FIG. 9 through the mutual cooperation between various internal components.

[0277] More specifically, in some implementations, the arithmetic circuit 2403 in the NPU 2400 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 2403 is a two-dimensional systolic array. The arithmetic circuit 2403 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2403 is a general-purpose matrix processor.

[0278] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 2402 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 2401 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 2408.

[0279] Unified memory 2406 is used to store input and output data. Weight data is directly transferred to weight memory 2402 through the Direct Memory Access Controller (DMAC) 2405. Input data is also transferred to unified memory 2406 through the DMAC.

[0280] BIU stands for Bus Interface Unit, i.e., bus interface unit 2410 , which is used for interaction between the AXI bus and DMAC and instruction fetch buffer (IFB) 2409 .

[0281] The bus interface unit 2410 (BIU) is used for the instruction fetch memory 2409 to obtain instructions from the external memory, and is also used for the storage unit access controller 2405 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0282] DMAC is mainly used to move input data in the external memory DDR to the unified memory 2406 or to move weight data to the weight memory 2402 or to move input data to the input memory 2401.

[0283] The vector calculation unit 2407 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit 2403, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0284] In some implementations, the vector calculation unit 2407 can store the processed output vector in the unified memory 2406. For example, the vector calculation unit 2407 can apply a linear function or a nonlinear function to the output of the operation circuit 2403, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 2407 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2403, for example, for use in subsequent layers in a neural network.

[0285] An instruction fetch buffer 2409 connected to the controller 2404 is used to store instructions used by the controller 2404;

[0286] Unified memory 2406, input memory 2401, weight memory 2402, and instruction fetch memory 2409 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0287] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0288] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0289] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0290] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0291] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A road topology detection method, characterized in that: include: Acquiring target information, wherein the target information is data collected on the road condition of the target road; According to the target information, sequence information is obtained through a deep learning model, wherein the sequence information includes point information of feature points in the topology of the target road and edge information of feature lines in the topology of the target road; the point information includes a physical position and a category corresponding to the feature point, and the edge information includes a connection relationship and a connection direction between the feature points; The topology of the target road is determined according to the sequence information.

2. The method according to claim 1, characterized in that The feature points are road key points, lane line key points, lane centerline key points, or topological derivative points of the target road, and the feature lines are lane line segments, lane centerline segments, road segments, or traffic entities.

3. The method according to claim 1 or 2, characterized in that: The edge information also includes: shape features of the feature lines.

4. The method according to any one of claims 1 to 3, characterized in that: The topological information further includes: an arrangement order of the feature points, and the arrangement order is consistent with an arrangement obtained by performing a topological sorting method specified in graph theory on the feature points.

5. The method according to claim 4, characterized in that The topological sorting method is a depth-first traversal, a breadth-first traversal, or a topological sorting method obtained through network learning.

6. The method according to any one of claims 1 to 5, characterized in that: The category is one of the following: Starting node, relay node, fork node and merge node.

7. The method according to any one of claims 1 to 6, characterized in that: The step of obtaining sequence information through a deep learning model according to the target information includes: According to the target information and the first mask result, the first sequence information and the confidence of each basic unit in the first sequence information are obtained through a deep learning model; the first mask result is a mask result outputted from an intermediate iteration process of the language model; Masking the basic units in the first sequence information whose confidence is less than a threshold value to obtain a second masking result; According to the target information and the second mask result, second sequence information is obtained through a deep learning model; the second sequence information is used to obtain the sequence information.

8. The method according to any one of claims 1 to 7, characterized in that: The target information is image data or radar data, and position information of collecting the image data or radar data.

9. A model training method, characterized in that: The method comprises: Acquire sequence information, the sequence information including point information of a first feature point and a second feature point in the topology of a target road, and edge information of a first feature line and a second feature line in the topology of the target road; the point information includes physical positions and categories corresponding to the first feature point and the second feature point, and the edge information includes a connection relationship and a connection direction between the first feature point and the second feature point; Determine, by a deep learning model, predicted values ​​of the point information of the second feature point and the side information of the second feature line according to the point information of the first feature point and the side information of the first feature line; The language model is updated according to the predicted value, the point information of the second feature point, and the side information of the second feature line.

10. The method according to claim 9, characterized in that The sequence information is determined by image data, radar data or map data.

11. A data processing device, characterized in that: The device comprises: An acquisition module, used to acquire target information, wherein the target information is data collected on the road condition of the target road; A processing module, configured to obtain sequence information through a deep learning model according to the target information, wherein the sequence information includes point information of feature points in the topology of the target road and edge information of feature lines in the topology of the target road; the point information includes a physical position and a category corresponding to the feature point, and the edge information includes a connection relationship and a connection direction between the feature points; The topology of the target road is determined according to the sequence information.

12. The device according to claim 11, characterized in that The feature points are road key points, lane line key points, lane centerline key points, or topological derivative points of the target road, and the feature lines are lane line segments, lane centerline segments, road segments, or traffic entities.

13. The device according to claim 11 or 12, characterized in that The edge information also includes: shape features of the feature lines.

14. The device according to any one of claims 11 to 13, characterized in that: The topological information further includes: an arrangement order of the feature points, and the arrangement order is consistent with an arrangement obtained by performing a topological sorting method specified in graph theory on the feature points.

15. The device according to claim 14, characterized in that The topological sorting method is a depth-first traversal, a breadth-first traversal, or a topological sorting method obtained through network learning.

16. The device according to any one of claims 11 to 15, characterized in that The category is one of the following: Starting node, relay node, fork node and merge node.

17. The device according to any one of claims 11 to 16, characterized in that The processing module is specifically used for: According to the target information and the first mask result, the first sequence information and the confidence of each basic unit in the first sequence information are obtained through a deep learning model; the first mask result is a mask result outputted from an intermediate iteration process of the language model; Masking the basic units in the first sequence information whose confidence is less than a threshold value to obtain a second masking result; Obtaining second sequence information through a deep learning model according to the target information and the second mask result; The second sequence information is used to obtain the sequence information.

18. The device according to any one of claims 11 to 17, characterized in that The target information is image data or radar data, and position information of collecting the image data or radar data.

19. A model training device, characterized in that: The device comprises: an acquisition module, configured to acquire sequence information, wherein the sequence information includes point information of a first feature point and a second feature point in the topology of a target road, and edge information of a first feature line and a second feature line in the topology of the target road; the point information includes physical locations and categories corresponding to the first feature point and the second feature point, and the edge information includes a connection relationship and a connection direction between the first feature point and the second feature point; A processing module, configured to determine predicted values ​​of the point information of the second feature point and the side information of the second feature line through a deep learning model according to the point information of the first feature point and the side information of the first feature line; An updating module is used to update the language model according to the predicted value, the point information of the second feature point and the side information of the second feature line.

20. The device according to claim 19, characterized in that The sequence information is determined by image data, radar data or map data.

21. A computing device, characterized in that The computing device includes a memory and a processor; the memory stores codes, and the processor is configured to obtain the codes and execute the method according to any one of claims 1 to 10.

22. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, which, when executed by one or more computers, enable the one or more computers to implement the method of any one of claims 1 to 10.

23. A computer program product comprising code, characterized in that When the code is executed, it is used to implement the method according to any one of claims 1 to 10.

24. A chip, comprising a processor, characterized in that: The processor is used to support a data processing device to implement the method according to any one of claims 1 to 10.

Images