Map data processing method and device, equipment, storage medium and program product
Through multimodal map data feature extraction and autoregressive processing technology, the problems of low efficiency and insufficient accuracy of map data updates are solved, and efficient, precise processing and customized demand response to map data of autonomous driving vehicles are achieved.
Patent Information
- Application Number
- CN202310588239.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-23
- Publication Date
- 2024-11-26
AI Technical Summary
The existing map data processing technology has problems such as low data update efficiency, insufficient accuracy, and inflexible customer customization needs, especially in the multimodal map data processing of autonomous vehicles.
Using multimodal map data feature extraction, fusion and autoregression iterative processing methods, the multimodal data collected by the vehicle during driving is extracted and fused through neural network models to generate globally perceived map features and according to user intentions Perform autoregression processing to generate target map data.
It improves the efficiency and accuracy of map data updates, can quickly respond to customer customized needs, and generate more accurate and rich map data.
Smart Images

Figure CN119025601A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to computer technology and map application technology, and in particular to a map data processing method, device, electronic device, computer-readable storage medium and computer program product. Background Art
[0002] Computer vision studies related theories and technologies, attempting to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technologies generally include image processing, image semantic understanding, image retrieval, video processing, video semantic understanding, three-dimensional object reconstruction, virtual reality, augmented reality, biometric recognition, simultaneous positioning and map construction, etc.
[0003] In map construction applications, the map production line in related technologies is mainly based on fixed processes combined with manual labor. Although some links have been automated, there are also a large number of manual corrections and quality inspections. Customized transformation of map data also requires the writing of special algorithms, resulting in a lack of agility in responding to customer customization needs and slow map data updates. Summary of the invention
[0004] The embodiments of the present application provide a map data processing method, device, electronic device, computer-readable storage medium and computer program product, which can improve the efficiency and accuracy of map data updating.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] The present application provides a map data processing method, including:
[0007] Performing feature extraction processing on multimodal map data collected by multiple sensors during the driving process of the target vehicle to obtain each modal feature corresponding to the target vehicle;
[0008] Performing feature fusion processing based on the various modal features to obtain multi-modal fusion features corresponding to the target vehicle;
[0009] Based on the multimodal fusion features, autoregressive iterative processing is performed through a first language model to obtain a global perception map feature corresponding to the target vehicle;
[0010] A map data processing intention is obtained, and based on the global perception map features and the map data processing intention, an autoregressive iterative process is performed through a second language model to obtain target map data corresponding to the map data processing intention.
[0011] The present application provides a map data processing device, including:
[0012] A feature extraction module is used to perform feature extraction processing on multimodal map data collected by multiple sensors during the driving process of the target vehicle to obtain each modal feature corresponding to the target vehicle;
[0013] A feature fusion module, used for performing feature fusion processing based on the features of each modality to obtain a multi-modal fusion feature corresponding to the target vehicle;
[0014] A first iteration module is used to perform autoregressive iterative processing through a first language model based on the multimodal fusion features to obtain a global perception map feature corresponding to the target vehicle;
[0015] The second iteration module is used to obtain the map data processing intention, and based on the global perception map features and the map data processing intention, perform autoregressive iterative processing through a second language model to obtain target map data corresponding to the map data processing intention.
[0016] In the above scheme, the feature extraction module is also used to use the neural network corresponding to each modality to perform feature extraction processing on the multimodal map data collected by multiple sensors during the driving process of the target vehicle, so as to obtain the features of each modality corresponding to the target vehicle.
[0017] In the above scheme, the feature fusion module is also used to obtain sensor extrinsic parameters indicating the positional relationship between each of the sensors; based on the sensor extrinsic parameters, feature fusion processing is performed on the modal features to obtain multimodal fusion features corresponding to the target vehicle.
[0018] In the above scheme, the feature fusion module is also used to map the various modal features into the same global three-dimensional space respectively to obtain the global features of each modality; determine the grid positions corresponding to the various modal global features in the global three-dimensional space based on the positional relationship between the various sensors indicated by the sensor external parameters; aggregate the various modal global features that fall at the same position to obtain the multimodal fusion features corresponding to the target vehicle.
[0019] In the above scheme, the feature fusion module is also used to map the features of the image to the global three-dimensional space according to the transformation relationship between the coordinate system of the image and the coordinate system of the global three-dimensional space, so as to obtain the global features of the image; to divide the point cloud into subspaces along the three-dimensional space dimension according to the distribution of the target perception area to obtain multiple subspace grids, and to perform feature abstraction on the point cloud features in each subspace network in the multiple subspace grids, and to combine the results of the feature abstraction to obtain the global features of the point cloud; or, to establish an adjacency graph according to the spatial distribution of the point cloud, to perform feature aggregation on the point cloud features on the adjacency graph using a graph convolutional deep network, and to project the position of each data point of the point cloud into the global three-dimensional space, so as to obtain the global features of the point cloud based on the position and the results of the feature aggregation.
[0020] In the above scheme, the first iteration module is also used to obtain the previous round of iteration results output by the first language model for each round of iteration of the autoregressive iterative processing, wherein the previous round of iteration results are obtained by performing autoregressive iterative processing on the first language model based on the multimodal fusion features corresponding to the target vehicle at the previous moment; combining the previous round of iteration results and the multimodal fusion features corresponding to the target vehicle at the current moment, performing autoregressive iterative processing on the first language model, and using the output of this round of iteration results as the global perception map features corresponding to the target vehicle at the current moment.
[0021] In the above scheme, the first language model is implemented based on a fusion training framework, which includes an encoding network, a fusion network, the first language model, and a decoding network; the device also includes: a first training module, which is used to obtain, for each vehicle sample in the vehicle sample set, multimodal map data collected by multiple sensors during the driving process of each vehicle sample, and global map data corresponding to each vehicle sample; through the encoding network, feature extraction processing is performed on the multimodal map data collected by multiple sensors during the driving process of each vehicle sample to obtain each modal feature corresponding to each vehicle sample; through the fusion network, feature fusion processing is performed on each modal feature corresponding to each vehicle sample to obtain a multimodal fusion feature corresponding to each vehicle sample ; Based on the multimodal fusion features corresponding to each of the vehicle samples, autoregressive iterative processing is performed through the first language model to obtain the global perception map features corresponding to each of the vehicle samples; the global perception map features corresponding to each of the vehicle samples are decoded through the decoding network to obtain the global perception map data corresponding to each of the vehicle samples; based on the global map data of each vehicle sample and the global perception map data corresponding to each of the vehicle samples, the loss function of the fusion training framework is determined; based on the loss function, the parameters of the fusion training framework are updated to obtain the updated fusion training framework, and the first language model in the updated fusion training framework is used as the model for autoregressive iterative processing of the multimodal fusion features corresponding to the target vehicle.
[0022] In the above scheme, the second iteration module is also used to obtain the previous round of iteration results output by the second language model for each round of iteration of the autoregressive iterative processing, wherein the previous round of iteration results are obtained by performing autoregressive iterative processing on the second language model based on the global perception map features corresponding to the target vehicle at the previous moment and the map data processing intention; combining the previous round of iteration results and the global perception map features corresponding to the target vehicle at the current moment, performing autoregressive iterative processing on the second language model, and using the output of this round of iteration results as the target map data of the target vehicle at the current moment corresponding to the previous round of iteration results.
[0023] In the above scheme, the second iteration module is also used to encode the map to be processed to obtain the encoding features of the map to be processed when the map data processing intention is used to indicate the execution of a target operation on the map to be processed; based on the encoding features, the global perception map features and the map data processing intention, autoregressive iterative processing is performed through a second language model to obtain target map data corresponding to the map data processing intention, wherein the target map data is map data obtained after the target operation is executed on the map to be processed.
[0024] In the above scheme, the second iterative module is also used to fuse the coding features and the global perception map features to obtain fused coding features; the fused coding features and the map data processing intent are input into the second language model for autoregressive iterative processing to obtain target map data corresponding to the map data processing intent.
[0025] In the above scheme, the second iterative module is also used to directly input the encoding features, the global perception map features and the map data processing intention into the second language model for autoregressive iterative processing to obtain target map data corresponding to the map data processing intention.
[0026] In the above scheme, the device also includes: a second training module, which is used to sample a first number of instruction samples from the instruction sample set, and label the first number of instruction samples to obtain corresponding instruction labels; use the first number of instruction samples as input and the corresponding instruction labels as output to train a second language model; sample a second number of instruction samples from the instruction sample set, and use the second number of instruction samples as input to output output instructions corresponding to the second number of instruction samples from the second language model; sort the output instructions, and train a reward model based on the sorting results; sample a third number of instruction samples from the instruction sample set, and use the third number of instruction samples as input, and use reinforcement learning to optimize the second language model in combination with the reward model obtained through training.
[0027] An embodiment of the present application provides an electronic device, including:
[0028] A memory for storing computer executable instructions or computer programs;
[0029] The processor is used to implement the map data processing method provided in the embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.
[0030] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program for implementing the map data processing method provided in the embodiment of the present application when executed by a processor.
[0031] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the map data processing method provided in the embodiment of the present application is implemented.
[0032] The embodiments of the present application have the following beneficial effects:
[0033] By using the embodiments of the present application, since the multimodal map data can reflect the multimodal or multidimensional map data of the target vehicle during driving, it can provide data support for the subsequent improvement of the effect of map data processing; the multimodal fusion features obtained by fusing the relevant features of the multimodal or multidimensional map data can well reflect the map perception information of the driving path of the target vehicle during driving, and the autoregressive iterative processing is performed based on the multimodal fusion features, so that the complete global perception map representation (i.e., global perception map features) of the target vehicle in the entire driving path can be obtained, and the autoregressive iterative processing is performed according to the user's map processing intention and the global perception map features corresponding to the target vehicle. Since the iterative process adopts the autoregressive iterative method, even if a certain map perception information is missing, it can be corrected by other or subsequent perception results, so that a more accurate global perception map representation is finally obtained, which can improve the accuracy of the map data; and during the autoregressive iteration, new map data can be output according to the user's map processing intention, that is, the target map data that meets the map data processing intention, which meets the user's map customization requirements and improves the map data update efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 This is a schematic diagram of the map generation process provided by the embodiment of the present application;
[0035] Figure 2 is a structural diagram of a map data processing system provided in an embodiment of the present application;
[0036] Figure 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0037] Figure 4 is a flowchart of a map data processing method provided in an embodiment of the present application;
[0038] Figure 5 is a flowchart of a map data processing method provided in an embodiment of the present application;
[0039] Figure 6 is a flowchart of a map data processing method provided in an embodiment of the present application;
[0040] Figure 7 It is a schematic diagram of model training provided in an embodiment of the present application;
[0041] Figure 8 is a flowchart of a map data processing method provided in an embodiment of the present application;
[0042] Fig. 9 It is a schematic diagram of the composition of the map data processing model provided in the embodiment of the present application;
[0043] Fig.10 It is a training diagram of the second language model provided in an embodiment of the present application. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.
[0045] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0046] In the following description, the terms "first\second..." are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second..." can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0048] See also Figure 1 , Figure 1 It is a schematic diagram of the map generation flow provided in the embodiment of the present application. When generating map data, the relevant technology is roughly implemented in accordance with multiple links such as demand-oriented-customization-cloud-based-delivery. Before generating map data, the map product team customizes the vehicle map (SD, Standard Definition map) Pro specifications according to the original equipment manufacturer (OEM, Original Equipment Manufacturer)'s autonomous driving function requirements and map compliance requirements to obtain customized SD Pro specifications; the map data process includes a high-definition map (HD, High-precision Map) production line and an SD Pro production line, wherein the HD Map production line includes HD Map data acquisition and HD Map data production; the SD Pro production line matches the vehicle map with the high-definition map, and then integrates the map data based on the customized SD Pro specifications and matching results, based on the DOT generation algorithm and the map data update in the cloud; finally, the integrated map data is quality tested in the delivery stage and released to obtain a customized map engine.
[0049] It can be seen that the map production line in the relevant technology is mainly based on fixed processes combined with manual labor. Although some links have been automated, there are also a large number of manual corrections and quality inspection links. Therefore, there are at least the following problems: 1) Due to the complexity of the process in the data update process, the reliability of the data decreases with the increase of the process, and the personnel and time costs increase with the increase of the process; 2) Some links still rely on manual labor. The reliability of the manual links is unstable and depends on experience and concentration. At the same time, it is difficult to quickly expand or reduce production capacity. The data production cycle is long and the data update is slow (such as quarterly or annual updates); 3) It is not agile to meet customer customization needs. Customized transformation of map data requires the writing of special algorithms. Different needs often require the development of separate processes and algorithms, and will involve the problem of retraining of personnel; 4) It has high requirements for production tools and environment, such as requiring the collection vehicle to have higher accuracy, or less interference in the collection environment.
[0050] To this end, embodiments of the present application provide a map data processing method, device, electronic device, computer-readable storage medium, and computer program product to at least solve the above technical problems.
[0051] See also Figure 2 , Figure 2 It is a structural diagram of a map data processing system provided in an embodiment of the present application. The terminal 400 is connected to the server 200 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.
[0052] In some embodiments, the map data processing method provided in the embodiments of the present application can be implemented by a terminal and a server in collaboration, the terminal 400 receives a map data processing request from a user, the terminal 400 sends the map data processing request to the server 200, and the server 200 performs feature extraction processing on the multimodal map data collected by multiple sensors during the driving process of the target vehicle carried in the map data processing request in response to the map data processing request, and obtains each modal feature corresponding to the target vehicle; performs feature fusion processing based on each modal feature to obtain the multimodal fusion feature corresponding to the target vehicle; performs autoregressive iterative processing through a first language model based on the multimodal fusion feature to obtain the global perception map feature corresponding to the target vehicle; obtains the map data processing intent, and performs autoregressive iterative processing through a second language model based on the global perception map feature and the map data processing intent to obtain the target map data corresponding to the map data processing intent; the server 200 returns the obtained target map data to the terminal 400 for display. For example, when the map data processing request is used to request the generation of a high-definition map with higher accuracy and richer elements based on a standard-definition map, the target map data is a high-definition map. After the server 200 generates the high-definition map, the generated high-definition map is returned to the terminal 400 for display.
[0053] In some embodiments, the map data processing method provided in the embodiments of the present application can also be implemented independently by a terminal or a server. Taking the implementation by a terminal alone as an example, the terminal 400 receives a map data processing request from a user, and the terminal 400 performs feature extraction processing on the multimodal map data collected by multiple sensors during the driving process of the target vehicle carried in the map data processing request to obtain each modal feature corresponding to the target vehicle; performs feature fusion processing based on each modal feature to obtain the multimodal fusion feature corresponding to the target vehicle; based on the multimodal fusion feature, performs autoregressive iterative processing through a first language model to obtain the global perception map feature corresponding to the target vehicle; obtains the map data processing intent, and based on the global perception map feature and the map data processing intent, performs autoregressive iterative processing through a second language model to obtain the target map data corresponding to the map data processing intent, and displays the target map data in the terminal 400. For example, when the map data processing request is used to request the generation of a high-definition map with higher accuracy and richer elements based on a standard-definition map, the target map data is a high-definition map, and the terminal 400 generates and displays the high-definition map.
[0054] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.
[0055] In some embodiments, the terminal or server can implement the map data processing method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a video APP or a live broadcast APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.
[0056] See also Figure 3 , Figure 3 It is a structural diagram of an electronic device provided in an embodiment of the present application. The electronic device may be a terminal or a server. In the embodiment of the present application, the electronic device is taken as an example for description. Figure 3 The terminal 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 3 Various buses are labeled as bus system 440 .
[0057] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0058] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.
[0059] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.
[0060] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.
[0061] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.
[0062] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;
[0063] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;
[0064] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);
[0065] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.
[0066] In some embodiments, the map data processing device provided in the embodiments of the present application can be implemented in software. Figure 3 A map data processing device 455 stored in a memory 450 is shown, which may be software in the form of a program or a plug-in, and includes the following software modules: a feature extraction module 4551, a feature fusion module 4552, a first iteration module 4553, and a second iteration module 4554. These modules are logical, and thus may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.
[0067] The map data processing method provided by the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the terminal provided by the embodiment of the present application. As mentioned above, the electronic device that implements the map data processing method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeatedly described below.
[0068] See also Figure 4 , Figure 4 is a flowchart of a map data processing method provided in an embodiment of the present application, which will be combined with Figure 4 Steps 101 to 104 are shown for explanation.
[0069] In step 101, feature extraction processing is performed on multimodal map data collected by multiple sensors during the driving process of the target vehicle to obtain each modal feature corresponding to the target vehicle.
[0070] In practical applications, the target vehicle can be an autonomous driving vehicle or a non-autonomous driving vehicle. Sensors can be divided into on-board sensors and roadside sensors according to their locations. On-board sensors refer to sensors installed on the target vehicle, such as cameras, lidars, millimeter-wave radars, and microphones. These sensors are used to obtain map data such as multi-perspective images, radar point clouds, and voice information (such as the horn sound of vehicles around the target vehicle) around the target vehicle. In addition, sensors for obtaining navigation information can also be set on the target vehicle, such as the Global Navigation Satellite System (GNSS). These sensors are used to obtain navigation map data of the target vehicle during driving (i.e., data collected from the navigation map), such as lane lines, sidewalks, or stop lines on the road, as well as the relationship between different nodes on the road, such as bifurcations or intersections.
[0071] Roadside sensors refer to sensors installed at intersections or roadsides of the roads that the target vehicle passes through during its driving process, such as cameras, lidars, and millimeter-wave radars. These sensors are used to collect map data within a fixed range on intersections and other roads, such as lane lines, signs, traffic lights, and other specific location data.
[0072] During the driving process of the target vehicle, multiple sensors observe the road environment, vehicles, pedestrians, etc. from different perspectives and obtain map data of different modalities. These multimodal map data can be used as the basis for map data processing (map drawing, map updating, map acceptance, etc.). Multimodal or multi-dimensional map data is conducive to improving the accuracy of map data processing.
[0073] In some embodiments, feature extraction processing can be performed on multimodal map data collected by multiple sensors during the driving of the target vehicle to obtain each modal feature corresponding to the target vehicle in the following manner: feature extraction processing is performed on multimodal map data collected by multiple sensors during the driving of the target vehicle using a neural network corresponding to each modality to obtain each modal feature corresponding to the target vehicle.
[0074] In practical applications, the multimodal map data collected by multiple sensors during the driving process of the target vehicle can be input into the encoding network for feature extraction to obtain the corresponding features of each modality for subsequent use, wherein the encoding network is a neural network model based on weight parameters, including but not limited to convolutional neural networks, recurrent neural networks, transformer networks (such as Transformer), etc.; in this way, feature extraction of multimodal map data through a neural network can improve the efficiency of feature extraction.
[0075] In addition, considering the different characteristics of map data of different modalities, feature extraction can be performed on map data of different modalities separately, that is, the neural network corresponding to each modality (essentially a feature extractor composed of a neural network) is used to perform feature extraction processing on the multimodal map data collected by multiple sensors during the driving process of the target vehicle to obtain the features corresponding to the map data of each modality (i.e., features of each modality). For example, for image-type map data (i.e., images), the image-type map data extracted by multiple cameras can be encoded through a feature extractor composed of stacked convolutional networks to obtain image features of each perspective, which contain rich semantic context information.
[0076] For example, for map data in the form of laser radar point cloud, a deep neural network adapted to three-dimensional data can be used, or the spatial position, reflectivity and other information of the point cloud can be used for feature extraction. Similarly, the features of millimeter-wave radar can also be extracted in a similar way, and the features of this mode can express accurate geometric spatial position information. For navigation map data, deep neural network encoding can be used for feature extraction. In this way, for map data of different modes, different neural networks can be used for feature extraction, which can improve the accuracy of feature extraction.
[0077] It should be noted that, in the training process of the neural network corresponding to the map data of the same modality, the map data of the modality at different times can be used for training to improve the training effect.
[0078] In step 102, feature fusion processing is performed based on each modal feature to obtain a multi-modal fusion feature corresponding to the target vehicle.
[0079] In some embodiments, see Figure 5 , Figure 5 is a flowchart of a map data processing method provided in an embodiment of the present application, Figure 4 Step 102 in the above example can be accomplished by Figure 5 Steps 1021 to 1022 in the embodiment are implemented as follows:
[0080] In step 1021, sensor extrinsic parameters indicating the positional relationship between the sensors are obtained.
[0081] Here, it is necessary to perform external parameter registration on the sensors that collect the map data of each modality to obtain the sensor extrinsic parameters, so as to calibrate each sensor uniformly to the world coordinate system. The sensor extrinsic parameters can indicate the positional relationship between the sensors, such as the angle between one camera and another, the orientation and distance of a camera relative to the lidar, etc.
[0082] In step 1022, feature fusion processing is performed on each modal feature based on the sensor external parameters to obtain a multi-modal fusion feature corresponding to the target vehicle.
[0083] Here, the fusion network can be used to fuse the features of each modal feature according to the positional relationship between each sensor to obtain the multimodal features corresponding to the target vehicle. The multimodal fusion features can reflect the relevant features of all environmental information during the driving process of the target vehicle. For example, a sensor can cover an area of 90 degrees, then 4 sensors can be installed at different positions on the target vehicle to obtain relevant features covering 360 degrees, which can provide data support for the subsequent improvement of the map data processing effect.
[0084] In some embodiments, see Figure 6 , Figure 6 is a flowchart of a map data processing method provided in an embodiment of the present application, Figure 5 Step 1022 in can be Figure 6 Steps 10221 to 10223 in the above example are used to implement:
[0085] In step 10221, based on the positional relationship between the sensors indicated by the sensor external parameters, each modal feature is mapped to the same global three-dimensional space to obtain the global features of each modality.
[0086] Here, since the spatial differences of map data of different modalities are large and the features of each modality are relatively independent, it is necessary to establish a unified global three-dimensional space to achieve the fusion of features of each modality. The global three-dimensional space can be a bird's-eye view space of the target vehicle. The bird's-eye view space refers to the target vehicle as the origin of the coordinate system, the forward direction of the target vehicle as the Y axis, and the direction perpendicular to the Y axis as the X axis. It is used to describe the perspective of the target vehicle's perception of the world from top to bottom. Through the global three-dimensional space, features from different modalities can be easily fused under a unified representation, and powerful and consistent feature representations can be learned from multimodal map data from different sensors such as cameras and lidars, providing strong feature data support for subsequent perception.
[0087] Features of different modes can be mapped to the global three-dimensional space using different methods. For example, image features can be mapped to the global three-dimensional space through inverse perspective transformation (IPM) based on the approximate assumption, that is, all objects are considered to be approximately on a plane; and the radar point cloud itself is three-dimensional information, so the mapping can be achieved by using the transformation relationship between the point cloud perspective and the global three-dimensional space perspective.
[0088] In some embodiments, multimodal map data includes images and point clouds, and the features of each modality can be mapped to the same global three-dimensional space in the following manner to obtain the global features of each modality: for image-type map data (i.e., images) in the multimodal map data, the features of the image are mapped to the global three-dimensional space based on the transformation relationship between the coordinate system of the image and the coordinate system of the global three-dimensional space to obtain the global features of the image; in addition, the features of the image can also be mapped to the global three-dimensional space based on a deep learning network to obtain the global features of the image, wherein the deep learning network includes but is not limited to convolutional neural networks, multilayer perceptrons (MLP, Multilayer Perceptron), etc.
[0089] For point clouds in multimodal map data, subspaces are divided along the three-dimensional space dimension according to the distribution of the target perception area to obtain multiple subspace grids, and feature abstraction is performed on the point cloud features in each subspace network in the multiple subspace grids. The results of feature abstraction are combined to obtain the global features of the point cloud, where the target perception area represents the range that currently needs to be perceived. Feature abstraction can use methods such as feature averaging and / or maximum pooling, and finally the features of all subspaces are used to form the global features of the point cloud data.
[0090] Alternatively, an adjacency graph is established based on the spatial distribution of the point cloud, and a graph convolutional deep network is used to perform feature aggregation on the point cloud features on the adjacency graph, and the position of each data point of the point cloud is projected into the global three-dimensional space, so as to obtain the global features of the point cloud based on the results of position and feature aggregation; wherein the vertices of the adjacency graph are the points in the point cloud, the weights of the edges of the adjacency graph are defined by the distances between the point cloud points, the features of the points in the adjacency graph include information such as the position and reflectivity of the point cloud points, and the results of feature aggregation are calculated according to the projection position of the position of each data point in the global three-dimensional space, so as to obtain the global features of the point cloud in the global three-dimensional space.
[0091] In step 10222, the grid position corresponding to each modal global feature in the global three-dimensional space is determined; in step 10223, the modal global features falling at the same position are aggregated to obtain the multimodal fusion features corresponding to the target vehicle.
[0092] Here, the global three-dimensional space can be divided into grids according to actual needs to obtain a certain number of grids. When each modal feature is mapped to the global three-dimensional space to obtain each modal global feature, the corresponding grid position of each modal global feature in the global three-dimensional space can be determined, and the modal global features falling at the same position can be aggregated. For example, feature stacking or aggregation based on the attention mechanism can be used to obtain multimodal fusion features corresponding to the target vehicle; in this way, multimodal feature fusion is realized by grid-based feature aggregation, and feature aggregation in different grids can be calculated in parallel, which can improve data processing efficiency.
[0093] In step 103, based on the multimodal fusion features, autoregressive iterative processing is performed through the first language model to obtain global perception map features corresponding to the target vehicle.
[0094] In some embodiments, the global perception map features corresponding to the target vehicle can be obtained by performing autoregressive iterative processing through the first language model based on the multimodal fusion features in the following manner: for each round of iteration of the autoregressive iterative processing, the previous round of iteration results output by the first language model are obtained, wherein the previous round of iteration results are obtained by performing autoregressive iterative processing through the first language model based on the multimodal fusion features corresponding to the target vehicle at the previous moment; combining the previous round of iteration results and the multimodal fusion features corresponding to the target vehicle at the current moment, performing autoregressive iterative processing through the first language model, and using the output of this round of iteration results as the global perception map features corresponding to the target vehicle at the current moment.
[0095] Among them, the first language model can be a neural network model that iterates based on the self-attention method, such as a neural network model represented by a transformer, or other types of neural network models. The embodiment of the present application does not limit the first language model. In each round of iteration of the autoregressive iterative processing, taking the current round of iteration corresponding to the current moment as an example, the first language model simultaneously inputs the previous round of iteration results output in the previous round (such as the global perception map features corresponding to the target vehicle at the previous moment) and the multimodal fusion features corresponding to the target vehicle at the current moment into the first language model for iteration, and obtains the global perception map features corresponding to the target vehicle at the current moment.
[0096] Since the multimodal fusion features corresponding to the target vehicle are obtained by fusing relevant features of multimodal or multi-dimensional map data, the multimodal fusion features can also be called feature representations of map perception information, that is, a map representation of the target vehicle's driving path during driving, such as specific locations such as lane lines, signs, and red lights. The multimodal fusion features are input into the first language model for autoregressive iterative processing, and the map representation of the target vehicle's driving path during the entire driving process is updated to obtain a complete global perception map representation of the target vehicle in the entire driving path, that is, the global perception map features corresponding to the target vehicle. In this way, during the iteration process, even if a certain map perception information is missing, it can be corrected by other or subsequent perception results, so that a more accurate global perception map representation is finally obtained.
[0097] In some embodiments, the first language model is implemented based on a fusion training framework, wherein the fusion training framework includes an encoding network, a fusion network, a first language model, and a decoding network; see Figure 7 , Figure 7 This is a schematic diagram of the model training provided in the embodiment of the present application, which can be Figure 7 Steps 201 to 207 in the training process are as follows:
[0098] In step 201, for each vehicle sample in the vehicle sample set, multimodal map data collected by multiple sensors during the driving process of each vehicle sample and global map data corresponding to each vehicle sample are obtained.
[0099] As an example, a vehicle sample in a vehicle sample set is a sample having corresponding multimodal map data during driving and having marked global map data, such as multimodal map data collected in real time by multiple sensors when the vehicle sample is driving from location 1 to location 2, and the global map data is a perceived map of the driving path of the vehicle sample when it is driving from location 1 to location 2 (i.e., the ground truth of the driving path).
[0100] In step 202, feature extraction processing is performed on multimodal map data collected by multiple sensors during the driving of each vehicle sample through an encoding network to obtain each modal feature corresponding to each vehicle sample.
[0101] Among them, the method of performing feature extraction and processing on the multimodal map data collected by multiple sensors during the driving of each vehicle sample is the same as the above-mentioned method of performing feature extraction and processing on the multimodal map data collected by multiple sensors during the driving of the target vehicle, and will not be repeated here.
[0102] In step 203, feature fusion processing is performed on each modal feature corresponding to each vehicle sample through a fusion network to obtain a multimodal fusion feature corresponding to each vehicle sample.
[0103] Among them, the method of performing feature fusion processing on each modal feature corresponding to each pair of vehicle samples is the same as the above-mentioned method of performing feature fusion processing on each modal feature corresponding to the target vehicle, and will not be repeated here.
[0104] In step 204, based on the multimodal fusion features corresponding to each vehicle sample, an autoregressive iterative process is performed through the first language model to obtain a global perception map feature corresponding to each vehicle sample.
[0105] Among them, the feature fusion processing of each modal feature corresponding to each vehicle sample is based on the multimodal fusion features corresponding to each vehicle sample, and the autoregressive iterative processing method is performed through the first language model. This is the same as the above-mentioned feature fusion processing of each modal feature corresponding to the target vehicle is based on the multimodal fusion features corresponding to each vehicle sample, and the autoregressive iterative processing method is performed through the first language model, and will not be repeated here.
[0106] In step 205, the global perception map features corresponding to each vehicle sample are decoded through a decoding network to obtain global perception map data corresponding to each vehicle sample.
[0107] Here, the global perception map features corresponding to each vehicle sample are decoded through the decoding network, and the global perception map data corresponding to each vehicle sample is predicted, that is, the perception map of the complete driving path corresponding to each vehicle sample is obtained (that is, the ground truth of the entire driving path).
[0108] In step 206 , based on the global map data of each vehicle sample and the global perception map data corresponding to each vehicle sample, a loss function of the fusion training framework is determined.
[0109] Here, the entire fusion training framework is constructed based on the difference between the global map data marked by each vehicle sample and the predicted global perception map data corresponding to each vehicle sample, wherein the embodiment of the present application is not limited to the form of the loss function, for example, it can be a cross entropy loss function, L2 loss function, etc.
[0110] In step 207, the parameters of the fusion training framework are updated based on the loss function to obtain an updated fusion training framework, and the first language model in the updated fusion training framework is used as a model for autoregressive iterative processing of the multimodal fusion features corresponding to the target vehicle.
[0111] In practical applications, the fusion training framework provided in the embodiment of the present application is essentially a neural network model. After determining the loss function of the fusion training framework, it can be determined whether the value of the loss function exceeds the preset threshold. When the value of the loss function exceeds the preset threshold, the error signal of the fusion training framework is determined based on the loss function, the error information is back-propagated in the fusion training framework, and the model parameters of each network or each layer are updated during the propagation process.
[0112] Here, back propagation is explained. The training sample data is input into the input layer of the neural network model, passes through the hidden layer, and finally reaches the output layer and outputs the result. This is the forward propagation process of the neural network model. Since there is an error between the output result of the neural network model and the actual result, the error between the output result and the actual value is calculated, and the error is backpropagated from the output layer to the hidden layer until it propagates to the input layer. In the process of back propagation, the value of the model parameter is adjusted according to the error, that is, the loss function is constructed according to the error between the output result and the actual value, and the partial derivatives of the loss function with respect to the model parameters are calculated layer by layer to generate the gradient of the loss function with respect to the model parameters of each layer. Since the direction of the gradient indicates the direction of error expansion, the gradient of the model parameters is inverted and summed with the original parameters of each layer model. The summed result is used as the updated model parameters of each layer, thereby reducing the error caused by the model parameters, and the above process is continuously iterated until convergence.
[0113] In this way, training the first language model separately not only helps to improve the performance and training efficiency of the entire map data processing network, but also allows the selection of the ground truth information that is desired to be obtained during the training process, which is controllable. At the same time, the global perception map data (i.e., the map representation of the driving path) finally output by the fusion training framework can be regarded as a map representation of the driving path. Figure 1 The map also includes a bird's-eye view of information such as red lights and lane line locations, allowing users to perceive the acquired map and improving the interpretability of the first language model.
[0114] In step 104, the map data processing intention is obtained, and based on the global perception map features and the map data processing intention, the second language model is used to perform autoregressive iterative processing to obtain target map data corresponding to the map data processing intention.
[0115] In some embodiments, the target map data corresponding to the map data processing intent can be obtained by performing autoregressive iterative processing through a second language model based on global perception map features and map data processing intent in the following manner: for each round of iteration of the autoregressive iterative processing, obtaining the previous round of iteration results output by the second language model, wherein the previous round of iteration results are obtained by performing autoregressive iterative processing through the second language model based on the global perception map features and map data processing intent corresponding to the target vehicle at the previous moment; combining the previous round of iteration results and the global perception map features corresponding to the target vehicle at the current moment, performing autoregressive iterative processing through the second language model, and using the output of this round of iteration results as the target map data corresponding to the previous round of iteration results at the current moment.
[0116] Among them, the second language model can be a neural network model that iterates based on the self-attention method, such as a neural network model represented by a transformer, or it can be other types of neural network models. The embodiment of the present application does not limit the second language model.
[0117] In actual applications, users can express map data processing intentions by inputting corresponding instructions, where map data processing intentions are used to indicate the goals of map data processing, such as map data processing intentions are used to indicate the generation of customized target maps, or to update the map to be processed (such as updating the standard definition map to a high definition map if the map to be processed is a standard definition map, or pointing out which map data in the map to be processed is insufficient, such as occlusion or low resolution, and correcting the deficiencies), or accepting target indicators of the map to be processed (such as positioning accuracy indicators), etc.
[0118] In the first round of iteration, the map data processing intention initially input by the user and the global perception map features are input into the second language model for autoregressive iterative processing, and the updated map data processing intention is output. The updated map data processing intention is input into the second language model again for the second round of iteration. By analogy, in the current round of iteration of the autoregressive iterative processing, the previous round of iteration results output in the previous round (such as the map data processing intention corresponding to the target vehicle at the previous moment) and the global perception map features corresponding to the target vehicle at the current moment are input into the second language model for autoregressive iterative processing, and the output current round of iteration results are used as the target map data corresponding to the target vehicle at the current moment (i.e., the target map data obtained by the current round of iteration update). If the autoregressive iterative processing performs N (positive integer) iterations, the Nth output iteration result is used as the final target map data for the initially input map data processing intention.
[0119] In some embodiments, see Figure 8 , Figure 8 is a flowchart of a map data processing method provided in an embodiment of the present application, Figure 4 In step 104, based on the global perception map features and the map data processing intention, the second language model is used to perform autoregressive iterative processing to obtain the target map data pseudo props corresponding to the map data processing intention, which can be obtained by Figure 8 Steps 1041 to 1042 in the above embodiment are used to implement:
[0120] In step 1041, when the map data processing intention is used to indicate that a target operation is to be performed on the map to be processed, the map to be processed is encoded to obtain encoding features of the map to be processed.
[0121] The map to be processed is the map that the user wants to modify or update, also known as the map base map. The map to be processed is encoded through the encoding network to obtain the encoding features of the map to be processed, and the encoding features are subsequently processed by autoregression iteration through the second language model. The encoding network can be a neural network model based on weight parameters, including but not limited to convolutional neural networks, recurrent neural networks, Transformer, etc.
[0122] In step 1042, based on the coding features of the map to be processed, the global perception map features corresponding to the target vehicle and the map data processing intent, autoregressive iterative processing is performed through the second language model to obtain the target map data corresponding to the map data processing intent.
[0123] The target map data is the map data obtained after performing a target operation on the map to be processed.
[0124] In some embodiments, the target map data corresponding to the map data processing intent can be obtained by performing autoregressive iterative processing through a second language model based on the coding features of the map to be processed, the global perception map features corresponding to the target vehicle, and the map data processing intent in the following manner: the coding features of the map to be processed and the global perception map features corresponding to the target vehicle are fused to obtain fused coding features; the fused coding features and the map data processing intent are input into the second language model for autoregressive iterative processing to obtain the target map data corresponding to the map data processing intent.
[0125] Here, after the coded features of the map to be processed and the global perception features corresponding to the target vehicle are fused through a fusion network to obtain fused coded features, the second language model is trained to develop language interaction capabilities according to the user's map data processing intentions (i.e., user instructions) based on the information of the map to be processed and the relevant information during the driving process of the target vehicle included in the fused coded features, and then outputs new map data according to the user's map processing intentions, i.e., target map data that meets the map data processing intentions, such as high-definition map data, accuracy assessment data, etc., which can improve data processing accuracy.
[0126] In some embodiments, the target map data corresponding to the map data processing intent can be obtained by performing autoregressive iterative processing through a second language model based on the coding features of the map to be processed, the global perception map features corresponding to the target vehicle, and the map data processing intent in the following manner: the coding features of the map to be processed, the global perception map features corresponding to the target vehicle, and the map data processing intent are directly input into the second language model for autoregressive iterative processing to obtain the target map data corresponding to the map data processing intent.
[0127] Here, the coded features of the map to be processed and the global perception map features corresponding to the target vehicle are directly input into the second language model, and autoregressive iterative processing is performed in combination with the map data processing intention, which can improve the data processing efficiency.
[0128] In some embodiments, the second language model can be trained in the following manner: sampling a first number of instruction samples from an instruction sample set, and labeling the first number of instruction samples to obtain corresponding instruction labels; taking the first number of instruction samples as input and the corresponding instruction labels as output, training to obtain the second language model; sampling a second number of instruction samples from the instruction sample set, and taking the second number of instruction samples as input, outputting output instructions corresponding to the second number of instruction samples from the second language model; sorting the output instructions, and training to obtain a reward model based on the sorting result; sampling a third number of instruction samples from the instruction sample set, and taking the third number of instruction samples as input, and optimizing the second language model using reinforcement learning in combination with the reward model obtained through training.
[0129] In practical applications, the intention of map data processing can be expressed through the instructions input by the user. The second large language model is a pre-trained language model based on the Transformer architecture, which is trained using reinforcement learning from human feedback, that is, the model is trained through human feedback on the generated text (i.e., manually annotated data), so that the text generated by the model (i.e., the output target map data) is more in line with human expectations. The training method of the second large voice model is as follows: first, a first number (which can be set according to actual conditions) of instruction samples are sampled from a large number of instruction sample sets containing real human intentions as input data, and full-time annotators are hired to annotate the outputs of these instruction samples (i.e., the corresponding instruction annotations) to obtain a supervised training data set, and the obtained supervised training data set is used for supervised model training to obtain a supervised model (i.e., the second language model). Then, a second number (which can be set according to actual conditions) of instruction samples are sampled from the instruction sample set as input data again, and multiple different results are output from the trained supervised model (i.e., the second language model), and full-time annotators are hired to annotate the order of the output results, such as output D> output C> output A> output B. After obtaining the order of the output results of these manually annotated results, a reward model is trained. Finally, some new third number (which can be set according to actual conditions) of instruction samples are sampled from the instruction sample set as input data, and combined with the reward model, the second language model is optimized using reinforcement learning (such as a proximal policy optimization algorithm), and the reward model is optimized to better supervise the performance of the second language model until the final target map data meets expectations.
[0130] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0131] The following will describe an exemplary application of the present application embodiment in a practical application scenario. The map data processing method provided in the present application embodiment can be implemented by a map data processing model, see Fig. 9 , Fig. 9 is a schematic diagram of the composition of the map data processing model provided in the embodiment of the present application, including encoder (Encoder) A (i.e. the encoding network mentioned above), fusion network, first language model, encoder B (i.e. the encoding network mentioned above), decoder (Decoder) B (i.e. the fusion network mentioned above), second language model, which will be combined with Fig. 9 The map data processing method provided in the embodiment of the present application is described.
[0132] First, multimodal map data collected by multiple sensors of the target vehicle during driving (such as images collected by on-board sensors, lidar point cloud data, millimeter-wave radar data, GNSS signals, etc.) are obtained, and the multimodal map data is input into encoder A for feature extraction operation to obtain features of each modality; wherein encoder A is a neural network model based on weight parameters, including but not limited to convolutional neural network, recurrent neural network, Transformer, etc.; in this way, feature extraction of multimodal map data through a neural network can improve feature extraction efficiency.
[0133] In the second step, the sensors that collect the map data of each modality are registered with external parameters to obtain sensor external parameters, so as to calibrate each sensor to the world coordinate system. That is, the sensor external parameters can indicate the positional relationship between each sensor, such as the angle between one camera and another camera, the orientation and distance of a camera relative to the laser radar, etc. After obtaining the sensor external parameters, the features of each modality are fused according to the positional relationship between each sensor to obtain the multimodal features corresponding to the target vehicle, wherein the multimodal fusion features can reflect the relevant features of all environmental information during the driving process of the target vehicle. For example, if a sensor can cover a 90-degree area, then 4 sensors can be installed at different orientations on the target vehicle to obtain relevant features including 360-degree orientations, which can provide data support for the subsequent improvement of the map data processing effect. It can be understood that in actual applications, the sensor external parameters and each modal feature can also be directly input into the subsequent network for subsequent processing, and the multimodal features obtained by fusing each modal feature are subsequently input into the subsequent network, which can save the burden of the subsequent network and improve the information expression ability of the subsequent network.
[0134] The third step is to perform autoregressive iterative processing based on the multimodal fusion features through the first language model to obtain the global perception map features corresponding to the target vehicle; wherein the first language model can be a neural network model that iterates based on the self-attention method, such as a neural network model represented by a transformer, or other types of neural network models, and the embodiment of the present application does not limit the first language model. In each round of iteration of the autoregressive iterative processing, taking the current round of iteration corresponding to the current moment as an example, the first language model simultaneously inputs the previous round of iteration results output in the previous round (such as the global perception map features corresponding to the target vehicle at the previous moment) and the multimodal fusion features corresponding to the target vehicle at the current moment into the first language model for iteration, and obtains the global perception map features corresponding to the target vehicle at the current moment.
[0135] Since the multimodal fusion features corresponding to the target vehicle are obtained by fusing relevant features of multimodal or multi-dimensional map data, the multimodal fusion features can also be called feature representations of map perception information, that is, a map representation of the target vehicle's driving path during driving, such as specific locations such as lane lines, signs, and red lights. The multimodal fusion features are input into the first language model for autoregressive iterative processing, and the map representation of the target vehicle's driving path during the entire driving process is updated to obtain a complete global perception map representation of the target vehicle in the entire driving path, that is, the global perception map features corresponding to the target vehicle. In this way, during the iteration process, even if a certain map perception information is missing, it can be corrected by other or subsequent perception results, so that a more accurate global perception map representation is finally obtained.
[0136] In addition, we can see that there is a dotted branch, which is to decode the global perception map features corresponding to the target vehicle through decoder A (i.e., the decoding network mentioned above), and predict the perception map of the complete driving path corresponding to the target vehicle (i.e., the ground truth of the entire driving path); but this step is not included in the final map data processing application, but the first language model is trained separately, which not only helps to improve the performance and training efficiency of the entire map data processing network, but also can select the ground truth information that is desired to be obtained during the training process, which is controllable. At the same time, the global perception map data (i.e., the map representation of the driving path) finally output by this dotted branch can be regarded as a map representation of the driving path. Figure 1 The map also includes a bird's-eye view of information such as red lights and lane line locations, allowing users to perceive the acquired map and improving the interpretability of the first language model.
[0137] The first language model provided in the embodiment of the present application is an autoregressive model, and is not implemented using a simple feature extractor or encoder. This is to enable control of the map generation process and output goals during the process of generating perceptual map encoding (or map drawing process), that is, to provide a human interaction interface, thereby improving convenience and controllability.
[0138] In the fourth step, the map base map that we want to correct or update (i.e., the map to be processed mentioned above) is passed to the encoder (Encoder) B, and the map base map is encoded by the encoder B to obtain the corresponding map encoding features, which provides a ground truth for the subsequent process of correcting or updating the map. It should be noted that the input of this step can be empty, because in theory, the map data processing method provided in the embodiment of the present application can update the temporarily blank areas in the existing map according to the perceived map, so this ability should also be valid when the entire existing map is blank, that is, the map data processing method provided in the embodiment of the present application has both the ability to draw maps from 0 to 1 and from 90 to 100.
[0139] In the fifth step, the global perception map features corresponding to the target vehicle obtained in the third step and the map encoding features of the existing map base map obtained in the fourth step are fused through Decoder B to form a unified representation for input into the second language model. The significance of this step can be compared to giving the second language model a map encoding (i.e., the fused encoding features mentioned above). This map encoding contains two maps (one is the map perceived by the target vehicle during driving, and the other is the existing map base map). The second language model needs to output new text information (i.e., target map data, such as map data of high-definition maps, accuracy assessment, specific drawing content, etc.) according to the information of the two maps in this map encoding and the language interaction ability trained by the human instruction process.
[0140] It can be understood that in practical applications, Decoder B can also concatenate the global perception map features corresponding to the target vehicle obtained in the third step and the map encoding features of the existing map base map obtained in the fourth step, and input the concatenation result into the subsequent second language model, or the global perception map features corresponding to the target vehicle obtained in the third step and the map encoding features of the existing map base map obtained in the fourth step can be respectively input into different self-attention heads; in addition, Decoder B can also be omitted, that is, the global perception map features corresponding to the target vehicle obtained in the third step and the map encoding features of the existing map base map obtained in the fourth step are directly input into the subsequent second language model, but this will put higher requirements on the scale and training process of the second language model.
[0141] In the sixth step, the fused coding features obtained in the fifth step are input into the second language model (the purpose of which is to provide the ability of human language interaction and output in a specified format) in coordination with the user instructions (i.e., the map data processing intention mentioned above) for autoregressive iterative processing to obtain the target map data that meets the user instructions (such as high-definition map data, accuracy assessment, specific drawing content, etc.).
[0142] The training of the second language model also relies on the human feedback reinforcement learning mechanism, see Fig.10 , Fig.10: is a training diagram of the second language model provided in an embodiment of the present application. During training, the first step is to collect training data (i.e., a set of instruction samples) and formulate a supervision strategy, sampling a certain number of instruction samples as input data from a large number of instruction sample sets containing real human intentions, and hiring full-time labelers to label the outputs of these instruction samples (i.e., full-time labelers write output behaviors that meet expectations, such as giving an updated map or pointing out where there are problems with the map base map), obtaining a supervised training data set, and using the obtained supervised training data set to perform supervised model training to obtain a supervised model (i.e., a second language model); the second step is to collect comparison data and Training the reward model, that is, sampling a certain number of instruction samples from the instruction sample set as input data, outputting multiple different results from the trained supervised model (i.e., the second language model), and hiring full-time labelers to label the order of these output results, such as output D>output C>output A>output B. After obtaining the order of these manually labeled output results, a reward model is trained; the third step is to then sample some new third number (which can be set according to actual conditions) of instruction samples from the instruction sample set as input data, and combine with the reward model, use reinforcement learning methods (such as using the proximal policy optimization algorithm (PPO, Proximal Policy Optimization)) to optimize the second language model, and at the same time optimize the reward model to better urge the optimization of the performance of the second language model until the map data processing model meets expectations.
[0143] It should be noted that the map data processing method provided in the embodiment of the present application can be run on the terminal (on a motor vehicle), or it can be collected by sensors and then uploaded or copied to the cloud for processing. When humans interact with the map data processing model, the interaction method can be diverse, such as through voice or typing, or even through multimedia methods such as translating drawn images into structured language (that is, marking on an existing map, and converting the marked map into a map format input) to interact with the map data processing model, such as pointing out which parts require the help of human operators, or which input data may be insufficient (such as occlusion or low resolution, etc.), or directly modifying the map data according to human instructions, or accepting the target indicators of existing maps (such as positioning accuracy indicators), etc.
[0144] Through the above-mentioned method, the embodiment of the present application implements the map data processing method by using a large autoregressive model that can input multimodal map data such as images, point clouds, and maps, and integrates the human natural language understanding and interaction capabilities provided by human feedback reinforcement learning, which can at least achieve the following beneficial effects:
[0145] 1) End-to-end update mode, such as inputting information such as images, lidar point clouds and basic map information, combined with basic maps or map base maps, to output the required target map data according to given instructions, such as maps with higher accuracy or richer elements;
[0146] 2) The only links that rely on manual work are command input and command interaction. The map data processing model itself is a culmination of knowledge, experience, and models in the field of maps, and its output is efficient and stable.
[0147] 3) It can understand customized requirements in human language (such as outputting maps with preset accuracy, without speed limit information, etc.), obtain better quality maps, and customize the maps you want without the need to develop special algorithms or retrain personnel. This significantly improves map production efficiency, flexibility and reliability while reducing map generation costs.
[0148] 4) It is highly robust and can perform global optimization through autoregression to reduce accuracy requirements. It can also eliminate interference from some complex environments through its own knowledge representation capabilities.
[0149] So far, the map data processing method provided by the embodiment of the present application has been described in combination with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. The following will continue to describe the cooperation of various modules in the map data processing device 555 provided by the embodiment of the present application to implement the map data processing solution.
[0150] The feature extraction module 5551 is used to perform feature extraction processing on the multimodal map data collected by multiple sensors during the driving process of the target vehicle, so as to obtain each modal feature corresponding to the target vehicle; the feature fusion module 5552 is used to perform feature fusion processing based on the each modal feature, so as to obtain the multimodal fusion feature corresponding to the target vehicle; the first iteration module 5553 is used to perform autoregressive iterative processing through a first language model based on the multimodal fusion feature, so as to obtain the global perception map feature corresponding to the target vehicle; the second iteration module 5554 is used to obtain the map data processing intent, and based on the global perception map feature and the map data processing intent, perform autoregressive iterative processing through a second language model, so as to obtain the target map data corresponding to the map data processing intent.
[0151] In some embodiments, the feature extraction module is also used to perform feature extraction processing on multimodal map data collected by multiple sensors during the driving of the target vehicle using the neural network corresponding to each modality, so as to obtain each modal feature corresponding to the target vehicle.
[0152] In some embodiments, the feature fusion module is further used to obtain sensor extrinsic parameters indicating the positional relationship between each of the sensors; based on the sensor extrinsic parameters, feature fusion processing is performed on the modal features to obtain multimodal fusion features corresponding to the target vehicle.
[0153] In some embodiments, the feature fusion module is also used to map the modal features into the same global three-dimensional space respectively to obtain global features of each modality; determine the grid positions corresponding to the global features of each modality in the global three-dimensional space based on the positional relationship between the sensors indicated by the sensor external parameters; aggregate the global features of each modality that fall at the same position to obtain multimodal fusion features corresponding to the target vehicle.
[0154] In some embodiments, the feature fusion module is also used to map the features of the image to the global three-dimensional space according to the transformation relationship between the coordinate system of the image and the coordinate system of the global three-dimensional space, so as to obtain the global features of the image; to divide the point cloud into subspaces along the three-dimensional space dimension according to the distribution of the target perception area to obtain multiple subspace grids, and to perform feature abstraction on the point cloud features in each subspace network in the multiple subspace grids, and to combine the results of the feature abstraction to obtain the global features of the point cloud; or, to establish an adjacency graph according to the spatial distribution of the point cloud, to perform feature aggregation on the point cloud features on the adjacency graph using a graph convolutional deep network, and to project the position of each data point of the point cloud into the global three-dimensional space, so as to obtain the global features of the point cloud according to the position and the results of the feature aggregation.
[0155] In some embodiments, the first iteration module is further used to obtain the previous iteration result output by the first language model for each iteration of the autoregressive iterative processing, wherein the previous iteration result is obtained by performing autoregressive iterative processing on the first language model based on the multimodal fusion features corresponding to the target vehicle at the previous moment; combining the previous iteration result and the multimodal fusion features corresponding to the target vehicle at the current moment, performing autoregressive iterative processing on the first language model, and using the output of this iteration result as the global perception map feature corresponding to the target vehicle at the current moment.
[0156] In some embodiments, the first language model is implemented based on a fusion training framework, which includes an encoding network, a fusion network, the first language model, and a decoding network; the device also includes: a first training module, which is used to obtain, for each vehicle sample in the vehicle sample set, multimodal map data collected by multiple sensors during the driving process of each vehicle sample, and global map data corresponding to each vehicle sample; through the encoding network, feature extraction processing is performed on the multimodal map data collected by multiple sensors during the driving process of each vehicle sample to obtain each modal feature corresponding to each vehicle sample; through the fusion network, feature fusion processing is performed on each modal feature corresponding to each vehicle sample to obtain a multimodal fusion feature corresponding to each vehicle sample. feature; based on the multimodal fusion features corresponding to each of the vehicle samples, autoregressive iterative processing is performed through the first language model to obtain the global perception map features corresponding to each of the vehicle samples; the global perception map features corresponding to each of the vehicle samples are decoded through the decoding network to obtain the global perception map data corresponding to each of the vehicle samples; based on the global map data of each vehicle sample and the global perception map data corresponding to each of the vehicle samples, the loss function of the fusion training framework is determined; based on the loss function, the parameters of the fusion training framework are updated to obtain the updated fusion training framework, and the first language model in the updated fusion training framework is used as the model for autoregressive iterative processing of the multimodal fusion features corresponding to the target vehicle.
[0157] In some embodiments, the second iteration module is further used to obtain the previous round of iteration results output by the second language model for each round of iteration of the autoregressive iterative processing, wherein the previous round of iteration results are obtained by performing autoregressive iterative processing on the second language model based on the global perception map features corresponding to the target vehicle at the previous moment and the map data processing intention; combining the previous round of iteration results and the global perception map features corresponding to the target vehicle at the current moment, performing autoregressive iterative processing on the second language model, and using the output of this round of iteration results as the target map data of the target vehicle at the current moment corresponding to the previous round of iteration results.
[0158] In some embodiments, the second iteration module is further used to encode the map to be processed to obtain the encoding features of the map to be processed when the map data processing intention is used to indicate the execution of a target operation on the map to be processed; based on the encoding features, the global perception map features and the map data processing intention, autoregressive iterative processing is performed through a second language model to obtain target map data corresponding to the map data processing intention, wherein the target map data is map data obtained after the target operation is executed on the map to be processed.
[0159] In some embodiments, the second iteration module is further used to fuse the coding features and the global perception map features to obtain fused coding features; input the fused coding features and the map data processing intent into the second language model for autoregressive iterative processing to obtain target map data corresponding to the map data processing intent.
[0160] In some embodiments, the second iteration module is further used to directly input the encoding features, the global perception map features and the map data processing intention into the second language model for autoregressive iterative processing to obtain target map data corresponding to the map data processing intention.
[0161] In some embodiments, the device also includes: a second training module, which is used to sample a first number of instruction samples from the instruction sample set, and label the first number of instruction samples to obtain corresponding instruction labels; use the first number of instruction samples as input and the corresponding instruction labels as output to train a second language model; sample a second number of instruction samples from the instruction sample set, and use the second number of instruction samples as input to output output instructions corresponding to the second number of instruction samples from the second language model; sort the output instructions, and train a reward model based on the sorting results; sample a third number of instruction samples from the instruction sample set, and use the third number of instruction samples as input, and use reinforcement learning to optimize the second language model in combination with the reward model obtained through training.
[0162] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the map data processing method described in the embodiment of the present application.
[0163] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the map data processing method provided by the embodiment of the present application, for example, Figure 4 The map data processing method shown.
[0164] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.
[0165] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.
[0166] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).
[0167] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.
[0168] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.
Claims
1. A map data processing method, characterized in that: The method comprises: Performing feature extraction processing on multimodal map data collected by multiple sensors during the driving process of the target vehicle to obtain each modal feature corresponding to the target vehicle; Performing feature fusion processing based on the various modal features to obtain multi-modal fusion features corresponding to the target vehicle; Based on the multimodal fusion features, autoregressive iterative processing is performed through a first language model to obtain a global perception map feature corresponding to the target vehicle; A map data processing intention is obtained, and based on the global perception map features corresponding to the target vehicle and the map data processing intention, an autoregressive iterative processing is performed through a second language model to obtain target map data corresponding to the map data processing intention.
2. The method according to claim 1, characterized in that The feature extraction process is performed on the multimodal map data collected by multiple sensors during the driving process of the target vehicle to obtain each modal feature corresponding to the target vehicle, including: The neural network corresponding to each mode is used to perform feature extraction processing on the multimodal map data collected by multiple sensors during the driving process of the target vehicle to obtain the features of each mode corresponding to the target vehicle.
3. The method according to claim 1, characterized in that The performing feature fusion processing based on the modal features to obtain the multi-modal fusion features corresponding to the target vehicle includes: Acquiring sensor external parameters for indicating the positional relationship between each of the sensors; Based on the sensor external parameters, feature fusion processing is performed on the modal features to obtain multi-modal fusion features corresponding to the target vehicle.
4. The method according to claim 3, characterized in that The step of performing feature fusion processing on each modal feature based on the sensor external parameter to obtain a multi-modal fusion feature corresponding to the target vehicle includes: Based on the positional relationship between the sensors indicated by the sensor external parameters, the modal features are respectively mapped into the same global three-dimensional space to obtain the global features of each modality; Determine the grid positions corresponding to the global features of each modality in the global three-dimensional space; The global features of each mode falling at the same position are aggregated to obtain the multimodal fusion features corresponding to the target vehicle.
5. The method according to claim 4, characterized in that The multimodal map data includes images and point clouds, and the features of each modality are mapped to the same global three-dimensional space to obtain global features of each modality, including: According to a transformation relationship between a coordinate system of the image and a coordinate system of the global three-dimensional space, features of the image are mapped to the global three-dimensional space to obtain global features of the image; According to the target perception area, the distribution of the point cloud is sub-divided along the three-dimensional space dimension to obtain a plurality of sub-space grids, and feature abstraction is performed on the point cloud features in each sub-space network in the plurality of sub-space grids, and the results of the feature abstraction are combined to obtain the global features of the point cloud; or An adjacency graph is established according to the spatial distribution of the point cloud, point cloud features are aggregated on the adjacency graph using a graph convolutional deep network, and the position of each data point of the point cloud is projected into the global three-dimensional space to obtain the global features of the point cloud based on the position and the result of the feature aggregation.
6. The method according to claim 1, characterized in that The method of performing autoregressive iterative processing based on the multimodal fusion features through the first language model to obtain the global perception map features corresponding to the target vehicle includes: For each round of iteration of the autoregressive iterative processing, obtaining a previous round of iteration results output by the first language model, wherein the previous round of iteration results are obtained by performing autoregressive iterative processing on the first language model based on the multimodal fusion features corresponding to the target vehicle at the previous moment; Combining the previous round of iteration results and the multimodal fusion features corresponding to the target vehicle at the current moment, autoregressive iterative processing is performed through the first language model, and the output of this round of iteration results is used as the global perception map features corresponding to the target vehicle at the current moment.
7. The method according to claim 1, characterized in that The first language model is implemented based on a fusion training framework, which includes an encoding network, a fusion network, the first language model, and a decoding network; The method further comprises: For each vehicle sample in the vehicle sample set, obtaining multimodal map data collected by multiple sensors during the driving of each vehicle sample and global map data corresponding to each vehicle sample; Through the encoding network, feature extraction processing is performed on the multimodal map data collected by multiple sensors during the driving of each vehicle sample to obtain each modal feature corresponding to each vehicle sample; Through the fusion network, feature fusion processing is performed on each modal feature corresponding to each of the vehicle samples to obtain a multimodal fusion feature corresponding to each of the vehicle samples; Based on the multimodal fusion features corresponding to each of the vehicle samples, autoregressive iterative processing is performed through the first language model to obtain a global perception map feature corresponding to each of the vehicle samples; Decoding the global perception map features corresponding to each of the vehicle samples through the decoding network to obtain the global perception map data corresponding to each of the vehicle samples; Determining a loss function of the fusion training framework based on the global map data of each vehicle sample and the global perception map data corresponding to each of the vehicle samples; The fusion training framework is subjected to parameter update processing based on the loss function to obtain an updated fusion training framework, and the first language model in the updated fusion training framework is used as a model for performing autoregressive iterative processing on the multimodal fusion features corresponding to the target vehicle.
8. The method according to claim 1, characterized in that The method of performing autoregressive iterative processing based on the global perception map features corresponding to the target vehicle and the map data processing intention through a second language model to obtain target map data corresponding to the map data processing intention includes: For each round of iteration of the autoregressive iterative processing, obtaining a previous round of iteration results output by the second language model, wherein the previous round of iteration results are obtained by performing autoregressive iterative processing on the second language model based on the global perception map features corresponding to the target vehicle at the previous moment and the map data processing intention; Combining the previous round of iteration results and the global perception map features corresponding to the target vehicle at the current moment, autoregressive iterative processing is performed through the second language model, and the output of this round of iteration results is used as the target map data corresponding to the target vehicle at the current moment.
9. The method according to claim 1, characterized in that The method of performing autoregressive iterative processing based on the global perception map features corresponding to the target vehicle and the map data processing intention through a second language model to obtain target map data corresponding to the map data processing intention includes: When the map data processing intention is used to indicate that a target operation is to be performed on a map to be processed, encoding processing is performed on the map to be processed to obtain encoding features of the map to be processed; Based on the coding features of the map to be processed, the global perception map features corresponding to the target vehicle and the map data processing intention, autoregressive iterative processing is performed through a second language model to obtain target map data corresponding to the map data processing intention, wherein the target map data is map data obtained after performing the target operation on the map to be processed.
10. The method according to claim 9, characterized in that The method of performing autoregressive iterative processing based on the coding features of the map to be processed, the global perception map features corresponding to the target vehicle and the map data processing intention through a second language model to obtain target map data corresponding to the map data processing intention includes: Fusing the coding features of the map to be processed and the global perception map features corresponding to the target vehicle to obtain fused coding features; The fused coding features and the map data processing intent are input into a second language model for autoregressive iterative processing to obtain target map data corresponding to the map data processing intent.
11. The method according to claim 9, characterized in that The method of performing autoregressive iterative processing based on the coding features of the map to be processed, the global perception map features corresponding to the target vehicle and the map data processing intention through a second language model to obtain target map data corresponding to the map data processing intention includes: The coding features of the map to be processed, the global perception map features corresponding to the target vehicle and the map data processing intention are directly input into the second language model for autoregressive iterative processing to obtain target map data corresponding to the map data processing intention.
12. The method according to claim 1, characterized in that The method further comprises: Sampling a first number of instruction samples from the instruction sample set, and marking the first number of instruction samples to obtain corresponding instruction markings; Taking the first number of instruction samples as input and the corresponding instruction annotations as output, training to obtain a second language model; Sampling a second number of instruction samples from the instruction sample set, and taking the second number of instruction samples as input, outputting output instructions respectively corresponding to the second number of instruction samples from the second language model; Sorting the output instructions, and training a reward model based on the sorting results; A third number of instruction samples are sampled from the instruction sample set, and the third number of instruction samples are used as input, combined with the reward model obtained through training, to optimize the second language model using a reinforcement learning method.
13. A map data processing device, characterized in that: The device comprises: A feature extraction module is used to perform feature extraction processing on multimodal map data collected by multiple sensors during the driving process of the target vehicle to obtain each modal feature corresponding to the target vehicle; A feature fusion module, used for performing feature fusion processing based on the features of each modality to obtain a multi-modal fusion feature corresponding to the target vehicle; A first iteration module is used to perform autoregressive iterative processing through a first language model based on the multimodal fusion features to obtain a global perception map feature corresponding to the target vehicle; The second iteration module is used to obtain the map data processing intention, and based on the global perception map features and the map data processing intention, perform autoregressive iterative processing through a second language model to obtain target map data corresponding to the map data processing intention.
14. An electronic device, characterized in that: include: A memory for storing computer executable instructions or computer programs; The processor is used to implement the map data processing method according to any one of claims 1 to 12 when executing the computer executable instructions or computer programs stored in the memory.
15. A computer-readable storage medium, characterized in that: Computer executable instructions or computer programs are stored, and when the computer executable instructions or computer programs are executed by a processor, the map data processing method according to any one of claims 1 to 12 is implemented.
16. A computer program product comprising a computer program or computer executable instructions, characterized in that When the computer program or computer executable instruction is executed by a processor, the map data processing method according to any one of claims 1 to 12 is implemented.
Citation Information
Cited By
Visual management method and system based on large model and AR live-action map
CN120372037A
