High-precision vehicle positioning, vectorized map construction and positioning model training method

Through the high-precision vehicle positioning method, the use of environmental and map feature encoding, the problem of low positioning accuracy of autonomous vehicles in complex environments is solved, and accurate positioning in complex environments is achieved.

CN116698051BActive Publication Date: 2024-11-05BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310628177.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-11-05
Estimated Expiration
2043-05-30

AI Technical Summary

Technical Problem

The existing autonomous driving vehicle positioning system has low positioning accuracy in complex environments, especially in the case of satellite signal loss or large errors, and cannot provide continuous and reliable positioning services.

Method used

A high-precision vehicle positioning method is adopted to obtain the vehicle's initial position, multimodal sensor data and map elements, encode environmental characteristics and map features, determine the target position offset, correct the initial position and improve positioning accuracy . The method includes environment coding, map coding and pose solving, and vehicle positioning using vectorized maps and training models.

Benefits of technology

In complex environments, the positioning accuracy of autonomous driving vehicles is improved, the accurate positioning of vehicles is ensured, and the operational safety in complex road environments is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116698051B_ABST
    Figure CN116698051B_ABST
Patent Text Reader

Abstract

The present disclosure provides a method for high-precision vehicle positioning, vectorized map construction and positioning model training, which relates to the field of artificial intelligence technology, especially to the technical fields of autonomous driving, deep learning, computer vision, etc. The implementation scheme is: obtaining the initial posture of the vehicle, the multimodal sensor data of the vehicle and multiple map elements used to locate the vehicle; encoding the multimodal sensor data to obtain environmental features; encoding multiple map elements to obtain map features; based on environmental features and map features, determining the target posture offset for correcting the initial posture; and superimposing the initial posture with the target posture offset to obtain the corrected posture of the vehicle. The present disclosure can improve the positioning accuracy of autonomous driving vehicles.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as autonomous driving, deep learning, and computer vision, and specifically to a high-precision vehicle positioning method and device, a vectorized map construction method and device, a positioning model training method and device, an electronic device, a computer-readable storage medium, a computer program product, and an autonomous driving vehicle. Background Art

[0002] Autonomous driving technology involves multiple aspects such as environmental perception, behavioral decision-making, trajectory planning, and motion control. Relying on the coordinated cooperation of sensors, visual computing systems, and positioning systems, vehicles with autonomous driving functions can operate automatically without the need for driver operation or with only a small amount of driver operation. Accurate positioning of autonomous driving vehicles is an important prerequisite for ensuring the safe and smooth operation of autonomous driving vehicles.

[0003] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0004] The present disclosure provides a high-precision vehicle positioning method and device, a vectorized map construction method and device, a positioning model training method and device, an electronic device, a computer-readable storage medium, a computer program product, and an autonomous driving vehicle.

[0005] According to one aspect of the present disclosure, a vehicle positioning method is provided, comprising: acquiring an initial posture of the vehicle, multimodal sensor data of the vehicle, and a plurality of map elements for positioning the vehicle; encoding the multimodal sensor data to obtain environmental features; encoding the plurality of map elements to obtain map features; determining a target posture offset for correcting the initial posture based on the environmental features and the map features; and superimposing the initial posture with the target posture offset to obtain a corrected posture of the vehicle.

[0006] According to one aspect of the present disclosure, a vectorized map construction method is provided, comprising: acquiring a point cloud in a point cloud map; dividing a projection plane of the point cloud map into a plurality of two-dimensional grids of a first unit size; for any two-dimensional grid among the plurality of two-dimensional grids: extracting a plane in the two-dimensional grid based on a point cloud in a three-dimensional space corresponding to the two-dimensional grid; and storing the plane as a surface element in the vectorized map.

[0007] According to one aspect of the present disclosure, a training method for a positioning model is provided, wherein the positioning model includes an environment encoder, a map encoder and a pose solver, and the method includes: obtaining an initial pose of a sample vehicle, a pose truth value corresponding to the initial pose, multimodal sensor data of the sample vehicle and a plurality of map elements for positioning the sample vehicle; inputting the multimodal sensor data into the environment encoder to obtain environment features; inputting element information of the plurality of map elements into the map encoder to obtain map features; inputting the environment features, the map features and the initial pose into the pose solver, so that the pose solver: performs sampling within a first offset sampling range to obtain a plurality of first candidate pose offsets; for any of the plurality of first candidate pose offsets, A first candidate pose offset is provided to determine a first matching degree between the environmental feature and the map feature under the first candidate pose offset; and based on the first matching degrees corresponding to each of the multiple first candidate pose offsets, a predicted pose offset is determined and output; based on the predicted pose offset and the pose offset true value, a first loss is determined, wherein the pose offset true value is the difference between the pose true value and the initial pose; based on the first matching degrees corresponding to each of the multiple first candidate pose offsets, a second loss is determined, wherein the second loss indicates the difference between the predicted probability distribution of the pose true value and the true probability distribution of the pose true value; based on at least the first loss and the second loss, an overall loss of the positioning model is determined; and based on the overall loss, parameters of the positioning model are adjusted.

[0008] According to one aspect of the present disclosure, a vehicle positioning device is provided, comprising: an acquisition module configured to acquire an initial posture of the vehicle, multimodal sensor data of the vehicle and a plurality of map elements for positioning the vehicle; an environment encoding module configured to encode the multimodal sensor data to obtain environment features; a map encoding module configured to encode the plurality of map elements to obtain map features; a determination module configured to determine a target posture offset for correcting the initial posture based on the environment features and the map features; and a superposition module superimposing the initial posture with the target posture offset to obtain a corrected posture of the vehicle.

[0009] According to one aspect of the present disclosure, a vectorized map construction device is provided, including: an acquisition module, configured to acquire a point cloud in a point cloud map; a division module, configured to divide the projection plane of the point cloud map into a plurality of two-dimensional grids of a first unit size; an extraction module, configured to extract, for any two-dimensional grid of the plurality of two-dimensional grids, a plane in the two-dimensional grid based on a point cloud in a three-dimensional space corresponding to the two-dimensional grid; and a storage module, configured to store the plane as a surface element in the vectorized map.

[0010] According to one aspect of the present disclosure, a training device for a positioning model is provided, wherein the positioning model includes an environment encoder, a map encoder and a pose solver, and the device includes: an acquisition module, configured to acquire an initial pose of a sample vehicle, a pose truth value corresponding to the initial pose, multimodal sensor data of the sample vehicle and a plurality of map elements for positioning the sample vehicle; a first input module, configured to input the multimodal sensor data into the environment encoder to obtain environment features; a second input module, configured to input element information of the plurality of map elements into the map encoder to obtain map features; a third input module, configured to input the environment features, the map features and the initial pose into the pose solver, so that the pose solver: performs sampling within a first offset sampling range to obtain a plurality of first candidate pose offsets; for each of the plurality of first candidate pose offsets, any first candidate pose offset, determine the first matching degree between the environmental feature and the map feature under the first candidate pose offset; and determine and output the predicted pose offset based on the first matching degrees corresponding to each of the multiple first candidate pose offsets; a first determination module, configured to determine a first loss based on the predicted pose offset and the pose offset true value, wherein the pose offset true value is the difference between the pose true value and the initial pose; a second determination module, configured to determine a second loss based on the first matching degrees corresponding to each of the multiple first candidate pose offsets, wherein the second loss indicates the difference between the predicted probability distribution of the pose true value and the true probability distribution of the pose true value; a determination module, configured to determine the overall loss of the positioning model based on at least the first loss and the second loss; and an adjustment module, configured to adjust the parameters of the positioning model based on the overall loss.

[0011] According to one aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the methods described above.

[0012] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the above methods.

[0013] According to one aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions implement any of the above methods when executed by a processor.

[0014] According to one aspect of the present disclosure, an autonomous driving vehicle is provided, comprising the above-mentioned electronic device.

[0015] According to one or more embodiments of the present disclosure, the positioning accuracy of an autonomous driving vehicle can be improved.

[0016] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 A schematic diagram showing an exemplary system in which various methods described herein may be implemented according to some embodiments of the present disclosure;

[0019] Figure 2 A flow chart of a vehicle positioning method according to some embodiments of the present disclosure is shown;

[0020] Figure 3 A flowchart for solving the target posture offset according to some embodiments of the present disclosure is shown;

[0021] Figure 4 A schematic diagram of a vehicle positioning process based on a trained positioning model according to some embodiments of the present disclosure is shown;

[0022] Figure 5 A flowchart of a method for constructing a vectorized map according to some embodiments of the present disclosure is shown;

[0023] Figure 6 A flowchart of a method for training a positioning model according to some embodiments of the present disclosure is shown;

[0024] Figure 7A structural block diagram of a vehicle positioning device according to some embodiments of the present disclosure is shown;

[0025] Figure 8 A structural block diagram of a vectorized map construction device according to some embodiments of the present disclosure is shown;

[0026] Fig. 9 A structural block diagram of a training device for a positioning model according to some embodiments of the present disclosure is shown; and

[0027] Fig.10 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0029] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0030] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0031] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0032] In related technologies, a combined positioning system is usually used to locate autonomous vehicles. The combined positioning system usually includes a global navigation satellite system (GNSS) and an inertial navigation system (INS). Global positioning is achieved by receiving satellite signals through GNSS, and the calibration of positioning information is achieved through IMU. However, in complex road environments, such as tunnels, overpasses, and urban roads with high-rise buildings, satellite signals are often lost or errors are large, resulting in low positioning accuracy of the combined positioning system, which cannot provide continuous and reliable positioning services.

[0033] In response to the above problems, the present disclosure provides a vehicle positioning method that can improve the positioning accuracy of an autonomous driving vehicle.

[0034] The present disclosure also provides a method for constructing a vectorized map and a method for training a positioning model. The constructed vectorized map and the trained positioning model can be used for vehicle positioning to improve the positioning accuracy of the autonomous driving vehicle.

[0035] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0036] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes a motor vehicle 110 , a server 120 , and one or more communication networks 130 coupling the motor vehicle 110 to the server 120 .

[0037] In an embodiment of the present disclosure, the motor vehicle 110 may include an electronic device according to an embodiment of the present disclosure and / or be configured to perform a method according to an embodiment of the present disclosure.

[0038] The server 120 may run one or more services or software applications that enable the execution of the vectorized map construction method or the positioning model training method of the disclosed embodiment. In some embodiments, the server 120 may also provide other services or software applications, which may include non-virtual environments and virtual environments. Figure 1In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. A user of the motor vehicle 110 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0039] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0040] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0041] In some embodiments, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from motor vehicle 110. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of motor vehicle 110.

[0042] The network 130 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including, but not limited to, TCP / IP, SNA, IPX, etc. By way of example only, one or more networks 130 may be a satellite communications network, a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (including, for example, Bluetooth, Wi-Fi), and / or any combination of these and other networks.

[0043] The system 100 may also include one or more databases 150. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 150 may be used to store information such as audio files and video files. The data repository 150 may reside in various locations. For example, the data repository used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The data repository 150 may be of different types. In some embodiments, the data repository used by the server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to a command.

[0044] In some embodiments, one or more of the databases 150 may also be used by an application to store application data. The databases used by the application may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0045] The motor vehicle 110 may include a sensor 111 for sensing the surrounding environment. The sensor 111 may include one or more of the following sensors: a visual camera, an infrared camera, an ultrasonic sensor, a millimeter wave radar, and a laser radar (LiDAR). Different sensors may provide different detection accuracy and range. The camera may be installed at the front, rear, or other location of the vehicle. The visual camera may capture the situation inside and outside the vehicle in real time and present it to the driver and / or passengers. In addition, by analyzing the images captured by the visual camera, information such as traffic light indications, intersection conditions, and the operating status of other vehicles may be obtained. The infrared camera may capture objects under night vision conditions. Ultrasonic sensors may be installed around the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of strong ultrasonic directionality. Millimeter wave radars may be installed at the front, rear, or other location of the vehicle to measure the distance of objects outside the vehicle from the vehicle using the characteristics of electromagnetic waves. Laser radars may be installed at the front, rear, or other location of the vehicle to detect object edges and shape information, thereby identifying and tracking objects. Due to the Doppler effect, the radar device may also measure the speed change of the vehicle and the moving object.

[0046] The motor vehicle 110 may also include a communication device 112. The communication device 112 may include a satellite positioning module capable of receiving satellite positioning signals (e.g., Beidou, GPS, GLONASS, and GALILEO) from satellites 141 and generating coordinates based on these signals. The communication device 112 may also include a module for communicating with a mobile communication base station 142, and the mobile communication network may implement any suitable communication technology, such as GSM / GPRS, CDMA, LTE, and other current or evolving wireless communication technologies (e.g., 5G technology). The communication device 112 may also have a vehicle-to-vehicle or vehicle-to-everything (V2X) module, which is configured to implement, for example, vehicle-to-vehicle (V2V) communication with other vehicles 143 and vehicle-to-infrastructure (V2I) communication with infrastructure 144. In addition, the communication device 112 may also have a module configured to communicate with a user terminal 145 (including but not limited to a smartphone, a tablet computer, or a wearable device such as a watch) by, for example, using a wireless LAN or Bluetooth using the IEEE 802.11 standard. Using the communication device 112, the motor vehicle 110 may also access the server 120 via the network 130.

[0047] The motor vehicle 110 may further include an inertial navigation module. The inertial navigation module and the satellite positioning module may be combined into a combined positioning system to achieve initial positioning of the motor vehicle 110.

[0048] The motor vehicle 110 may also include a control device 113. The control device 113 may include a processor that communicates with various types of computer-readable storage devices or media, such as a central processing unit (CPU) or a graphics processing unit (GPU), or other dedicated processors. The control device 113 may include an automatic driving system for automatically controlling various actuators in the vehicle, and accordingly, the motor vehicle 110 is an automatic driving vehicle. The automatic driving system is configured to control the powertrain, steering system, and braking system of the motor vehicle 110 (not shown) via multiple actuators in response to inputs from multiple sensors 111 or other input devices to control acceleration, steering, and braking, respectively, without human intervention or limited human intervention. Some processing functions of the control device 113 can be implemented through cloud computing. For example, some processing can be performed using an on-board processor, while other processing can be performed using computing resources in the cloud. The control device 113 can be configured to perform a method according to the present disclosure. In addition, the control device 113 can be implemented as an example of an electronic device on the motor vehicle side (client) according to the present disclosure.

[0049] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0050] According to some embodiments, the server 120 may construct a vectorized map by executing the vectorized map construction method of the embodiment of the present disclosure, and train the positioning model by executing the positioning model training method of the embodiment of the present disclosure. The constructed vectorized map and the trained positioning model may be transmitted to the motor vehicle 110. The motor vehicle 110 may use the vectorized map and the positioning model to execute the vehicle positioning method of the embodiment of the present disclosure to achieve accurate positioning of itself.

[0051] According to other embodiments, the vectorized map construction method and the positioning model training method may also be executed by the motor vehicle 110. This generally requires the motor vehicle 110 to have a higher hardware configuration and a stronger computing capability.

[0052] According to some embodiments, the vehicle positioning method may also be executed by the server 120. In this case, the motor vehicle 110 uploads relevant data (including initial position and multimodal sensor data) to the server 120. Accordingly, the server 120 obtains the data uploaded by the motor vehicle 110, and processes the data by executing the vehicle positioning method, thereby accurately positioning the motor vehicle 110.

[0053] The high-precision positioning information obtained by executing the vehicle positioning method of the embodiment of the present disclosure can be applied to tasks such as trajectory planning, behavior decision-making, and motion control of the motor vehicle 110.

[0054] Figure 2 FIG. 2 is a flow chart of a vehicle positioning method 200 according to an embodiment of the present disclosure. As described above, the execution subject of the method 200 may be an autonomous driving vehicle (eg, Figure 1 ), or a server (e.g., a motor vehicle 110 shown in FIG. Figure 1 120 shown in FIG. Figure 2 As shown, the method 200 includes steps S210 - S250 .

[0055] In step S210, an initial position and posture of the vehicle, multimodal sensor data of the vehicle, and a plurality of map elements for locating the vehicle are obtained.

[0056] In step S220 , the multimodal sensor data is encoded to obtain environmental features.

[0057] In step S230, a plurality of map elements are encoded to obtain map features.

[0058] In step S240 , a target posture offset for correcting the initial posture is determined based on the environmental features and the map features.

[0059] In step S250, the initial posture and the target posture offset are superimposed to obtain a corrected posture of the vehicle.

[0060] According to the embodiments of the present disclosure, encoding multimodal sensor data can make full use of the data of each sensor, reduce information loss, and enable environmental features to fully and accurately express the surrounding environment of the vehicle. Determining the target posture offset based on environmental features and map features, and correcting the initial posture based on the target posture offset can improve the positioning accuracy of the vehicle, and can still achieve accurate positioning of the vehicle even in complex environments.

[0061] The following is a detailed description of each step of method 200.

[0062] In step S210, an initial position and posture of the vehicle, multimodal sensor data of the vehicle, and a plurality of map elements for locating the vehicle are obtained.

[0063] The vehicle in step S210 may be a vehicle with an autonomous driving function, that is, an autonomous driving vehicle.

[0064] In the embodiments of the present disclosure, the initial posture refers to an uncorrected posture.

[0065] According to some embodiments, the initial position of the vehicle may be a position output by a combined positioning system of the vehicle. The combined positioning system generally includes a satellite positioning system and an inertial navigation system.

[0066] According to some embodiments, the vehicle may be positioned at a preset frequency (eg, 1 Hz). The initial position at the current moment may be the corrected position at the previous moment.

[0067] The vehicle's posture (including the uncorrected initial posture and the corrected posture) is used to indicate the position and posture of the vehicle. The position of the vehicle can be represented by three-dimensional coordinates such as (x, y, z). The posture of the vehicle can be represented by posture angles, for example. The posture angles further include roll angle (roll or φ), pitch angle (pitch or θ) and yaw angle (yaw or ψ).

[0068] Since the vehicle usually does not leave the ground during driving and does not roll or pitch, in practice, the accuracy of the z coordinate, roll angle, and pitch angle is usually not concerned. Accordingly, in an embodiment of the present disclosure, only the x coordinate, y coordinate, and yaw angle yaw in the initial posture can be corrected, without correcting the z coordinate, roll angle roll, and pitch angle pitch. In other words, the z coordinate, roll angle roll, and pitch angle pitch of the corrected posture are the same as the initial posture, but the x coordinate, y coordinate, and yaw angle yaw may be different from the initial posture.

[0069] Vehicles are usually equipped with a variety of sensors for environmental perception, such as visual cameras, lidars, millimeter-wave radars, etc. Modality refers to the form in which data exists. The data collected by different sensors are usually in different forms, so the data collected by different sensors usually corresponds to different data modalities. For example, the data collected by a visual camera is an image. Multiple visual cameras with different viewing directions can be deployed on a vehicle, and accordingly, multiple images with different viewing angles can be obtained using these visual cameras. The data collected by the lidar is a point cloud. It can be understood that a point cloud usually includes the position coordinates and reflection intensity values ​​of multiple three-dimensional space points.

[0070] The vehicle's multimodal sensor data can express the vehicle's surroundings in different forms, achieving comprehensive perception of the surrounding environment.

[0071] According to some embodiments, a vectorized map may be stored locally in the vehicle or on a server.

[0072] Vector map is a data set that represents geographic elements by using information such as identification, name, location, attributes, and topological relationships between them. Vector map includes multiple geographic elements, each of which is stored in a vector data structure. Vector data structure is a data organization method that uses points, lines, surfaces, and their combinations in geometry to represent the spatial distribution of geographic elements. It expresses the location of elements by recording the coordinates of the elements and their spatial relationships.

[0073] According to some embodiments, the geographic elements in the vectorized map include road elements and geometric elements. Road elements refer to elements with specific semantics in the road, including lane lines, curbs, stop lines, crosswalks, traffic signs, poles, etc. Among them, poles further include tree trunks, traffic sign posts, and street light poles. Geometric elements refer to elements with a certain shape, including surface elements (surfel), line elements, etc. Surface elements represent planes in the physical world, such as the outer surface of a building, the surface of a traffic light, a traffic sign, etc. It should be noted that surface elements can have a certain overlap with road elements. For example, some surface elements are also road elements.

[0074] Road elements are usually sparse. Some sections have few or even no road elements. On sections with few or no road elements, it is difficult to accurately locate the vehicle through road elements. According to the above embodiment, the vectorized map also includes geometric elements such as surface elements. As a supplement to road elements, geometric elements can improve the richness and density of geographical elements in the vectorized map, thereby enabling accurate positioning of the vehicle.

[0075] According to the embodiments of the present disclosure, a vectorized map is used for vehicle positioning. The vectorized map is small in size and easy to update, which reduces storage costs, thereby improving the availability of the vehicle positioning method and meeting mass production requirements.

[0076] According to some embodiments, in a vectorized map, lane lines, curbs, and stop lines are represented as line segments, and the endpoints of the line segments are two-dimensional xy coordinates in a global coordinate system, such as a UTM (Universal Transverse Mercator) coordinate system. A crosswalk is represented as a polygon, and the vertices of the polygon are represented by two-dimensional xy coordinates in a UTM coordinate system. Traffic signs are represented as rectangles perpendicular to the xy plane, and the vertices are three-dimensional UTM coordinates, where the z coordinate is represented by the height relative to the ground. The pole is represented by the two-dimensional xy coordinates in the UTM coordinate system and the height of the pole.

[0077] According to some embodiments, in a vectorized map, a surface element P is represented as in, represents the field of real numbers, Represents the xy coordinates of the surface element P in the UTM coordinate system, represents the unit normal vector of the surface element, λ1≤λ2≤λ3 are singular values ​​of the covariance matrix of the surface elements. The extraction method of the surface elements will be described in detail in the vector map construction method 500 below.

[0078] According to some embodiments, the multiple map elements used for vehicle positioning in step S210 can be obtained by screening multiple geographic elements in the vectorized map based on the initial posture. According to some embodiments, geographic elements located near the initial posture (i.e., the distance to the initial posture is less than a threshold) can be used as map elements for vehicle positioning. For example, geographic elements within a range of 100 meters near the initial posture (i.e., the distance to the initial posture is less than 100 meters) are used as map elements for vehicle positioning.

[0079] According to some embodiments, in order to balance the computational efficiency and the reliability of the positioning result, a preset number of geographic elements whose distance to the initial posture is less than a threshold value can be used as map elements for vehicle positioning. The preset number can be set as needed. For example, the preset number can be set to 100, 500, 1000, etc. If the number of geographic elements near the initial posture is more than the preset number, the nearby geographic elements can be sampled to obtain the preset number of map elements. Further, for road elements, the sampling can be carried out in order from near to far from the initial posture. For surface elements, random sampling can be carried out. Surface elements may correspond to different types of entities in the physical world, such as the outer surface of a building, traffic signs, etc. Different types of surface elements can impose positioning constraints on the vehicle in different directions. For example, the outer surface of a building (parallel to the lane line) can constrain the positioning of the left and right directions of the vehicle, and the traffic signs can constrain the positioning of the forward direction of the vehicle. Random sampling of surface elements can make the sampling results cover various types of surface elements more evenly, thereby ensuring the accuracy of vehicle positioning. If the number of geographic elements near the initial posture is less than the preset number, the geographic elements can be expanded to the preset number by copying the existing geographic elements.

[0080] According to some embodiments, the multimodal sensor data and the plurality of map elements acquired in step S210 may be preprocessed to improve the accuracy of subsequent vehicle positioning.

[0081] Multimodal sensor data may include images and point clouds. According to some embodiments, the image may be subjected to preprocessing operations such as dedistortion, scaling to a preset size, and standardization. According to some embodiments, the point cloud may be screened based on the initial pose, and only the point cloud near the initial pose is retained. For example, only the point cloud with the initial pose as the origin, the vehicle's forward direction (positive direction of the x-axis) [-40m, 40m], the vehicle's left direction (positive direction of the y-axis) [-40m, 40m], and the vehicle's top (positive direction of the z-axis) [-3m, 5m] may be retained. Furthermore, the point cloud may be voxelized, that is, the space may be divided into a plurality of non-intersecting blocks, with a maximum of 32 points retained in each block.

[0082] As described above, the plurality of map elements obtained from the vectorized map include lane lines, curbs, stop lines, crosswalks, traffic signs, poles, and surface elements. According to some embodiments, lane lines, curbs, and stop lines may be broken into line segments of the same length, each of which is represented by a four-dimensional vector The four values ​​in the vector are the xy coordinates of the starting and ending points of the line segment. The traffic sign is represented as The first two values ​​in the vector are the xy coordinates of the center of the card, and the last value in the vector represents the height of the center of the card relative to the ground. The pole is represented by The first two values ​​in the vector are the xy coordinates of the pole, and the last value in the vector represents the height of the pole relative to the ground. For area elements, no preprocessing is required. That is, the representation of area elements can be the same as that in the vectorized map.

[0083] In step S220 , the multimodal sensor data is encoded to obtain environmental features.

[0084] According to some embodiments, as described above, the multimodal sensor data may include a point cloud and an image. Accordingly, step S220 may include steps S221 - S223 .

[0085] In step S221, the point cloud is encoded to obtain a point cloud feature map.

[0086] In step S222, the image is encoded to obtain an image feature map.

[0087] In step S223, the point cloud feature map and the image feature map are fused to obtain environmental features.

[0088] According to the above embodiment, sensor data of different modes are encoded respectively, and the encoding results of each sensor are fused, which can fully retain the original data information of different sensors and reduce information loss while achieving a comprehensive expression of the environment.

[0089] According to some embodiments, for step S221, the point cloud can be encoded as a point cloud feature map in the target three-dimensional space. The target three-dimensional space can be, for example, a bird's eye view (BEV) space of the vehicle. The bird's eye view refers to a bird's-eye view. The bird's eye view space refers to the space in a right-handed rectangular Cartesian coordinate system with the position of the vehicle (i.e., the initial posture) as the origin. In some embodiments, the bird's eye view space can take the position of the vehicle as the origin, the right direction of the vehicle as the positive direction of the x-axis, the forward direction of the vehicle as the positive direction of the y-axis, and the vertical direction above the vehicle as the positive direction of the z-axis. In other embodiments, the bird's eye view space can also take the position of the vehicle as the origin, the forward direction of the vehicle as the positive direction of the x-axis, the left direction of the vehicle as the positive direction of the y-axis, and the vertical direction above the vehicle as the positive direction of the z-axis. The point cloud feature map can be a feature map in the target three-dimensional space.

[0090] According to some embodiments, a trained point cloud encoder may be used to encode the point cloud. The point cloud encoder may be implemented as a neural network.

[0091] According to some embodiments, the point cloud near the vehicle can be divided into multiple cylindrical spaces with a square cross-section (parallel to the xy plane) (for example, 0.5m*0.5m). The point cloud near the vehicle can be, for example, a point cloud in the vehicle's forward direction (positive direction of the x-axis) [-40m, 40m], to the left of the vehicle (positive direction of the y-axis) [-40m, 40m], and above the vehicle (positive direction of the z-axis) [-3m, 5m]. After division, the point cloud near the vehicle will fall into the corresponding cylindrical space. Each cylindrical space is a grid in the BEV space and corresponds to a pixel in the point cloud feature map under the BEV space. The resolution of the point cloud feature map (i.e., the resolution of the BEV space) refers to the length in the physical world corresponding to a single pixel (i.e., a grid in the BEV space), that is, the side length of the cross-section of the above-mentioned cylindrical space, for example 0.5m per pixel.

[0092] Each point in the point cloud can be encoded into a D (D = 9) dimensional vector: (x, y, z, r, xc, yc, zc, xp, yp), where x, y, z, r represent the three-dimensional coordinates and reflection intensity of the point respectively; xc, yc, zc represent the distance from the point to the arithmetic mean point of all points in the cylindrical space where it is located, and xp, yp represent the offset value of the point to the x, y center of the cylindrical space where it is located. Due to the sparsity of point cloud data, many cylindrical spaces may not contain point clouds or contain a relatively small number of point clouds. Considering the problem of computational complexity, it is set that each cylindrical space contains a maximum of N point cloud feature vectors. If the number of point clouds is greater than N, N are selected from them by random sampling; conversely, if the number of point clouds is less than N, they are filled to N by zero filling. According to the above embodiment, the point cloud is encoded as a dense tensor of dimension (D, P, N). Wherein, P represents the number of cylindrical spaces.

[0093] Each D-dimensional vector is linearly mapped to obtain a C-dimensional vector (for example, C=256), thereby mapping the tensor (D, P, N) to the tensor (C, P, N). Further, by performing a pooling operation on (C, P, N), the tensor (C, P) is obtained.

[0094] Each columnar space corresponds to a pixel in the point cloud feature map. The size of the point cloud feature map is H*W*C. Among them, H, W, and C are the height, width, and number of channels of the point cloud feature map, respectively. Specifically, H is the quotient of the point cloud range of the x-axis and the resolution of the point cloud feature map; W is the quotient of the point cloud range of the y-axis and the resolution of the point cloud feature map; C is the dimension of the feature vector corresponding to each pixel. For example, in the above embodiment, the point cloud range of the x-axis and the y-axis are both 80m (ie, [-40m, 40m]), the resolution of the point cloud feature map is 0.5m per pixel, C=256, and accordingly, the point cloud feature map has H=W=80 / 0.5=160, and the size of the point cloud feature map is 160*160*256.

[0095] According to some embodiments, for step S222, a trained image encoder may be used to encode the image. The image encoder may be implemented as a neural network.

[0096] According to some embodiments, the image encoder may include a backbone module Backbone and a multi-layer feature pyramid fusion module. The backbone module Backbone may, for example, use networks such as VoVNet-19, VGG, ResNet, and EfficientNet. The multi-layer feature pyramid fusion module may use a basic top-to-bottom fusion method, such as Feature Pyramid Networks (FPN), or may use networks such as BiFPN and Recursive Feature Pyramid (RFP). The image encoder receives images of different perspectives (for example, 6 perspectives) and generates multi-scale feature maps. The size of the image is H c ×W c ×3. For example, you can set the image size to H c =448,W c =640. The sizes of the last two layers of the multi-scale feature map are, for example, The last two layers of the multi-scale feature map are input into the multi-layer feature pyramid fusion module to obtain an image feature map that integrates multi-scale information. The size of the image feature map can be, for example,

[0097] According to some embodiments, step S223 may include steps S2231 - S2233 .

[0098] In step S2231, an initial environment feature map in the target three-dimensional space is determined based on the point cloud feature map.

[0099] In step S2232, the initial environment feature map is fused with the image feature map to obtain a first environment feature map in the target three-dimensional space.

[0100] In step S2233, environmental characteristics are determined based on the first environmental characteristics map.

[0101] According to the above embodiment, multimodal feature fusion is performed in the target three-dimensional space, which can eliminate the coordinate system differences of different sensors and improve the accuracy of environmental expression.

[0102] As described above, the target three-dimensional space may be a bird's-eye view space of the vehicle.

[0103] According to some embodiments, for step S2231, the point cloud feature map can be used as the initial environment feature map, or the point cloud feature map can be processed in a certain way (such as convolution processing) and the processing result can be used as the initial environment feature map.

[0104] According to some embodiments, in step S2232, the initial environment feature map and the image feature map may be fused at least once based on an attention mechanism to obtain a first environment feature map in the target three-dimensional space. The attention mechanism can capture the correlation between features. According to this embodiment, feature fusion using the attention mechanism can improve the accuracy of feature fusion.

[0105] According to some embodiments, in each of the at least one fusion described above, the following steps S22321 and S22322 are performed.

[0106] In step S22321, based on the self-attention mechanism, the current environment feature map is updated to obtain an updated environment feature map.

[0107] In step S22322, based on the cross-attention mechanism, the updated environmental feature map obtained in step S22321 is fused with the image feature map to obtain a fused environmental feature map.

[0108] It should be noted that the current environment feature map in the first fusion is the initial environment feature map obtained in step S2231. The current environment feature map in the second and subsequent fusions is the environment feature map after the last fusion. For example, the current environment feature map in step S22321 in the second fusion is the fused environment feature map obtained in step S22322 in the first fusion. The environment feature map after the last fusion is used as the first environment feature map in the target three-dimensional space.

[0109] According to some embodiments, for step S22321, the size of the current environment feature map is H*W*C. Wherein H, W, and C are the height, width, and number of channels of the current environment feature map, respectively. In step S22321, based on the self-attention mechanism, the feature vector of each pixel (i, j) in the current environment feature map is updated to obtain an updated feature vector of each pixel. Wherein, 1≤i≤H, 1≤j≤W. The updated feature vectors of each pixel constitute the updated environment feature map. It can be understood that the size of the updated environment feature map is still H*W*C.

[0110] Specifically, for any pixel in the current environment feature map, the feature vector of the pixel can be used as the query vector (Query), and based on the self-attention mechanism, the correlation between the pixel and other pixels (i.e., attention weight) can be obtained. Then, based on the correlation between the pixel and other pixels, the feature vector of the pixel is fused with the feature vectors of other pixels to obtain the updated feature vector of the pixel.

[0111] According to some embodiments, in step S22321, a deformable attention mechanism (DA) can be used to implement the update of the current environment feature map. In this embodiment, for each pixel (i, j) in the current environment feature map, the pixel is used as a reference point. Based on the deformable attention mechanism, the correlation (i.e., attention weight) between the pixel and multiple neighboring pixels near the reference point is determined. Then, based on the correlation between the pixel and the neighboring pixels, the feature vector of the pixel is fused with the feature vector of the neighboring pixels to obtain the updated feature vector of the pixel.

[0112] As described above, after step S22321, an updated environment feature map can be obtained. The updated environment feature map includes an updated feature vector of each pixel.

[0113] According to some embodiments, in step S22322, based on the cross attention mechanism, the updated feature vector of each pixel obtained in step S22321 is fused with the image feature map to obtain a fused environment feature map. It should be noted that the size of the fused environment feature map is still H*W*C.

[0114] Specifically, for any pixel in the updated environment feature map, the updated feature vector of the pixel can be used as the query vector, and based on the cross-attention mechanism, the correlation between the pixel and each pixel in the image feature map (i.e., attention weight) is obtained. Then, based on the correlation between the pixel and each pixel in the image feature map, the updated feature vector of the pixel is fused with the feature vectors of each pixel in the image feature map to obtain the fused feature vector of the pixel.

[0115] According to some embodiments, in step S22322, a deformable attention mechanism can be used to achieve the fusion of feature maps. In this embodiment, for each pixel (i, j) in the updated environmental feature map, the xy coordinates of the pixel in the global coordinate system (e.g., UTM coordinate system) are determined based on the initial posture of the vehicle. A certain number (e.g., 4) of spatial points are sampled at equal intervals in the height direction at the xy coordinates, and these spatial points are mapped to the image feature map using the posture and internal parameters of the visual camera, and the obtained projection points are used as reference points. Based on the deformable attention mechanism, the correlation (i.e., attention weight) between the pixel and multiple neighboring pixels near the reference point is determined. Then, based on the correlation between the pixel and the neighboring pixels, the feature vector of the pixel is fused with the feature vector of the neighboring pixels to obtain the fused feature vector of the pixel, thereby obtaining the fused environmental feature map.

[0116] According to some embodiments, step S2232 may be implemented by a trained first transform decoder (TransformerDecoder). Specifically, the initial environment feature map and the image feature map may be input into the trained first transform decoder to obtain a first environment feature map output by the first transform decoder.

[0117] According to some embodiments, the first transform decoder includes at least one transform layer (Transformer Layer), and each transform layer is used to fuse the environment feature map and the image feature map once.

[0118] Furthermore, each transformation layer may include a self-attention module and a cross-attention module. The self-attention module is used to update the current environment feature map to obtain an updated environment feature map, that is, to implement the above step S22321. The cross-attention module is used to fuse the updated environment feature map with the image feature map to obtain a fused environment feature map, that is, to implement the above step S22322.

[0119] After obtaining the first environment feature map in the target three-dimensional space through step S2232, in step S2233, the environment features can be determined based on the first environment feature map.

[0120] According to some embodiments, the first environmental characteristic map may be used as the environmental characteristic.

[0121] According to some other embodiments, the first environment feature map may be upsampled at least once to obtain at least one second environment feature map corresponding to the at least one upsample; and the first environment feature map and the at least one second environment feature map are determined as environment features. For example, the size of the first environment feature map is 160*160*256, and its resolution is 0.5m per pixel. By upsampling the first environment feature map, a first second environment feature map is obtained, whose size is 320*320*128 and whose resolution is 0.25m per pixel. The first second environment feature map is upsampled to obtain a second second environment feature map, whose size is 640*640*64 and whose resolution is 0.125m per pixel.

[0122] The resolution of the first environmental feature map is usually low. Using only the first environmental feature map as an environmental feature and determining the target pose offset based on it may result in the target pose offset being inaccurate. According to the above embodiment, a second environmental feature map with a higher resolution is obtained by upsampling the first environmental feature map. Using the first environmental feature map and the second environmental feature map as environmental features improves the precision of the environmental features, thereby improving the accuracy of the target pose offset subsequently determined based on the environmental features.

[0123] For the convenience of description, the first environment feature map is recorded as the 0th layer environment feature map, and the second environment feature map obtained by the lth (l=1, 2, 3, ...) upsampling is recorded as the lth layer environment feature map. It can be understood that the larger the number of the environment feature map, the larger its size and resolution.

[0124] In step S230, a plurality of map elements are encoded to obtain map features.

[0125] As described above, the plurality of map elements are obtained by screening the plurality of geographic elements in the vectorized map based on the initial pose. The geographic elements in the vectorized map include road elements and geometric elements, and accordingly, the plurality of map elements obtained by screening also include at least one road element and at least one geometric element. The at least one road element includes any one of a lane line, a curb, a crosswalk, a stop line, a traffic sign or a pole, and the at least one geometric element includes a surface element.

[0126] According to some embodiments, the surface elements are obtained by extracting planes from the point cloud map. The method of extracting the surface elements will be described in detail in the vectorized map construction method 500 below.

[0127] According to some embodiments, step S230 may include steps S231 and S232.

[0128] In step S231, for any map element among the multiple map elements, element information of the map element is encoded to obtain an initial encoding vector of the map element.

[0129] In step S232, the initial coding vector is updated based on the environmental features to obtain the target coding vector of the map element, wherein the map feature includes the target coding vectors of the respective map elements.

[0130] According to some embodiments, the element information of the map element includes location information and category information (ie, semantic information). Accordingly, step S231 may include steps S2311-S2313.

[0131] In step S2311, the position information is encoded to obtain a position code.

[0132] In step S2312, the category information is encoded to obtain a semantic code.

[0133] In step S2313, the position coding and the semantic coding are fused to obtain an initial coding vector.

[0134] According to the above embodiment, the position information and category information of the map elements are encoded respectively, and the encoding results are merged, so that the expression ability of the map elements can be improved.

[0135] According to some embodiments, in step S2311, a trained position encoder may be used to encode the position information. The position encoder may be implemented as a neural network, for example.

[0136] According to some embodiments, as described above, the map elements include road elements and surface elements. The position information of the road elements is represented as a four-dimensional vector, and the position information of the surface elements is represented as a seven-dimensional vector. In order to achieve a better encoding effect, different position encoders can be used to encode the road elements and the surface elements respectively.

[0137] According to some embodiments, a first position encoder may be used to encode the position information of road elements. Road elements include lane lines, curbs, crosswalks, stop lines, traffic signs, and poles. The position information of the i-th road element is represented as Where K hd is the number of road elements used to locate the vehicle. Based on the xy coordinates of the initial pose in the UTM coordinate system O xy =[x o y o ] T and the xy range of the point cloud R xy =[x r y r ]T , according to the following formula (1) the location information of the road element Normalize:

[0138]

[0139] In formula (1), Normalized position information.

[0140] The first position encoder is used to normalize the position information Encode and get the position code Wherein, C is the dimension of the position encoding, which is usually equal to the number of channels of the environment feature map, that is, equal to the dimension of the feature vector of each pixel in the environment feature map. The first position encoder can be implemented as a multi-layer perceptron (MLP). The first position encoder can, for example, be composed of a set of one-dimensional convolutional layers, batch normalization layers, and activation function layers, in the order of Conv1D(4,32,1), BN(32), ReLU, Conv1D(32,64,1), BN(64), ReLU, Conv1D(64,128,1), BN(128), ReLU, Conv1D(128,256,1), BN(256), ReLU, Conv1D(256,256,1).

[0141] According to some embodiments, a second position encoder may be used to encode the position information of the surface element. The position information of the i-th surface element is represented as Among them, p x 、p y are the xy coordinates of the surface element in the UTM coordinate system, n is the unit normal vector of the surface element, λ1≤λ2≤λ3 is the singular value of the surface element covariance matrix, K surfel is the number of surface elements used to locate the vehicle. Based on the xy coordinates of the initial pose in the UTM coordinate system O xy =[x o y o ] T and the xy range of the point cloud R xy =[x r y r ] T , according to the following formula (2) the position information of the surface element Normalize:

[0142]

[0143] In formula (2), is the normalized position information.

[0144] The second position encoder is used to normalize the position information Encode and get the position code Wherein, C is the dimension of the position encoding, which is usually equal to the number of channels of the environment feature map, that is, equal to the dimension of the feature vector of each pixel in the environment feature map. Similar to the first position encoder, the second position encoder can also be implemented as a multi-layer perceptron. The second position encoder can, for example, be composed of a set of one-dimensional convolutional layers, batch normalization layers, and activation function layers in the order of Conv1D(7,32,1), BN(32), ReLU, Conv1D(32,64,1), BN(64), ReLU, Conv1D(64,128,1), BN(128), ReLU, Conv1D(128,256,1), BN(256), ReLU, Conv1D(256,256,1).

[0145] The position codes of all map elements have the same dimension C. C can be set to 256, for example. The position codes of map elements are uniformly expressed as Where K is the number of map elements used to locate the vehicle. It can be understood that K = K surfel +K hd .

[0146] According to some embodiments, in step S2312, the semantic coding of the map element may be determined based on the correspondence between the multiple types of category information and the multiple semantic codings, wherein the multiple semantic codings are parameters of the positioning model and are obtained by training the positioning model.

[0147] According to the above embodiment, the semantic coding is trainable, thereby improving the ability of the semantic coding to express map element category information, thereby improving positioning accuracy. The training method of the semantic coding will be described in detail in the positioning model training method 600 of the following embodiment.

[0148] Semantic encoding of the j-th category information It can be determined according to the following formula (3):

[0149]

[0150] Among them, f() represents the mapping relationship between category information and semantic coding, j is the number of category information, N e is the number of category information. C is the dimension of semantic encoding (the same as the dimension of position encoding). According to some embodiments, as described above, map elements include lane lines, curbs, crosswalks, stop lines, traffic signs, poles, and surface elements, a total of seven types. Accordingly, N e=7, the category information numbers 1 to 7 correspond to the above seven map elements respectively.

[0151] The map element set is denoted as {M i |i=1,2,…,K}, where K is the number of map elements. The category information of each map element is recorded as s i According to the above formula (3), the semantic encoding of each map element can be obtained as

[0152] After the position code and semantic code of the map element are obtained through steps S2311 and S2312, in step S2313, the initial code vector of the map element can be obtained by fusing the position code and the semantic code.

[0153] According to some embodiments, the sum of the position code and the semantic code may be used as an initial code vector of the map element.

[0154] According to some other embodiments, the weighted sum of the position code and the semantic code may also be used as the initial code vector of the map element.

[0155] After the initial coding vector of the map element is obtained in step S231, in step S232, the initial coding vector is updated based on the environmental feature to obtain the target coding vector of the map element. The set of target coding vectors of each map element is the map feature.

[0156] According to some embodiments, when the environmental feature includes multiple environmental feature maps of different sizes in the target three-dimensional space, in step S232, the initial encoding vector may be updated based only on the environmental feature map with the smallest size among the multiple environmental feature maps, thereby improving computational efficiency.

[0157] For example, in the example described above for step S2233, the environment features include a first environment feature map with a size of 160*160*256 and two second environment feature maps with sizes of 320*320*128 and 640*640*64. The initial encoding vector of the map element is updated only based on the environment feature map with the smallest size, that is, the first environment feature map.

[0158] According to some embodiments, in step S232, the initial encoding vector of the map element may be updated at least once using environmental features based on an attention mechanism to obtain a target encoding vector.

[0159] The environmental features are located in the target three-dimensional space (BEV space). According to the above embodiment, the initial coding vector of the map element is updated at least once using the environmental features, and the coding vector of the map element can be converted to the target three-dimensional space to obtain the target coding vector in the target three-dimensional space. At the same time, the attention mechanism can capture the correlation between the features. According to the above embodiment, the coding vector of the map element is updated using the attention mechanism, which can improve the accuracy of the target coding vector.

[0160] According to some embodiments, in each update of the at least one update mentioned above, the following steps S2321 and S2322 are performed.

[0161] In step S2321, based on the self-attention mechanism, the current encoding vector is updated to obtain an updated encoding vector.

[0162] In step S2322, based on the cross-attention mechanism, the updated coding vector is fused with the environmental features to obtain a fused coding vector.

[0163] It should be noted that the current encoding vector in the first update is the initial encoding vector obtained in step S231. That is, in the first update, the current encoding vector Q of the i-th map element can be i Initialized to:

[0164]

[0165] The current encoding vector in the second and subsequent updates is the encoding vector after the previous fusion. For example, the current encoding vector in step S2321 in the second update is the encoding vector after the fusion obtained in step S2322 in the first update.

[0166] The last fused coding vector is used as the target coding vector of the map element in the target three-dimensional space.

[0167] Map features are the set of target encoding vectors of each map element, that is, map features in, is the target encoding vector of the ith map element, C is the dimension of the target encoding vector, and K is the number of map elements.

[0168] According to some embodiments, for step S2321, the current encoding vector of each map element may be used as a query vector (Query), and based on the self-attention mechanism, the relevance of the map element to other map elements (i.e., attention weight) may be obtained. Then, based on the relevance of the map element to other map elements, the current encoding vector of the map element is fused with the current encoding vectors of other map elements to obtain an updated encoding vector of the map element.

[0169] According to some embodiments, the self-attention mechanism in step S2321 may be a multi-head attention mechanism, which is used to collect information between query vectors of map elements. According to some embodiments, the update of the current encoding vector of the map element may be implemented according to the following formula (5):

[0170]

[0171] Among them, SA(Q i ) represents the updated encoding vector based on the self-attention mechanism (SA), M represents the number of attention heads, and W m and W′ m represents the learnable projection matrix (which is a trainable parameter of the positioning model), A m (Q i ,Q j ) represents the encoding vector Q i and the encoding vector Q j The attention weights between

[0172] According to some embodiments, step S2322 may adopt a deformable attention mechanism, and use the smallest environmental feature map to achieve the fusion of the map encoding vector and the environmental features according to the following formula (6):

[0173]

[0174] in, Indicates that the encoding vector Q is converted based on the cross attention mechanism (CA) i and the 0th layer environmental feature map (i.e. the smallest environmental feature map) in the target three-dimensional space (BEV space) The encoding vector obtained after fusion, DA represents the deformable attention mechanism, Indicates the position of the reference point. The initialization value of the reference point is the position coordinate of the map element projected into the target three-dimensional space. Represents the position encoding of the environment feature map at layer 0.

[0175] According to some embodiments, step S232 may be implemented by a trained second transform decoder (TransformerDecoder). Specifically, the initial coding vector of each map element and the environmental features may be input into the trained second transform decoder to obtain the target coding vector of each map element output by the second transform decoder, that is, the map feature.

[0176] According to some embodiments, the second transform decoder comprises at least one transform layer (Transformer Layer), each transform layer being used to update a coding vector of a map element once.

[0177] Furthermore, each transformation layer may include a self-attention module and a cross-attention module. The self-attention module is used to update the current coding vector of the map element to obtain an updated coding vector, that is, to implement the above step S2321. The cross-attention module is used to fuse the updated coding vector with the environmental features to obtain a fused coding vector, that is, to implement the above step S2322.

[0178] After obtaining the environmental features and the map features through steps S220 and S230 respectively, in step S240, a posture offset for correcting the initial posture is determined based on the environmental features and the map features.

[0179] According to some embodiments, the target pose offset may be determined by matching environmental features with map features.

[0180] According to some embodiments, the environmental feature includes at least one environmental feature map in the target three-dimensional space, and the sizes of the at least one feature map are different. Accordingly, step S240 may include steps S241-S243.

[0181] In step S241, at least one environmental feature map is arranged in order of size from small to large. That is, at least one environmental feature map is arranged in order of layer number from small to large. The arrangement result may be, for example, the 0th layer environmental feature map, the 1st layer environmental feature map, the 2nd layer environmental feature map...

[0182] For any one of the at least one environmental feature graph, the following steps S242 and S243 are performed.

[0183] In step S242, the first pose offset is determined by matching the environment feature map with the map features.

[0184] In step S243, the current pose offset is superimposed with the first pose offset to obtain an updated pose offset.

[0185] Among them, the current pose offset corresponding to the first environment feature map is an all-zero vector, the current pose offset corresponding to the second and each subsequent environment feature map is the updated pose offset corresponding to the previous environment feature map, and the target pose offset is the updated pose offset corresponding to the last environment feature map.

[0186] According to the above embodiment, by solving the pose offset for each environmental feature map in order of size from small to large, the precision and accuracy of pose offset estimation can be gradually improved, thereby improving the accuracy of the target pose offset.

[0187] According to some embodiments, step S242 further includes steps S2421 - S2423 .

[0188] In step S2421, sampling is performed within a preset offset sampling range to obtain a plurality of candidate pose offsets.

[0189] In step S2422, for any candidate pose offset among the multiple candidate pose offsets, the matching degree between the environment feature map and the map feature under the candidate pose offset is determined.

[0190] In step S2423, based on the matching degrees corresponding to the multiple candidate pose offsets, the multiple candidate pose offsets are fused to obtain the first pose offset.

[0191] According to some embodiments, for step S2421, uniform sampling may be performed within the offset sampling range at a certain sampling interval to obtain a plurality of candidate posture offsets.

[0192] According to some embodiments, the size of the offset sampling range is negatively correlated with the size of the environmental feature map. In addition, for environmental feature maps of different sizes, the same number of candidate pose offsets are sampled. According to this embodiment, the larger the size and resolution of the environmental feature map, the smaller the offset sampling range and sampling interval, and the higher the sampling precision. This can improve the sampling precision of the candidate pose offsets, thereby improving the estimation accuracy of the pose offsets.

[0193] Taking the environment feature including l (l∈{0,1,2}) layers of environment feature maps as an example, for the lth layer of environment feature maps, the candidate pose offsets of the three degrees of freedom obtained by equally spaced sampling in the x, y, and yaw directions are for:

[0194]

[0195] Among them, r x Indicates the offset sampling range in the x direction, r y Indicates the offset sampling range in the y direction, ryaw Indicates the offset sampling range of the yaw direction (yaw angle), N s The maximum number of samples in each direction. For example, you can set r x =3m,r y =3m,r yaw =3°, N s =7, accordingly, for each layer of the environment feature map, 343 candidate pose offsets can be sampled.

[0196] According to some embodiments, as described above, the map feature includes target coding vectors of respective map elements. Accordingly, step S2422 may include steps S24221-S24224.

[0197] In step S24221, the current posture is superimposed with the candidate posture offset to obtain the candidate posture.

[0198] For example, the current position of the environment feature map of the lth layer is T est , the candidate pose offset is The candidate pose for in, Represents a generalized addition operation between poses.

[0199] It should be noted that the current posture is the sum of the initial posture and the first posture offset of each environmental feature map before the current environmental feature map.

[0200] For example, the current pose of the 0th layer environment feature map is the initial pose, the current pose of the 1st layer environment feature map is the sum of the initial pose and the first pose offset of the 0th layer environment feature map, and the current pose of the 2nd layer environment feature map is the sum of the initial pose and the first pose offsets of the 0th layer environment feature map and the 1st layer environment feature map respectively.

[0201] For any map element among the multiple map elements, execute steps S24222 and S24223.

[0202] In step S24222, based on the candidate poses, the map elements are projected into the target three-dimensional space (BEV space) to obtain the environment feature vector in the environment feature map corresponding to the map elements.

[0203] According to some embodiments, in order to unify the dimensions of the target coding vector and the environmental feature vector of the map element, a one-dimensional convolution layer and a two-dimensional convolution layer may be used to project the target coding vector and the l-th layer environmental feature map respectively, so as to convert the two into the same dimension. C can be 256, for example. The projected target encoding vector is The environmental feature map after projection is

[0204] According to some embodiments, the candidate poses may be used Project the coordinates of the map elements into the BEV space to obtain the projection coordinates of the map elements in the BEV space Furthermore, the environment feature map is interpolated by an interpolation algorithm (such as a bilinear interpolation algorithm). By interpolation, we can get the environmental feature map in the projection coordinates The characteristic vector at

[0205] In step S24223, the similarity between the target coding vector of the map element and the corresponding environmental feature vector is calculated.

[0206] According to some embodiments, the similarity between the target encoding vector and the environment feature vector can be calculated based on the dot product of the two. For example, the target encoding vector of the i-th map element and the corresponding environmental feature vector Similarity It can be calculated according to the following formula (8):

[0207]

[0208] Where ⊙ represents the dot product, and h() represents a learnable multilayer perceptron (MLP). The multilayer perceptron can be composed of a set of one-dimensional convolutional layers, normalization layers, and activation layers, and the order can be, for example, Conv1D(1,8,1), BN(8), LeakyReLU(0.1), Conv1D(8,8,1), BN(8), LeakyReLU(0.1), Conv1D(8,1,1).

[0209] In step S24224, based on the similarities corresponding to the multiple map elements, the matching degree between the environment feature map and the map feature under the candidate posture offset is determined.

[0210] According to some embodiments, the sum or average of the similarities of the map elements may be determined as the matching degree between the environment feature map and the map feature at the candidate pose offset.

[0211] For example, in the candidate pose offset The matching degree between the lower l-th layer environment feature map and the map feature can be calculated according to the following formula (9).

[0212]

[0213] Where K is the number of map elements.

[0214] According to the above step S2422, the matching degree between the environment feature map and the map feature under each candidate pose offset can be obtained. Subsequently, in step S2423, multiple candidate pose offsets can be fused based on the matching degrees corresponding to each of the multiple candidate pose offsets to obtain the first pose offset.

[0215] According to some embodiments, step S2423 may include step S24231 and step S24232.

[0216] In step S24231, for any candidate pose offset among multiple candidate pose offsets, the probability of the candidate pose offset is determined based on the ratio of the matching degree of the candidate pose offset to the sum of the matching degrees of multiple candidate pose offsets.

[0217] In step S24232, the expected value of multiple candidate pose offsets is determined as the first pose offset.

[0218] According to the above embodiment, the probability (posterior probability) of each candidate pose offset is calculated based on the matching degree, and the candidate pose offsets are fused according to the posterior probability. This has strong interpretability and makes it easy to analyze the reasons for positioning failure and explore directions that may further improve positioning accuracy.

[0219] According to some embodiments, the probability of the candidate pose offset under the current positioning condition X is It can be calculated according to the following formula (10):

[0220]

[0221] Correspondingly, the first pose offset of the l-th layer environment feature map is and covariance∑ l Calculate according to the following formulas (11) and (12):

[0222]

[0223]

[0224] Then we can use the first pose offset Update the current pose T est and the current posture offset ΔT est ,Right now:

[0225]

[0226] The arrow ← indicates that the calculation result on the right side of the arrow Assign to variable T est .

[0227] Figure 3FIG. 3 is a flowchart showing a process 300 for solving a target pose offset according to an embodiment of the present disclosure. Figure 3 In the illustrated embodiment, the environmental features include a three-layer environmental feature map in the BEV space, ie, l=0, 1, 2.

[0228] like Figure 3 As shown, in step S310, the current posture T est Initialized to the initial pose T init , the current posture offset ΔT est Initialize to an all-zero vector and initialize the level number l of the environment feature map to 0.

[0229] In step S320, for the environment feature map of the first layer, the target coding vector of the map element i and the environment feature map are first projected to the same dimension to obtain the projected environment feature map and the projected target encoding vector Map the map element value into the BEV space to obtain the environmental feature vector corresponding to the map element According to the target coding vector and environmental feature vector of each map element, the candidate pose is determined according to the above formula (9). Next (i.e., in the candidate pose offset Bottom) The matching degree between the environment feature map of the first layer and the map feature

[0230] In step S330, the probability of each candidate pose offset is calculated according to the above equations (10)-(12). First pose offset and covariance∑ l .

[0231] In step S340, according to the above formula (13), the current posture T is updated est and the current posture offset ΔT est .

[0232] In step S350, the value of l is increased by one.

[0233] In step S360, it is determined whether l is less than 3. If so, step S320 is executed; if not, step S370 is executed to output the current posture T est , current posture offset ΔT est and the covariance of each layer {∑ l |l∈{0,1,2}}.

[0234] The current posture offset ΔT outputted in step S370 est It is the target pose offset used to correct the initial pose.

[0235] According to some embodiments, step S240 may be implemented by a trained pose solver. Specifically, the environment features, map features, and initial pose may be input into the trained pose solver to obtain the target pose offset output by the pose solver.

[0236] Corresponding to the environmental features including at least one environmental feature map, the posture solver may also include at least one solution layer. At least one solution layer corresponds to at least one environmental feature map. Each solution layer is used to process the corresponding environmental feature map to update the current posture offset. The updated posture offset output by the last solution layer is the target posture offset used to correct the initial posture of the vehicle.

[0237] In step S250, the initial posture and the posture offset are superimposed to obtain a corrected posture of the vehicle.

[0238] The vehicle positioning method 200 of the embodiment of the present disclosure may be implemented using a trained positioning model. Figure 4 A schematic diagram of a vehicle positioning process based on a trained positioning model 400 according to some embodiments of the present disclosure is shown.

[0239] exist Figure 4 In the vehicle positioning process shown, first, the input of the vehicle positioning system is obtained. The system input includes a vector map 441 for positioning the vehicle, an initial position 442 of the vehicle in six degrees of freedom (including three-dimensional coordinates and three attitude angles), images 443 collected by six cameras deployed along the surround direction, and a point cloud 444 collected by a laser radar. The initial position 442 can be the position output by the combined positioning system at the current moment, or it can be the corrected position at the previous moment.

[0240] After obtaining the system input, preprocess the input. Figure 4 As shown, the preprocessing includes steps S451-S453.

[0241] In step S451 , map elements located near the initial position 442 are screened out from the vectorized map 441 , and position information 461 and semantic information (ie, category information) 462 of the map elements are obtained.

[0242] In step S452, the image 443 is preprocessed to obtain a preprocessed image 463. The image preprocessing operation may include de-distortion, scaling to a preset size, standardization, and the like.

[0243] In step S453, the point cloud 444 is preprocessed to obtain a preprocessed point cloud 464. The preprocessing operation of the point cloud may include filtering the point cloud based on the initial posture, and only retaining the point cloud near the initial posture. For example, only the point cloud with the initial posture 442 as the origin, the vehicle's forward direction (positive direction of the x-axis) [-40m, 40m], the vehicle's left direction (positive direction of the y-axis) [-40m, 40m], and the point cloud above the vehicle (positive direction of the z-axis) [-3m, 5m] can be retained. Furthermore, the point cloud can be voxelized, that is, the space can be divided into multiple non-intersecting blocks, and a maximum of 32 points can be retained in each block.

[0244] After preprocessing, the positioning model 400 is used to implement feature extraction and pose solving. Figure 4 As shown, the localization model 400 includes an environment encoder 410 , a map encoder 420 , and a pose solver 430 .

[0245] The environment encoder 410 is used to encode the multimodal sensor data. The environment encoder 410 includes an image encoder 411, a point cloud encoder 412, and a first transform decoder 413. The image encoder 411 is used to encode the preprocessed image 463 to obtain an image feature map 472. The point cloud encoder 412 is used to encode the preprocessed point cloud 464 to obtain a point cloud feature map 473 in the BEV space. The first transform decoder 413 is used to fuse the image feature map 472 and the point cloud feature map 473 in the BEV space to obtain the environment feature 481 in the BEV space.

[0246] The map encoder 420 is used to encode each map element. The map encoder 420 includes a position encoder 421, a semantic encoder 422, and a second transform decoder 423. The position encoder 421 is used to encode the position information 461 of the map element to obtain a position code. The semantic encoder 422 is used to encode the semantic information 462 of the map element to obtain a semantic code. The position code and the semantic code are added to obtain an initial coding vector 471 of the map element. The second transform decoder 423 updates the initial coding vector 471 of each map element based on the environmental feature 481 to map the initial coding vector 471 to the BEV space to obtain a target coding vector 482 of each map element in the BEV space, that is, a map feature.

[0247] The pose solver 430 takes environmental features 481, map features 482 and initial pose 442 as input, and after a series of processing (the processing of step S240 above), outputs a target pose offset 491, a current pose 492 (i.e., the corrected pose obtained by correcting the initial pose 442 using the target pose offset 491) and a pose covariance 493.

[0248] According to an embodiment of the present disclosure, a vectorized map construction method is also provided. The vectorized map constructed according to the method can be applied to the above-mentioned vehicle positioning method 200.

[0249] Figure 5 FIG. 5 is a flowchart of a vectorized map construction method 500 according to an embodiment of the present disclosure. The execution subject of the method 500 is usually a server (eg Figure 1 In some cases, the execution subject of method 500 may also be an autonomous driving vehicle (e.g. Figure 1 110 shown in FIG. Figure 5 As shown, method 500 includes steps S510-S540.

[0250] In step S510, a point cloud in a point cloud map is obtained.

[0251] In step S520, the projection plane of the point cloud map is divided into a plurality of two-dimensional grids of a first unit size.

[0252] For any two-dimensional grid among the multiple two-dimensional grids, steps S530 and S540 are performed.

[0253] In step S530, a plane in the two-dimensional grid is extracted based on a point cloud in the three-dimensional space corresponding to the two-dimensional grid.

[0254] In step S540, the plane is stored as a surface element in the vectorized map.

[0255] According to an embodiment of the present disclosure, a plane is extracted from a point cloud map, and the extracted plane is stored as a surface element in a vectorized map, thereby improving the richness and density of the geographic elements in the vectorized map, thereby improving the positioning accuracy of the vehicle.

[0256] Compared with point cloud maps, vectorized maps are much smaller and easier to update. Storing vectorized maps (rather than point cloud maps) on the vehicle side can greatly reduce the storage cost of the vehicle side, improve the availability of vehicle positioning methods, and meet mass production needs. According to experimental verification, the volume of vectorized maps is about 0.35M / km. Compared with point cloud maps, the volume is reduced by 97.5%.

[0257] The various steps of method 500 are described in detail below.

[0258] In step S510, a point cloud in a point cloud map is obtained.

[0259] Point cloud maps use dense point clouds to represent geographic elements, while vector maps use information such as identification, name, location, attributes, and topological relationships between them to represent geographic elements.

[0260] In step S520, the projection plane of the point cloud map is divided into a plurality of two-dimensional grids of a first unit size.

[0261] The projection plane of the point cloud map refers to the xy plane. The first unit size can be set as needed. For example, the first unit size can be set to 1m*1m, 2m*2m, etc.

[0262] In step S530, a plane in the two-dimensional grid is extracted based on the point cloud in the three-dimensional space corresponding to the two-dimensional grid. The three-dimensional space corresponding to the two-dimensional grid refers to a cylindrical space with the two-dimensional grid as a cross section.

[0263] According to some embodiments, step S530 may include steps S531 - S534 .

[0264] In step S531, the three-dimensional space is divided into a plurality of three-dimensional grids of a second unit size along the height direction. The second unit size can be set as needed. For example, the second unit size can be set to 1m*1m*1m, 2m*2m*2m, etc.

[0265] For any three-dimensional grid among the multiple three-dimensional grids, steps S532 and S533 are performed.

[0266] In step S532, based on the point cloud in the three-dimensional grid, the confidence level of the plane included in the three-dimensional grid is calculated.

[0267] In step S533, in response to the confidence being greater than the threshold, the plane in the three-dimensional grid is extracted. The threshold can be set as needed. For example, the threshold can be set to 10, 15, etc.

[0268] In step S534, the plane with the highest confidence in the multiple three-dimensional grids is determined as the plane corresponding to the two-dimensional grid.

[0269] According to some embodiments, for step S532, the confidence that the plane is included in the three-dimensional grid can be calculated according to the following steps: singular value decomposition is performed on the covariance matrix of the point cloud in the three-dimensional grid to obtain a first singular value λ1, a second singular value λ2, and a third singular value λ3, wherein the first singular value is less than or equal to the second singular value, and the second singular value is less than or equal to the third singular value, that is, λ1≤λ2≤λ3. The ratio λ2 / λ1 of the second singular value to the first singular value is determined as the confidence s, that is, s=λ2 / λ1.

[0270] According to the above embodiment, if λ2 / λ1 is large, it can be considered that the change (variance) of the point cloud data along the eigenvector direction corresponding to λ1 is very small relative to other directions and can be ignored, so the point cloud can be approximated as a plane. λ2 / λ1 can indicate the possibility of including a plane in the three-dimensional grid, so it can be used as the confidence that the plane is included in the three-dimensional grid.

[0271] In step S540, the plane is stored as a surface element in the vectorized map. According to some embodiments, the identifier of the surface element corresponding to the plane may be determined, and the coordinates of a point on the plane and the unit normal vector of the plane may be associated with the identifier and stored.

[0272] According to some embodiments, the identifier of the surface element may be generated according to a preset rule. It is understandable that the identifiers of the surface elements in the vectorized map are different.

[0273] According to some embodiments, the centroid of the point cloud in the three-dimensional grid to which the plane belongs may be taken as a point on the plane, and its coordinates may be stored. The unit normal vector of the plane is obtained by normalizing the eigenvector corresponding to the first singular value λ1.

[0274] In a vector map, a surface element P can be represented as

[0275] in, represents the field of real numbers, Represents the xy coordinates of a point on the surface element P in the UTM coordinate system. represents the unit normal vector of the surface element, λ1≤λ2≤λ3 are the singular values ​​of the surface element covariance matrix.

[0276] According to some embodiments, in addition to the surface elements, other geographic elements are stored in the vectorized map in the form of vectors, including road elements, such as lane lines, curbs, crosswalks, stop lines, traffic signs and poles.

[0277] In a vectorized map, lanes, curbs, and stop signs are represented as line segments, with endpoints in the 2D xy coordinates of the UTM coordinate system. A crosswalk is represented as a polygon, with the vertices in the 2D xy coordinates of the UTM coordinate system. Traffic signs are represented as rectangles perpendicular to the xy plane, with vertices in the 3D UTM coordinates, where the z coordinate is expressed as height relative to the ground. Poles are represented by the 2D xy coordinates in the UTM coordinate system and the height of the pole.

[0278] According to an embodiment of the present disclosure, a method for training a positioning model is also provided. The positioning model trained according to the method can be applied to the above-mentioned vehicle positioning method 200.

[0279] Figure 6 FIG. 6 is a flowchart of a method 600 for training a positioning model according to an embodiment of the present disclosure. The execution subject of the method 600 is usually a server (eg, Figure 1 In some cases, the execution subject of method 600 may also be an autonomous driving vehicle (e.g. Figure 1 The positioning model of the embodiment of the present disclosure includes an environment encoder, a map encoder and a pose solver. An exemplary structure of the positioning model can be found in Figure 4 .

[0280] like Figure 6 As shown, method 600 includes steps S610-S680.

[0281] In step S610, an initial posture of the sample vehicle, a true posture value corresponding to the initial posture, multimodal sensor data of the sample vehicle, and a plurality of map elements for locating the sample vehicle are obtained.

[0282] In step S620, the multimodal sensor data is input into an environment encoder to obtain environmental features.

[0283] In step S630, element information of a plurality of map elements is input into a map encoder to obtain map features.

[0284] In step S640, the environment features, the map features and the initial pose are input into a pose solver, so that the pose solver: performs sampling within a first offset sampling range to obtain a plurality of first candidate pose offsets; for any first candidate pose offset among the plurality of first candidate pose offsets, determines a first matching degree between the environment features and the map features under the first candidate pose offset; and determines and outputs a predicted pose offset based on the first matching degrees corresponding to each of the plurality of first candidate pose offsets;

[0285] In step S650, a first loss is determined based on the predicted pose offset and the true value of the pose offset, wherein the true value of the pose offset is the difference between the true value of the pose and the initial pose.

[0286] In step S660, a second loss is determined based on the first matching degrees corresponding to each of the plurality of first candidate pose offsets, wherein the second loss indicates the difference between the predicted probability distribution of the pose true value and the true probability distribution of the pose true value.

[0287] In step S670, an overall loss of the positioning model is determined based on at least the first loss and the second loss.

[0288] In step S680 , the parameters of the positioning model are adjusted based on the overall loss.

[0289] According to the embodiments of the present disclosure, the first loss can guide the positioning model to output a more accurate predicted pose offset. The second loss can guide the predicted probability distribution of the true pose value to be close to its true probability distribution, thereby avoiding multi-peak distribution. Determining the overall loss of the positioning model based on the first loss and the second loss and adjusting the parameters of the positioning model accordingly can improve the positioning accuracy of the positioning model.

[0290] According to some embodiments, the initial position and posture may be the position and posture output by the combined positioning system of the sample vehicle at the current moment, or may be the corrected position and posture at the previous moment.

[0291] According to some embodiments, the multimodal sensor data includes images and point clouds. The multiple map elements used to locate the sample vehicle can be geographic elements located near the initial pose selected from the vectorized map. The multiple map elements include, for example, road elements (lane lines, curbs, crosswalks, stop lines, traffic signs, poles, etc.) and surface elements.

[0292] Steps S620 and S630 correspond to steps S220 and S230 described above. The environment encoder and map encoder in steps S620 and S630 are used to execute steps S220 and S230 respectively. The internal processing logic of the environment encoder and the map encoder refers to the relevant description of steps S220 and S230 above, which will not be repeated here.

[0293] The pose solver in step S640 is used to execute step S240 described above. The internal processing logic of the pose solver is described in the above description of step S240, which will not be repeated here.

[0294] The first loss is the pose mean square error loss. According to some embodiments, the first loss L rmse It can be calculated according to the following formula:

[0295]

[0296] Among them, l is the level number of the environment feature map (that is, the number of the solution layer of the pose solver), and the matrix U l The covariance can be decomposed by SVD get, Is a diagonal matrix, the diagonal element value of the matrix is ​​the diagonal matrix S -1 The diagonal elements of are normalized values, is the predicted pose offset output by the lth solution layer (i.e., the first pose offset described in method 200), is the true value of the pose offset of the lth solution layer, that is, the difference between the true value of the pose and the initial pose. It can be understood that the true value of the pose offset of each solution layer is the same.

[0297] It should be noted that if you directly and As the first loss, the influence of each direction on positioning is the same. However, in actual situations, different directions have different effects on positioning. For example, in a lateral degradation scenario (such as a tunnel, lack of lateral constraints on the x-axis), the lateral positioning error will be relatively large, and it is difficult to improve its accuracy through optimization. Therefore, in this case, it is hoped to reduce the lateral weight, thereby reducing the impact of lateral uncertainty on positioning accuracy. The weight of the direction is determined by the covariance. According to formula (14), the larger the covariance of a direction, the greater the uncertainty, and the weight of the direction. The smaller the setting, the smaller the impact on the first loss.

[0298] The second loss is the pose distribution KL divergence loss. According to some embodiments, the second loss It can be calculated according to the following formula:

[0299]

[0300]

[0301]

[0302] in, It represents the true value of the pose of the lth solution layer. It can be understood that the true value of the pose of each solution layer is the same. It represents the matching degree between the environment feature map of the lth layer and the map feature under the true value of the pose, which can be calculated by referring to the above formula (9). Indicates that in the candidate pose Next (i.e., in the first candidate pose offset The first matching degree between the environment feature map of the first layer and the map feature can be calculated according to the above formula (9).

[0303] Formulas (15)-(17) are derived from the KL divergence formula and can indicate the difference between the predicted probability distribution of the true pose value and the true probability distribution of the true pose value. The predicted probability distribution of the true pose value is the probability distribution of multiple first candidate pose offsets, that is, the probability distribution calculated by formula (10). The true probability distribution of the true pose value is a Dirac distribution (peaked distribution) with a probability of 1 at the true pose value.

[0304] According to some embodiments, the overall loss of the positioning model may be a first loss L rmse With the second loss The weighted sum of .

[0305] According to some embodiments, the pose solver is further configured to: perform sampling within a second offset sampling range to obtain a plurality of second candidate pose offsets; and for any second candidate pose offset among the plurality of second candidate pose offsets, determine a second degree of matching between the environmental feature and the map feature at the second candidate pose offset.

[0306] Accordingly, method 600 also includes: determining a third loss based on a second matching degree corresponding to each of the multiple second candidate pose offsets, wherein the third loss indicates the difference between the predicted probability distribution of the multiple candidate poses and the true probability distribution of the multiple candidate poses, and the multiple candidate poses are obtained by superimposing the multiple second candidate pose offsets with the current pose respectively.

[0307] It should be noted that the second offset sampling range is usually larger than the first offset sampling range. The first offset sampling range is determined by step S2421 described above.

[0308] The second matching degree can be calculated by referring to the above formula (9).

[0309] The current pose is the sum of the initial pose and the predicted pose offsets of each solution layer before the current solution layer.

[0310] The third loss is the KL divergence loss of the sampled pose distribution. According to some embodiments, the third loss It can be calculated according to the following formula:

[0311]

[0312]

[0313]

[0314] in, represents the true value of the pose of the lth solution layer. It can be understood that the true value of the pose of each solution layer is the same. q(·) represents the probability density function of the sampling proposal distribution of the pose, where the sampling distribution of xy is a multivariate t distribution, and the sampling distribution of the yaw direction is a mixed distribution of the von Mises distribution and the uniform distribution. is the candidate pose sampled, N r is the number of candidate poses sampled. It represents the matching degree between the environment feature map of the lth layer and the map feature under the true value of the pose, which can be calculated by referring to the above formula (9). Indicates that in the candidate pose Next (i.e., in the second candidate pose offset The second matching degree between the environment feature map of the first layer and the map feature can be calculated by referring to the above formula (9).

[0315] Equations (18)-(20) are derived from the KL divergence formula and can indicate the difference between the predicted probability distribution of multiple candidate poses and their true probability distribution.

[0316] The third loss It can make feature learning more complete and serve as a supervisory signal to improve the feature learning effect.

[0317] According to some embodiments, the overall loss of the positioning model may be a first loss L rmse Second loss The third loss The weighted sum of .

[0318] According to some embodiments, the environmental features include an environmental feature map in a target space (e.g., a BEV space). The element information of the map element includes category information (i.e., semantic information). The map encoder is further configured to determine the semantic encoding corresponding to the category information of the map element based on the correspondence between the multiple category information and the multiple semantic encodings. The multiple semantic encodings are trainable parameters of the positioning model.

[0319] Accordingly, method 600 further includes: projecting a target map element belonging to a target category among a plurality of map elements into a target three-dimensional space to obtain a truth map of semantic segmentation in the target three-dimensional space. The value of a first pixel in the truth map indicates whether the first pixel is occupied by the target map element. Based on the environmental feature map, a prediction map of semantic segmentation is determined, wherein the value of a second pixel in the prediction map indicates the similarity between a corresponding environmental feature vector and a semantic encoding of the target category, and the corresponding environmental feature vector is a feature vector of a pixel in the environmental feature map corresponding to the position of the second pixel. Based on the truth map and the prediction map, a fourth loss can be determined.

[0320] For example, for target category j, the target map elements belonging to category j are projected into the BEV space to obtain the true value map of the semantic segmentation of category j in the environment feature map of layer l: in, Indicates that the first pixel (h, w) in the truth map is occupied by a target map element belonging to category j, Indicates that the first pixel (h, w) in the truth map is not occupied by a target map element belonging to category j.

[0321] The training goal of semantic coding is to make the semantic coding of the jth category and the truth map of BEV semantic segmentation BEV environment feature vector at As close as possible. The predicted map of semantic segmentation of category j in the environment feature map of layer l Constructed according to the following formula:

[0322]

[0323] in, Represents the prediction graph for category j The value of the second pixel whose coordinates are (h,w) in is the environmental feature map of the first layer The environmental feature vector corresponding to the pixel with coordinates (h, w) in l are learnable model parameters, is the semantic encoding of category j, and ⊙ represents the dot product.

[0324] The fourth loss is the semantic segmentation loss. According to some embodiments, the fourth loss L ss It can be calculated according to the following formula:

[0325]

[0326]

[0327]

[0328]

[0329] α=0.8,γ=2.0 (26)

[0330] Among them, N e is the number of category information.

[0331] According to the fourth loss L ss , semantic coding is trainable, which can improve the ability of semantic coding to express map element category information, thereby improving positioning accuracy.

[0332] According to some embodiments, the overall loss of the positioning model may be a first loss L rmse Second loss Fourth loss L ss The weighted sum of .

[0333] According to some embodiments, the overall loss L of the localization model sum It can be the first loss L rmse Second loss The third loss and the fourth loss L ss The weighted sum of is:

[0334]

[0335] Among them, α1 to α4 are the weights of the first loss to the fourth loss respectively.

[0336] After determining the overall loss of the localization model, the error back propagation method is used to adjust the parameters of the localization model according to the overall loss. The parameters of the localization model include semantic encoding, weights in the multi-layer perceptron, weights in the convolution kernel, and projection matrix in the attention module.

[0337] It is understandable that steps S610-S680 may be executed repeatedly for multiple times until a preset termination condition is reached, such as when the overall loss is less than a loss threshold, when the number of cycles reaches a number threshold, or when the overall loss converges.

[0338] According to some embodiments, when training the positioning model, data enhancement processing can be performed on the training data to improve the generalization and robustness of the positioning model. Data enhancement processing includes, for example, enhancing the color, contrast, and brightness of the image, randomly removing part of the image area, randomly removing certain map elements (such as pole elements) of a certain frame according to a certain probability, rotating the coordinates of the map elements and the global coordinate system, rotating the external parameters of the camera and the lidar, etc.

[0339] According to an embodiment of the present disclosure, a vehicle positioning device is also provided. Figure 7 FIG. 8 shows a structural block diagram of a vehicle positioning device 700 according to an embodiment of the present disclosure. Figure 7 As shown, the device 700 includes an acquisition module 710 , an environment encoding module 720 , a map encoding module 730 , a determination module 740 and an overlay module 750 .

[0340] The acquisition module 710 is configured to acquire an initial position of the vehicle, multimodal sensor data of the vehicle, and a plurality of map elements for locating the vehicle.

[0341] The environment encoding module 720 is configured to encode the multimodal sensor data to obtain environment features.

[0342] The map encoding module 730 is configured to encode the plurality of map elements to obtain map features.

[0343] The determination module 740 is configured to determine a target posture offset for correcting the initial posture based on the environmental features and the map features.

[0344] The superposition module 750 superimposes the initial posture and the target posture offset to obtain a corrected posture of the vehicle.

[0345] According to the embodiments of the present disclosure, encoding multimodal sensor data can make full use of the data of each sensor, reduce information loss, and enable environmental features to fully and accurately express the surrounding environment of the vehicle. Determining the target posture offset based on environmental features and map features, and correcting the initial posture based on the target posture offset can improve the positioning accuracy of the vehicle, and can still achieve accurate positioning of the vehicle even in complex environments.

[0346] According to some embodiments, the initial posture is a posture output by a combined positioning system of the vehicle.

[0347] According to some embodiments, the multimodal sensor data includes a point cloud and an image, and wherein the environment encoding module includes: a point cloud encoding unit configured to encode the point cloud to obtain a point cloud feature map in a target three-dimensional space; an image encoding unit configured to encode the image to obtain an image feature map; and a fusion unit configured to fuse the point cloud feature map and the image feature map to obtain the environment feature.

[0348] According to some embodiments, the target three-dimensional space is a bird's-eye view space of the vehicle.

[0349] According to some embodiments, the fusion unit includes: an initialization subunit, configured to determine an initial environment feature map in the target three-dimensional space based on the point cloud feature map; a first fusion subunit, configured to fuse the initial environment feature map with the image feature map to obtain a first environment feature map in the target three-dimensional space; and a determination subunit, configured to determine the environment features based on the first environment feature map.

[0350] According to some embodiments, the first fusion subunit is further configured to: based on an attention mechanism, fuse the initial environment feature map with the image feature map at least once to obtain the first environment feature map.

[0351] According to some embodiments, the first fusion subunit is further configured to: in each of the at least one fusion: based on the self-attention mechanism, update the current environment feature map to obtain an updated environment feature map; and based on the cross-attention mechanism, fuse the updated environment feature map with the image feature map to obtain a fused environment feature map, wherein the current environment feature map in the first fusion is the initial environment feature map, the current environment feature map in the second and subsequent fusions is the environment feature map after the previous fusion, and the first environment feature map is the environment feature map after the last fusion.

[0352] According to some embodiments, the first fusion subunit is further configured to: input the initial environment feature map and the image feature map into a trained first transform decoder to obtain the first environment feature map output by the first transform decoder.

[0353] According to some embodiments, the determination subunit is further configured to: upsample the first environmental characteristic map at least once to obtain at least one second environmental characteristic map corresponding to the at least one upsampling; and determine the first environmental characteristic map and the at least one second environmental characteristic map as the environmental characteristics.

[0354] According to some embodiments, the plurality of map elements are obtained by screening a plurality of geographic elements in the vectorized map based on the initial pose.

[0355] According to some embodiments, the multiple map elements include at least one road element and at least one geometric element; the at least one road element includes at least one of the following: a lane line, a curb, a crosswalk, a stop line, a traffic sign or a pole; the at least one geometric element includes a surface element.

[0356] According to some embodiments, the surface element is obtained by extracting a plane in a point cloud map.

[0357] According to some embodiments, the map encoding module includes: an initialization unit, configured to encode element information of any one of the multiple map elements to obtain an initial encoding vector of the map element; and an updating unit, configured to update the initial encoding vector based on the environmental feature to obtain a target encoding vector of the map element, wherein the map feature includes the target encoding vectors of each of the multiple map elements.

[0358] According to some embodiments, the element information includes position information and category information, and wherein the initialization unit includes: a first encoding subunit, configured to encode the position information to obtain a position code; a second encoding subunit, configured to encode the category information to obtain a semantic code; and a second fusion subunit, configured to fuse the position code and the semantic code to obtain the initial coding vector.

[0359] According to some embodiments, the second encoding subunit is further configured to: determine the semantic encoding of the map element based on the correspondence between multiple category information and multiple semantic encodings, wherein the multiple semantic encodings are parameters of a positioning model and are obtained by training the positioning model.

[0360] According to some embodiments, the updating unit is further configured to: based on an attention mechanism, utilize the environmental features to update the initial encoding vector at least once to obtain the target encoding vector.

[0361] According to some embodiments, the updating unit is further configured to: in each update of the at least one update: based on the self-attention mechanism, update the current coding vector to obtain an updated coding vector; and based on the cross-attention mechanism, fuse the updated coding vector with the environmental features to obtain a fused coding vector, wherein the current coding vector in the first update is the initial coding vector, the current coding vector in the second and subsequent updates is the last fused coding vector, and the target coding vector is the last fused coding vector.

[0362] According to some embodiments, the environmental features include multiple environmental feature maps in the target three-dimensional space, the sizes of the multiple environmental feature maps are different, and wherein the updating unit is further configured to: update the initial coding vector based on the environmental feature map with the smallest size among the multiple environmental feature maps.

[0363] According to some embodiments, the updating unit is further configured to: input the initial encoding vector and the environmental feature into a trained second transform decoder to obtain the target encoding vector output by the second transform decoder.

[0364] According to some embodiments, the determination module is further configured to determine the target pose offset by matching the environmental feature with the map feature.

[0365] According to some embodiments, the environmental features include at least one environmental feature map in the target three-dimensional space, the sizes of the at least one environmental feature map are different, and wherein the determination module includes: a sorting unit, configured to arrange the at least one environmental feature map in order of size from small to large; and a determination unit, configured to, for any one of the at least one environmental feature map: determine a first pose offset by matching the environmental feature map with the map feature; and superimpose the current pose offset with the first pose offset to obtain an updated pose offset, wherein the current pose offset corresponding to the first environmental feature map is a full zero vector, the current pose offset corresponding to the second and each subsequent environmental feature map is the updated pose offset corresponding to the previous environmental feature map, and the target pose offset is the updated pose offset corresponding to the last environmental feature map.

[0366] According to some embodiments, the determination unit includes: a sampling subunit, configured to perform sampling within a preset offset sampling range to obtain a plurality of candidate pose offsets; a determination subunit, configured to determine, for any one of the plurality of candidate pose offsets, a degree of matching between the environment feature map and the map feature under the candidate pose offset; and a third fusion subunit, configured to fuse the plurality of candidate pose offsets based on the matching degrees corresponding to each of the plurality of candidate pose offsets to obtain the first pose offset.

[0367] According to some embodiments, the size of the offset sampling range is negatively correlated with the size of the environmental feature map.

[0368] According to some embodiments, the map feature includes a target coding vector of each of the multiple map elements, and wherein the determination subunit is further configured to: superimpose the current posture with the candidate posture offset to obtain a candidate posture, wherein the current posture is the sum of the initial posture and the first posture offset of each environmental feature map located before the environmental feature map; for any map element among the multiple map elements: based on the candidate posture, project the map element into the target three-dimensional space to obtain an environmental feature vector in the environmental feature map corresponding to the map element; and calculate the similarity between the target coding vector of the map element and the environmental feature vector; and based on the corresponding similarities of each of the multiple map elements, determine the matching degree between the environmental feature map and the map feature under the candidate posture offset.

[0369] According to some embodiments, the third fusion subunit is further configured to: for any candidate pose offset among the multiple candidate pose offsets, determine the probability of the candidate pose offset based on the ratio of the matching degree of the candidate pose offset to the sum of the matching degrees of the multiple candidate pose offsets; and determine the expectation of the multiple candidate pose offsets as the first pose offset.

[0370] According to some embodiments, the determination module is further configured to: input the environmental features, the map features and the initial pose into a trained pose solver to obtain the target pose offset output by the pose solver.

[0371] It should be understood that Figure 7 The various modules or units of the apparatus 700 shown in FIG. Figure 2 The various steps in the described method 200 correspond to each other. Therefore, the operations, features and advantages described in the above method 200 are also applicable to the device 700 and the various modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0372] According to an embodiment of the present disclosure, a vectorized map construction device is also provided. Figure 8 FIG. 8 is a structural block diagram of a vectorized map construction device 800 according to an embodiment of the present disclosure. Figure 8 As shown, the device 800 includes an acquisition module 810 , a division module 820 , an extraction module 830 and a storage module 840 .

[0373] The acquisition module 810 is configured to acquire a point cloud in a point cloud map.

[0374] The division module 820 is configured to divide the projection plane of the point cloud map into a plurality of two-dimensional grids of a first unit size.

[0375] The extraction module 830 is configured to extract, for any two-dimensional grid among the multiple two-dimensional grids, a plane in the two-dimensional grid based on a point cloud in a three-dimensional space corresponding to the two-dimensional grid.

[0376] The storage module 840 is configured to store the plane as a surface element in the vectorized map.

[0377] According to an embodiment of the present disclosure, a plane is extracted from a point cloud map, and the extracted plane is stored as a surface element in a vectorized map, thereby improving the richness and density of the geographic elements in the vectorized map, thereby improving the positioning accuracy of the vehicle.

[0378] Compared with point cloud maps, vectorized maps are much smaller and easier to update. Storing vectorized maps (rather than point cloud maps) on the vehicle side can greatly reduce the storage cost of the vehicle side, improve the availability of vehicle positioning methods, and meet mass production needs. According to experimental verification, the volume of vectorized maps is about 0.35M / km. Compared with point cloud maps, the volume is reduced by 97.5%.

[0379] According to some embodiments, the extraction module includes: a division unit, configured to divide the three-dimensional space into multiple three-dimensional grids of a second unit size along the height direction; an extraction unit, configured to, for any three-dimensional grid among the multiple three-dimensional grids: based on the point cloud in the three-dimensional grid, calculate the confidence that the plane is included in the three-dimensional grid; and in response to the confidence being greater than a threshold, extract the plane in the three-dimensional grid; and a first determination unit, configured to determine the plane with the highest confidence among the multiple three-dimensional grids as the plane corresponding to the two-dimensional grid.

[0380] According to some embodiments, the extraction unit includes: a decomposition subunit, configured to perform singular value decomposition on the covariance matrix of the point cloud in the three-dimensional grid to obtain a first singular value, a second singular value and a third singular value, wherein the first singular value is less than or equal to the second singular value, and the second singular value is less than or equal to the third singular value; and a determination subunit, configured to determine the ratio of the second singular value to the first singular value as the confidence level.

[0381] According to some embodiments, the storage module includes: a second determination unit configured to determine the identifier of the surface element corresponding to the plane; and a storage unit configured to store the coordinates of a point on the plane and the unit normal vector of the plane in association with the identifier.

[0382] According to some embodiments, the vectorized map further includes a plurality of road elements, any of which is: a lane line, a curb, a crosswalk, a stop line, a traffic sign, or a pole.

[0383] It should be understood that Figure 8 The various modules or units of the apparatus 800 shown in FIG. Figure 5 The various steps in the described method 500 correspond to each other. Therefore, the operations, features and advantages described in the above method 500 are also applicable to the device 800 and the various modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0384] According to an embodiment of the present disclosure, a training device for a positioning model is also provided. Fig. 9 The structural block diagram of the training device 900 of the positioning model according to the embodiment of the present disclosure is shown. The positioning model includes an environment encoder, a map encoder and a pose solver.

[0385] like Fig. 9 As shown, the apparatus 900 includes an acquisition module 910 , a first input module 920 , a second input module 930 , a third input module 940 , a first determination module 950 , a second determination module 960 , a determination module 970 and an adjustment module 980 .

[0386] The acquisition module 910 is configured to acquire an initial posture of a sample vehicle, a true posture value corresponding to the initial posture, multimodal sensor data of the sample vehicle, and a plurality of map elements for locating the sample vehicle.

[0387] The first input module 920 is configured to input the multimodal sensor data into the environment encoder to obtain environmental features.

[0388] The second input module 930 is configured to input element information of the plurality of map elements into the map encoder to obtain map features.

[0389] The third input module 940 is configured to input the environmental features, the map features and the initial pose into the pose solver so that the pose solver: performs sampling within a first offset sampling range to obtain a plurality of first candidate pose offsets; for any first candidate pose offset among the plurality of first candidate pose offsets, determines a first degree of match between the environmental features and the map features at the first candidate pose offset; and determines and outputs a predicted pose offset based on the first degree of match corresponding to each of the plurality of first candidate pose offsets.

[0390] The first determination module 950 is configured to determine a first loss based on the predicted pose offset and a true pose offset value, wherein the true pose offset value is a difference between the true pose value and the initial pose.

[0391] The second determination module 960 is configured to determine a second loss based on the first matching degrees corresponding to each of the multiple first candidate pose offsets, wherein the second loss indicates the difference between the predicted probability distribution of the pose true value and the true probability distribution of the pose true value.

[0392] The determination module 970 is configured to determine an overall loss of the positioning model based at least on the first loss and the second loss.

[0393] The adjustment module 980 is configured to adjust the parameters of the positioning model based on the overall loss.

[0394] According to the embodiments of the present disclosure, the first loss can guide the positioning model to output a more accurate predicted pose offset. The second loss can guide the predicted probability distribution of the true pose value to be close to its true probability distribution, thereby avoiding multi-peak distribution. Determining the overall loss of the positioning model based on the first loss and the second loss and adjusting the parameters of the positioning model accordingly can improve the positioning accuracy of the positioning model.

[0395] According to some embodiments, the pose solver is configured to: sample within a second offset sampling range to obtain a plurality of second candidate pose offsets; and for any second candidate pose offset among the plurality of second candidate pose offsets, determine a second degree of match between the environmental feature and the map feature at the second candidate pose offset.

[0396] The device also includes: a third determination module, configured to determine a third loss based on a second matching degree corresponding to each of the multiple second candidate pose offsets, wherein the third loss indicates the difference between the predicted probability distribution of the multiple candidate poses and the true probability distribution of the multiple candidate poses, and the multiple candidate poses are obtained by superimposing the multiple second candidate pose offsets with the initial pose respectively.

[0397] The determination module is further configured to determine the overall loss based at least on the first loss, the second loss, and the third loss.

[0398] According to some embodiments, the environmental features include an environmental feature map in a target three-dimensional space, the element information includes category information, and the map encoder is configured to: determine the semantic coding corresponding to the category information based on the correspondence between multiple category information and multiple semantic codings, wherein the multiple semantic codings are parameters of the positioning model.

[0399] The device also includes: a projection module, configured to project target map elements belonging to the target category among the multiple map elements into the target three-dimensional space to obtain a truth map of semantic segmentation in the target three-dimensional space, wherein a value of a first pixel in the truth map indicates whether the first pixel is occupied by the target map element; a prediction module, configured to determine a prediction map of semantic segmentation based on the environmental feature map, wherein a value of a second pixel in the prediction map indicates a similarity between a corresponding environmental feature vector and a semantic encoding of the target category, and the corresponding environmental feature vector is a feature vector of a pixel in the environmental feature map corresponding to a position of the second pixel; and a fourth determination module, configured to determine a fourth loss based on the truth map and the prediction map.

[0400] The determination module is further configured to determine the overall loss based at least on the first loss, the second loss, and the fourth loss.

[0401] It should be understood that Fig. 9 The various modules or units of the apparatus 900 shown in FIG. Figure 6 The steps in the described method 600 correspond to each other. Therefore, the operations, features and advantages described in the above method 600 are also applicable to the device 900 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0402] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various modules discussed herein may be separated into multiple modules, and / or at least some functionality of multiple modules may be combined into a single module.

[0403] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 7-Figure 9 The various modules described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of modules 710-980 can be implemented together in a system on chip (System on Chip, SoC). SoC may include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and may optionally execute the received program code and / or include embedded firmware to perform functions.

[0404] According to an embodiment of the present disclosure, there is also provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the vehicle positioning method, vectorized map construction method, and positioning model training method of the embodiment of the present disclosure.

[0405] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable a computer to execute any one of the vehicle positioning method, vectorized map construction method, and positioning model training method according to the embodiment of the present disclosure.

[0406] According to an embodiment of the present disclosure, a computer program product is also provided, comprising computer program instructions, wherein the computer program instructions, when executed by a processor, implement any one of the vehicle positioning method, vectorized map construction method, and positioning model training method according to the embodiment of the present disclosure.

[0407] According to an embodiment of the present disclosure, there is also provided an autonomous driving vehicle, comprising the above-mentioned electronic device.

[0408] refer to Fig.10, a block diagram of an electronic device 1000 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0409] like Fig.10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0410] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, an output unit 1007, a storage unit 1008, and a communication unit 1009. The input unit 1006 can be any type of device that can input information to the electronic device 1000. The input unit 1006 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 1007 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1008 can include but is not limited to a disk and an optical disk. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as a Bluetooth device, an 802.11 device, a Wi-Fi device, a WiMAX device, a cellular communication device and / or the like.

[0411] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as methods 200, 500, and 600. For example, in some embodiments, methods 200, 500, and 600 may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute methods 200 , 500 , and 600 in any other appropriate manner (eg, by means of firmware).

[0412] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0413] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0414] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0415] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0416] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0417] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0418] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0419] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that with the evolution of technology, many of the elements described herein may be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A vehicle positioning method, comprising: Acquiring an initial position of the vehicle, multimodal sensor data of the vehicle, and a plurality of map elements for locating the vehicle; Encoding the multimodal sensor data to obtain environmental features, wherein the environmental features include at least one environmental feature map in the target three-dimensional space, and the at least one environmental feature map has different sizes; Encoding the plurality of map elements to obtain map features; Determining a target posture offset for correcting the initial posture based on the environmental feature and the map feature includes: Arrange the at least one environmental feature map in order of size from small to large; For any one of the at least one environmental feature graph: Sampling is performed within a preset offset sampling range to obtain multiple candidate pose offsets; For any candidate pose offset among the multiple candidate pose offsets, determining a matching degree between the environment feature map and the map feature at the candidate pose offset; Based on the matching degrees corresponding to the multiple candidate pose offsets, the multiple candidate pose offsets are fused to obtain a first pose offset; and Superimposing the current pose offset with the first pose offset to obtain an updated pose offset, wherein the current pose offset corresponding to the first environment feature map is an all-zero vector, and the current pose offset corresponding to the second and subsequent environment feature maps is the updated pose offset corresponding to the previous environment feature map; Determine the updated posture offset corresponding to the last environmental feature map as the target posture offset for correcting the initial posture; as well as The initial posture is superimposed with the target posture offset to obtain a corrected posture of the vehicle.

2. The method according to claim 1, wherein: The initial position and posture is the position and posture output by the combined positioning system of the vehicle.

3. The method according to claim 1 or 2, wherein: The multimodal sensor data includes a point cloud and an image, and wherein encoding the multimodal sensor data to obtain environmental features includes: Encoding the point cloud to obtain a point cloud feature map; Encoding the image to obtain an image feature map; and The point cloud feature map and the image feature map are fused to obtain the environmental features.

4. The method according to claim 3, wherein: The step of fusing the point cloud feature map and the image feature map to obtain the environmental feature includes: Based on the point cloud feature map, determining an initial environment feature map in the target three-dimensional space; fusing the initial environment feature map with the image feature map to obtain a first environment feature map in the target three-dimensional space; and Based on the first environmental characteristic map, the environmental characteristic is determined.

5. The method according to claim 4, wherein: The target three-dimensional space is the bird's-eye view space of the vehicle.

6. The method according to claim 4 or 5, wherein: The fusing the initial environment feature map with the image feature map to obtain a first environment feature map in the target three-dimensional space includes: Based on the attention mechanism, the initial environment feature map is fused with the image feature map at least once to obtain the first environment feature map.

7. The method according to claim 6, wherein: The step of fusing the initial environment feature map with the image feature map at least once based on the attention mechanism to obtain the first environment feature map includes: In each of the at least one fusion: Based on the self-attention mechanism, the current environment feature map is updated to obtain an updated environment feature map; and Based on the cross attention mechanism, the updated environment feature map is fused with the image feature map to obtain a fused environment feature map. Among them, the current environment feature map in the first fusion is the initial environment feature map, the current environment feature map in the second and subsequent fusions is the environment feature map after the last fusion, and the first environment feature map is the environment feature map after the last fusion.

8. The method according to claim 4, wherein: The fusing the initial environment feature map with the image feature map to obtain a first environment feature map in the target three-dimensional space includes: The initial environment feature map and the image feature map are input into a trained first transform decoder to obtain the first environment feature map output by the first transform decoder.

9. The method according to claim 4, wherein: The determining the environmental feature based on the first environmental feature map includes: Upsampling the first environment characteristic map at least once to obtain at least one second environment characteristic map corresponding to the at least one upsampling; and The first environmental characteristic graph and the at least one second environmental characteristic graph are determined as the environmental characteristics.

10. The method according to claim 1, wherein: The multiple map elements are obtained by screening multiple geographic elements in the vectorized map based on the initial pose.

11. The method according to claim 1, wherein: The plurality of map elements include at least one road element and at least one geometric element; The at least one road element comprises at least one of the following: a lane line, a curb, a crosswalk, a stop line, a traffic sign or a pole; The at least one geometric element comprises a surface element.

12. The method according to claim 11, wherein: The surface element is obtained by extracting a plane from a point cloud map.

13. The method according to claim 1, wherein: The encoding of the plurality of map elements to obtain map features comprises: For any map element among the plurality of map elements, encoding element information of the map element to obtain an initial encoding vector of the map element; and Based on the environmental feature, the initial coding vector is updated to obtain a target coding vector of the map element, wherein the map feature includes the target coding vectors of each of the plurality of map elements.

14. The method according to claim 13, wherein: The element information includes location information and category information, and wherein encoding the element information of the map element to obtain an initial encoding vector of the map element includes: Encoding the position information to obtain a position code; Encoding the category information to obtain a semantic code; and The position code and the semantic code are fused to obtain the initial code vector.

15. The method according to claim 14, wherein: The encoding of the category information to obtain a semantic code includes: The semantic coding of the map element is determined based on the correspondence between multiple types of category information and multiple types of semantic coding, wherein the multiple types of semantic coding are parameters of a positioning model and are obtained by training the positioning model.

16. The method according to any one of claims 13 to 15, wherein: The updating of the initial coding vector based on the environmental feature to obtain the target coding vector of the map element includes: Based on the attention mechanism, the initial encoding vector is updated at least once using the environmental features to obtain the target encoding vector.

17. The method according to claim 16, wherein: The updating of the initial encoding vector at least once by using the environmental features based on the attention mechanism to obtain the target encoding vector includes: In each of the at least one update: Based on the self-attention mechanism, the current encoding vector is updated to obtain an updated encoding vector; and Based on the cross attention mechanism, the updated encoding vector is fused with the environmental feature to obtain a fused encoding vector, Among them, the current encoding vector in the first update is the initial encoding vector, the current encoding vector in the second and subsequent updates is the encoding vector after the last fusion, and the target encoding vector is the encoding vector after the last fusion.

18. The method according to claim 13, wherein: The environmental feature includes a plurality of environmental feature maps in a target three-dimensional space, and the sizes of the plurality of environmental feature maps are different. The updating of the initial encoding vector based on the environmental feature includes: The initial encoding vector is updated based on the smallest environmental feature map among the multiple environmental feature maps.

19. The method according to claim 13, wherein: The updating of the initial coding vector based on the environmental feature to obtain the target coding vector of the map element includes: The initial encoding vector and the environmental features are input into a trained second transform decoder to obtain the target encoding vector output by the second transform decoder.

20. The method according to claim 1, wherein: The size of the offset sampling range is negatively correlated with the size of the environmental feature map.

21. The method according to claim 1 or 20, wherein: The map feature includes a target encoding vector for each of the plurality of map elements, and wherein determining a degree of matching between the environment feature map and the map feature at the candidate pose offset includes: Superimposing the current posture with the candidate posture offset to obtain a candidate posture, wherein the current posture is the sum of the initial posture and the first posture offset of each environmental feature map before the environmental feature map; For any map element among the plurality of map elements: Based on the candidate pose, projecting the map element into the target three-dimensional space to obtain an environment feature vector in the environment feature map corresponding to the map element; and Calculating the similarity between the target coding vector of the map element and the environmental feature vector; and Based on the similarities corresponding to the multiple map elements, the matching degree between the environment feature map and the map feature at the candidate posture offset is determined.

22. The method according to claim 1, wherein: The fusing the multiple candidate pose offsets based on the respective corresponding matching degrees of the multiple candidate pose offsets to obtain the first pose offset comprises: For any candidate pose offset among the plurality of candidate pose offsets, determining a probability of the candidate pose offset based on a ratio of a matching degree of the candidate pose offset to a sum of matching degrees of the plurality of candidate pose offsets; and An expectation of the plurality of candidate pose offsets is determined as the first pose offset.

23. The method according to claim 1, wherein: The determining, based on the environmental features and the map features, of a target posture offset for correcting the initial posture comprises: The environmental features, the map features and the initial pose are input into a trained pose solver to obtain the target pose offset output by the pose solver.

24. The method according to claim 10, wherein: The vectorized map includes surface elements, which are obtained by the following operations: Get the point cloud in the point cloud map; Dividing the projection plane of the point cloud map into a plurality of two-dimensional grids of a first unit size; For any two-dimensional grid in the plurality of two-dimensional grids: Extracting a plane in the two-dimensional grid based on a point cloud in a three-dimensional space corresponding to the two-dimensional grid; and The plane is stored as a surface element in the vectorized map.

25. The method according to claim 24, wherein: The extracting the plane in the two-dimensional grid based on the point cloud in the three-dimensional space corresponding to the two-dimensional grid comprises: Dividing the three-dimensional space into a plurality of three-dimensional grids of a second unit size along a height direction; For any three-dimensional grid of the plurality of three-dimensional grids: Calculating the confidence that the three-dimensional grid includes a plane based on the point cloud in the three-dimensional grid; and In response to the confidence being greater than a threshold, extracting a plane in the three-dimensional grid; as well as A plane with the highest confidence in the multiple three-dimensional grids is determined as the plane corresponding to the two-dimensional grid.

26. The method according to claim 25, wherein: The calculating the confidence of the plane included in the three-dimensional grid based on the point cloud in the three-dimensional grid comprises: Performing singular value decomposition on the covariance matrix of the point cloud in the three-dimensional grid to obtain a first singular value, a second singular value, and a third singular value, wherein the first singular value is less than or equal to the second singular value, and the second singular value is less than or equal to the third singular value; and A ratio of the second singular value to the first singular value is determined as the confidence level.

27. The method according to any one of claims 24 to 26, wherein: The storing the plane as a surface element in the vectorized map comprises: Determining the identity of the surface element corresponding to the plane; and The coordinates of a point on the plane and the unit normal vector of the plane are stored in association with the identifier.

28. The method of claim 24, wherein: The vectorized map further includes a plurality of road elements, any of which is: a lane line, a curb, a crosswalk, a stop line, a traffic sign or a pole.

29. A method for training a positioning model, wherein: The positioning model is used to implement the vehicle positioning method according to any one of claims 1 to 28, the positioning model includes an environment encoder, a map encoder and a pose solver, and the training method includes: Acquire an initial posture of a sample vehicle, a true posture value corresponding to the initial posture, multimodal sensor data of the sample vehicle, and a plurality of map elements for locating the sample vehicle; Inputting the multimodal sensor data into the environment encoder to obtain environmental features; Inputting element information of the plurality of map elements into the map encoder to obtain map features; The environment features, the map features and the initial pose are input into the pose solver so that the pose solver: Sampling within a first offset sampling range to obtain a plurality of first candidate pose offsets; For any first candidate pose offset among the plurality of first candidate pose offsets, determining a first degree of matching between the environment feature and the map feature under the first candidate pose offset; and Determine and output a predicted pose offset based on the first matching degrees corresponding to each of the plurality of first candidate pose offsets; Determine a first loss based on the predicted pose offset and a true value of the pose offset, wherein the true value of the pose offset is a difference between the true value of the pose and the initial pose; Determining a second loss based on the first matching degrees corresponding to each of the plurality of first candidate pose offsets, wherein the second loss indicates a difference between a predicted probability distribution of the pose truth value and a true probability distribution of the pose truth value; determining an overall loss of the positioning model based at least on the first loss and the second loss; and Based on the overall loss, parameters of the localization model are adjusted.

30. The method of claim 29, wherein: The pose solver is configured to: Sampling within the second offset sampling range to obtain a plurality of second candidate pose offsets; as well as For any second candidate pose offset among the plurality of second candidate pose offsets, determining a second degree of matching between the environment feature and the map feature under the second candidate pose offset; The method further comprises: Determining a third loss based on the second matching degrees corresponding to the plurality of second candidate pose offsets, wherein the third loss indicates a difference between a predicted probability distribution of the plurality of candidate poses and a true probability distribution of the plurality of candidate poses, the plurality of candidate poses being obtained by superimposing the plurality of second candidate pose offsets with the current pose, respectively; The determining of the overall loss of the positioning model based at least on the first loss and the second loss comprises: The overall loss is determined based on at least the first loss, the second loss, and the third loss.

31. The method according to claim 29 or 30, wherein: The environmental features include an environmental feature map in a target three-dimensional space, the element information includes category information, and the map encoder is configured as follows: Determine the semantic coding corresponding to the category information based on the correspondence between the multiple category information and the multiple semantic codings, wherein the multiple semantic codings are parameters of the positioning model; The method further comprises: Projecting a target map element belonging to a target category among the multiple map elements into the target three-dimensional space to obtain a truth map of semantic segmentation in the target three-dimensional space, wherein a value of a first pixel in the truth map indicates whether the first pixel is occupied by the target map element; Determining a prediction map for semantic segmentation based on the environmental feature map, wherein a value of a second pixel in the prediction map indicates a similarity between a corresponding environmental feature vector and a semantic encoding of the target category, and the corresponding environmental feature vector is a feature vector of a pixel in the environmental feature map corresponding to a position of the second pixel; and Determining a fourth loss based on the true value map and the predicted map; The determining of the overall loss of the positioning model based at least on the first loss and the second loss comprises: The overall loss is determined based on at least the first loss, the second loss, and the fourth loss.

32. A vehicle positioning device, comprising: A first acquisition module is configured to acquire an initial position of the vehicle, multimodal sensor data of the vehicle, and a plurality of map elements for locating the vehicle; An environment encoding module is configured to encode the multimodal sensor data to obtain environment features, wherein the environment features include at least one environment feature map in the target three-dimensional space, and the size of the at least one environment feature map is different; A map encoding module, configured to encode the plurality of map elements to obtain map features; A determination module is configured to determine a target posture offset for correcting the initial posture based on the environmental features and the map features, the determination module comprising: a sorting unit, configured to arrange the at least one environmental feature map in order of size from small to large; and A determining unit is configured to, for any one of the at least one environmental feature graphs: Sampling is performed within a preset offset sampling range to obtain multiple candidate pose offsets; For any candidate pose offset among the multiple candidate pose offsets, determining a matching degree between the environment feature map and the map feature at the candidate pose offset; Based on the matching degrees corresponding to the multiple candidate pose offsets, the multiple candidate pose offsets are fused to obtain a first pose offset; and The current posture offset is superimposed with the first posture offset to obtain an updated posture offset, Among them, the current pose offset corresponding to the first environmental feature map is a full zero vector, the current pose offset corresponding to the second and each subsequent environmental feature map is the updated pose offset corresponding to the previous environmental feature map, and the target pose offset is the updated pose offset corresponding to the last environmental feature map; as well as The superposition module superimposes the initial posture with the target posture offset to obtain a corrected posture of the vehicle.

33. The device according to claim 32, wherein: The initial position and posture is the position and posture output by the combined positioning system of the vehicle.

34. The device according to claim 32 or 33, wherein: The multimodal sensor data includes point clouds and images, and wherein the environment encoding module includes: A point cloud encoding unit, configured to encode the point cloud to obtain a point cloud feature map in a target three-dimensional space; an image encoding unit, configured to encode the image to obtain an image feature map; and The fusion unit is configured to fuse the point cloud feature map and the image feature map to obtain the environmental feature.

35. The device according to claim 34, wherein The target three-dimensional space is the bird's-eye view space of the vehicle.

36. The device according to claim 34, wherein The fusion unit comprises: an initialization subunit, configured to determine an initial environment feature map in the target three-dimensional space based on the point cloud feature map; a first fusion subunit configured to fuse the initial environment feature map with the image feature map to obtain a first environment feature map in the target three-dimensional space; and The first determining subunit is configured to determine the environmental feature based on the first environmental feature map.

37. The device according to claim 36, wherein The first fusion subunit is further configured as: Based on the attention mechanism, the initial environment feature map is fused with the image feature map at least once to obtain the first environment feature map.

38. The device according to claim 37, wherein The first fusion subunit is further configured as: In each of the at least one fusion: Based on the self-attention mechanism, the current environment feature map is updated to obtain an updated environment feature map; as well as Based on the cross attention mechanism, the updated environment feature map is fused with the image feature map to obtain a fused environment feature map. Among them, the current environment feature map in the first fusion is the initial environment feature map, the current environment feature map in the second and subsequent fusions is the environment feature map after the last fusion, and the first environment feature map is the environment feature map after the last fusion.

39. The device according to any one of claims 36 to 38, wherein: The first fusion subunit is further configured as: The initial environment feature map and the image feature map are input into a trained first transform decoder to obtain the first environment feature map output by the first transform decoder.

40. The apparatus of claim 36, wherein: The first determining subunit is further configured to: Upsampling the first environment characteristic map at least once to obtain at least one second environment characteristic map corresponding to the at least one upsampling; and The first environmental characteristic graph and the at least one second environmental characteristic graph are determined as the environmental characteristics.

41. The apparatus of claim 32, wherein: The multiple map elements are obtained by screening multiple geographic elements in the vectorized map based on the initial pose.

42. The apparatus of claim 32, wherein: The plurality of map elements include at least one road element and at least one geometric element; The at least one road element comprises at least one of the following: a lane line, a curb, a crosswalk, a stop line, a traffic sign or a pole; The at least one geometric element comprises a surface element.

43. The device according to claim 42, wherein: The surface element is obtained by extracting a plane from a point cloud map.

44. The apparatus of claim 32, wherein: The map encoding module includes: an initialization unit, configured to encode, for any map element among the plurality of map elements, element information of the map element to obtain an initial encoding vector of the map element; and The updating unit is configured to update the initial coding vector based on the environmental feature to obtain a target coding vector of the map element, wherein the map feature includes the target coding vectors of each of the plurality of map elements.

45. The device according to claim 44, wherein The element information includes location information and category information, and wherein the initialization unit includes: A first encoding subunit is configured to encode the position information to obtain a position code; a second encoding subunit, configured to encode the category information to obtain a semantic code; and The second fusion subunit is configured to fuse the position code and the semantic code to obtain the initial code vector.

46. ​​The device according to claim 45, wherein The second encoding subunit is further configured as follows: The semantic coding of the map element is determined based on the correspondence between multiple types of category information and multiple types of semantic coding, wherein the multiple types of semantic coding are parameters of a positioning model and are obtained by training the positioning model.

47. The device according to any one of claims 44 to 46, wherein: The updating unit is further configured to: Based on the attention mechanism, the initial encoding vector is updated at least once using the environmental features to obtain the target encoding vector.

48. The apparatus of claim 47, wherein: The updating unit is further configured to: In each of the at least one update: Based on the self-attention mechanism, the current encoding vector is updated to obtain an updated encoding vector; and Based on the cross attention mechanism, the updated encoding vector is fused with the environmental feature to obtain a fused encoding vector, Among them, the current encoding vector in the first update is the initial encoding vector, the current encoding vector in the second and subsequent updates is the encoding vector after the last fusion, and the target encoding vector is the encoding vector after the last fusion.

49. The apparatus of claim 44, wherein: The environmental feature comprises a plurality of environmental feature graphs in the target three-dimensional space, the sizes of the plurality of environmental feature graphs are different, and wherein the updating unit is further configured to: The initial encoding vector is updated based on the smallest environmental feature map among the multiple environmental feature maps.

50. The apparatus of claim 44, wherein: The updating unit is further configured to: The initial encoding vector and the environmental features are input into a trained second transform decoder to obtain the target encoding vector output by the second transform decoder.

51. The apparatus of claim 32, wherein: The size of the offset sampling range is negatively correlated with the size of the environmental feature map.

52. The apparatus of claim 32 or 51, wherein: The map feature comprises a target coding vector for each of the plurality of map elements, and wherein the determining unit is further configured to: Superimposing the current posture with the candidate posture offset to obtain a candidate posture, wherein the current posture is the sum of the initial posture and the first posture offset of each environmental feature map before the environmental feature map; For any map element among the plurality of map elements: Based on the candidate pose, projecting the map element into the target three-dimensional space to obtain an environment feature vector in the environment feature map corresponding to the map element; and Calculating the similarity between the target coding vector of the map element and the environmental feature vector; and Based on the similarities corresponding to the multiple map elements, the matching degree between the environment feature map and the map feature at the candidate posture offset is determined.

53. The apparatus of claim 32, wherein: The determining unit is further configured to: For any candidate pose offset among the multiple candidate pose offsets, determining a probability of the candidate pose offset based on a ratio of a matching degree of the candidate pose offset to a sum of matching degrees of the multiple candidate pose offsets; as well as An expectation of the plurality of candidate pose offsets is determined as the first pose offset.

54. The apparatus of claim 32, wherein: The determination module is further configured to: The environmental features, the map features and the initial pose are input into a trained pose solver to obtain the target pose offset output by the pose solver.

55. The apparatus of claim 41, wherein: The vectorized map includes surface elements, which are obtained through the following modules: A second acquisition module is configured to acquire a point cloud in a point cloud map; A division module, configured to divide the projection plane of the point cloud map into a plurality of two-dimensional grids of a first unit size; An extraction module is configured to extract, for any two-dimensional grid among the plurality of two-dimensional grids, a plane in the two-dimensional grid based on a point cloud in a three-dimensional space corresponding to the two-dimensional grid; as well as The storage module is configured to store the plane as a surface element in the vectorized map.

56. The apparatus of claim 55, wherein: The extraction module comprises: a dividing unit configured to divide the three-dimensional space into a plurality of three-dimensional grids of a second unit size along a height direction; The extracting unit is configured to: for any three-dimensional grid among the plurality of three-dimensional grids: Calculating the confidence that the three-dimensional grid includes a plane based on the point cloud in the three-dimensional grid; and In response to the confidence being greater than a threshold, extracting a plane in the three-dimensional grid; as well as The first determining unit is configured to determine a plane with the highest confidence in the multiple three-dimensional grids as a plane corresponding to the two-dimensional grid.

57. The apparatus of claim 56, wherein: The extraction unit comprises: a decomposition subunit configured to perform singular value decomposition on a covariance matrix of the point cloud in the three-dimensional grid to obtain a first singular value, a second singular value, and a third singular value, wherein the first singular value is less than or equal to the second singular value, and the second singular value is less than or equal to the third singular value; and The second determining subunit is configured to determine a ratio of the second singular value to the first singular value as the confidence level.

58. The device according to any one of claims 55 to 57, wherein: The storage module comprises: A second determining unit is configured to determine an identifier of a surface element corresponding to the plane; and The storage unit is configured to store the coordinates of a point on the plane and the unit normal vector of the plane in association with the identifier.

59. The apparatus of claim 55, wherein: The vectorized map further includes a plurality of road elements, any of which is: a lane line, a curb, a crosswalk, a stop line, a traffic sign or a pole.

60. A training device for a positioning model, wherein: The positioning model is used to implement the vehicle positioning device described in any one of claims 32-59, the positioning model includes an environment encoder, a map encoder and a pose solver, and the training device includes: An acquisition module is configured to acquire an initial posture of a sample vehicle, a true posture value corresponding to the initial posture, multimodal sensor data of the sample vehicle, and a plurality of map elements for locating the sample vehicle; A first input module is configured to input the multimodal sensor data into the environment encoder to obtain environmental features; A second input module is configured to input element information of the plurality of map elements into the map encoder to obtain map features; The third input module is configured to input the environment feature, the map feature and the initial pose into the pose solver so that the pose solver: Sampling within a first offset sampling range to obtain a plurality of first candidate pose offsets; For any first candidate pose offset among the plurality of first candidate pose offsets, determining a first degree of matching between the environment feature and the map feature under the first candidate pose offset; and Determine and output a predicted pose offset based on the first matching degrees corresponding to each of the plurality of first candidate pose offsets; A first determination module is configured to determine a first loss based on the predicted posture offset and a true value of the posture offset, wherein the true value of the posture offset is a difference between the true value of the posture and the initial posture; A second determination module is configured to determine a second loss based on the first matching degrees corresponding to each of the plurality of first candidate pose offsets, wherein the second loss indicates a difference between a predicted probability distribution of the pose truth value and a true probability distribution of the pose truth value; a determination module configured to determine an overall loss of the positioning model based at least on the first loss and the second loss; and The adjustment module is configured to adjust the parameters of the positioning model based on the overall loss.

61. The apparatus of claim 60, wherein: The pose solver is configured to: Sampling within the second offset sampling range to obtain a plurality of second candidate pose offsets; as well as For any second candidate pose offset among the plurality of second candidate pose offsets, determining a second degree of matching between the environment feature and the map feature under the second candidate pose offset; The device also includes: A third determination module is configured to determine a third loss based on the second matching degrees corresponding to the plurality of second candidate pose offsets, wherein the third loss indicates a difference between a predicted probability distribution of the plurality of candidate poses and a true probability distribution of the plurality of candidate poses, the plurality of candidate poses being obtained by superimposing the plurality of second candidate pose offsets with the initial pose, respectively; Wherein, the determining module is further configured to: The overall loss is determined based on at least the first loss, the second loss, and the third loss.

62. The device according to claim 60 or 61, wherein: The environmental features include an environmental feature map in a target three-dimensional space, the element information includes category information, and the map encoder is configured as follows: Determine the semantic coding corresponding to the category information based on the correspondence between the multiple category information and the multiple semantic codings, wherein the multiple semantic codings are parameters of the positioning model; The device also includes: a projection module configured to project a target map element belonging to a target category among the plurality of map elements into the target three-dimensional space to obtain a truth map of semantic segmentation in the target three-dimensional space, wherein a value of a first pixel in the truth map indicates whether the first pixel is occupied by the target map element; a prediction module configured to determine a prediction map for semantic segmentation based on the environmental feature map, wherein a value of a second pixel in the prediction map indicates a similarity between a corresponding environmental feature vector and a semantic encoding of the target category, and the corresponding environmental feature vector is a feature vector of a pixel in the environmental feature map corresponding to a position of the second pixel; and a fourth determination module, configured to determine a fourth loss based on the true value graph and the prediction graph; Wherein, the determining module is further configured to: The overall loss is determined based on at least the first loss, the second loss, and the fourth loss.

63. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 31.

64. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause a computer to execute the method according to any one of claims 1-31.

65. A computer program product comprising computer program instructions, wherein: The computer program instructions implement the method of any one of claims 1-31 when executed by a processor.

66. An autonomous driving vehicle comprising an electronic device according to claim 63.