Vehicle positioning method, neural network training method, and related device

Abstract

Embodiments of the present application provide a vehicle positioning method, a neural network training method, and a related device, in which the artificial intelligence technology can be applied to the field of intelligent driving. The method comprises: performing feature extraction on an image of a surrounding environment around a vehicle to obtain first feature information; performing feature extraction on map information in a preset range of a first location to obtain second feature information, wherein the first location is a location acquired by a positioning system deployed in the vehicle, and the map information comes from a standard-definition map; and on the basis of the first feature information and the second feature information, generating first prediction information by means of a first neural network, wherein the first prediction information is used for determining the actual location of the vehicle. The accuracy of the location (e.g. the location of a vehicle obtained by a GNSS) obtained by only a positioning system of the vehicle is low; therefore, in the present application, the location of the vehicle is finally determined on the basis of first location in combination with more visual information, so that the more accurate location of the vehicle can be obtained.

Description

A vehicle positioning method, a neural network training method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on February 29, 2024, with application number 202410231541.9 and invention name “A vehicle positioning method, a neural network training method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of intelligent driving, and in particular to a vehicle positioning method, a neural network training method, and related equipment. Background Art

[0003] In the field of intelligent driving, vehicle positioning is often required. Specifically, the vehicle can obtain vehicle location information with the help of the global navigation satellite system (GNSS). However, the accuracy of the vehicle location information obtained with the help of GNSS is limited, that is, there is often a large deviation between the vehicle location information obtained with the help of GNSS and the actual location of the vehicle. Therefore, a more accurate positioning solution is urgently needed. Summary of the Invention

[0004] The present application provides a vehicle positioning method, a neural network training method and related equipment, which integrates more visual information on the basis of the first position to finally determine the position of the vehicle, which is conducive to obtaining a more accurate position of the vehicle.

[0005] This application provides the following technical solutions:

[0006] In a first aspect, the present application provides a vehicle positioning method that applies artificial intelligence technology to the field of intelligent driving. In this method, an execution device can obtain at least one image of the vehicle's surroundings, and after obtaining a first position of the vehicle using the vehicle's positioning system, can obtain map information within a preset range of the first position. The vehicle's positioning system can include at least one of the following: a GNSS system, a positioning system using a base station for positioning, or other types of positioning systems deployed in the vehicle.

[0007] The execution device extracts features from an image of the vehicle's surroundings to obtain first feature information, and extracts features from map information within a preset range of the first position to obtain second feature information, where the map information within the preset range of the first position is derived from a precision map; and then, based on the first feature information and the second feature information, first prediction information is generated through a first neural network, where the first prediction information is used to determine the actual position of the vehicle.

[0008] For example, the accuracy of a standard map can be lower than that of a high-precision map. For example, the accuracy of a standard map can be at the meter level, while the accuracy of a high-precision map can be at the centimeter level. For example, various navigation maps deployed on mobile phones for people are generally standard maps, and the users of high-precision maps are generally computers. For example, a standard map only includes simple road lines, while a high-precision map will carry detailed lane lines, road components or other road information, etc., and a standard map will also include information about buildings in the environment.

[0009] For example, in one scenario, the first prediction information indicates a predicted offset between the vehicle's actual position and the first position; after obtaining the first prediction information generated by the first neural network, the vehicle can determine the vehicle's actual position based on the first prediction information and the first position, where the actual position is the actual position predicted by the neural network. In another scenario, the first prediction information indicates the vehicle's predicted actual position. For example, the first prediction information may include coordinate information corresponding to the vehicle's actual position and, optionally, the first prediction information may also include the vehicle's actual heading.

[0010] Since the accuracy of the vehicle's positioning system (such as the vehicle's position obtained by GNSS) alone is low in scenarios such as urban canyons, tunnels, or under overpasses with high buildings, in this implementation, after the vehicle's first position is obtained based on the vehicle's positioning system, map information within a preset range of the first position can be obtained, and feature extraction is performed on the map information within the preset range of the first position to obtain second feature information. Feature extraction is also performed on the image of the vehicle's surroundings to obtain first feature information, and then prediction information is generated through a neural network in combination with the first feature information and the second feature information. The prediction information is used to determine the actual position of the vehicle, that is, more visual information is integrated on the basis of the first position to finally determine the vehicle's position, which is conducive to obtaining a more accurate position of the vehicle. In addition, when this solution is applied to scenarios such as urban canyons or under overpasses with high buildings, there will be buildings in the vehicle's surroundings, and the image of the vehicle's surroundings will carry images of large buildings, and the standard map will also contain building information. Therefore, the map information within the preset range of the first position is derived from the standard map, which makes it easier to determine the vehicle's actual position.

[0011] In one possible implementation, the first neural network includes a transformer neural network module and a multi-layer perceptron (MLP). The execution device generates first prediction information based on the first feature information and the second feature information through the first neural network. Specifically, the execution device may fuse the first feature information and the second feature information to obtain fused feature information, and then input the fused feature information into the first neural network to obtain first prediction information generated by the first neural network. The first prediction information indicates the offset between the actual position of the vehicle and the first position.

[0012] For example, the first prediction information can use a first parameter and a second parameter to express the offset between the vehicle's actual position and the first position. In one case, the first parameter refers to the offset between the vehicle's actual position and the first position in the horizontal direction of the map, where "horizontal" can also be understood as the east-west direction, that is, the first parameter can refer to the offset distance between the vehicle's actual position and the first position in the east-west direction; the first parameter refers to the offset between the vehicle's actual position and the first position in the vertical direction of the map, where "longitudinal" can also be understood as the north-south direction, that is, the first parameter can refer to the offset distance between the vehicle's actual position and the first position in the north-south direction. In another case, the first parameter refers to the offset between the vehicle's actual position and the first position in the direction perpendicular to the vehicle's front, and the second parameter refers to the offset between the vehicle's actual position and the first position in the direction the vehicle's front is pointing. The vehicle can then decompose the first parameter into the east-west and north-south directions, and decompose the second parameter into the east-west and north-south directions, thereby obtaining the offset distance between the vehicle's actual position and the first position in the east-west direction, and the offset distance between the vehicle's actual position and the first position in the north-south direction.

[0013] Optionally, the first prediction information may also include a third parameter, which refers to the offset between the actual orientation of the vehicle and the first orientation. The first orientation can also be understood as the default orientation. The first orientation is often regarded as the orientation of due north or due east. The "offset between the actual orientation of the vehicle and the first orientation" can also be understood as the offset angle between the actual orientation of the vehicle and the due north orientation, or the "offset between the actual orientation of the vehicle and the first orientation" can also be understood as the offset angle between the actual orientation of the vehicle and the due east orientation.

[0014] In this implementation, a specific network structure of the first neural network is provided (that is, the first neural network includes a Transformer neural network module and a multi-layer perceptron MLP), which reduces the difficulty of implementing this solution; in addition, the use of the Transformer neural network module and the multi-layer perceptron MLP can generate the first prediction information more quickly, which is conducive to shortening the delay in the process of obtaining the actual position of the vehicle, that is, improving the efficiency of the process of obtaining the actual position of the vehicle.

[0015] In one possible implementation, the at least one captured image of the vehicle's surroundings may include at least two images of the vehicle's surroundings, where the at least two images include images captured from the vehicle in at least two of the following directions: the left front, the front, the right front, the left rear, the rear, or the right rear. In this implementation, capturing images from the vehicle in multiple directions facilitates a more accurate reflection of the vehicle's surroundings. Determining the vehicle's position information using images captured from the vehicle in multiple directions facilitates a more accurate final prediction of the vehicle's actual position.

[0016] In one possible implementation, the first feature information includes feature information of the vehicle's surroundings from a bird's-eye view. The execution device extracts features from an image of the vehicle's surroundings, which may include: the execution device extracts features from the image of the vehicle's surroundings using a second neural network. During the process of training the second neural network using a first loss function (i.e., during the training phase of the second neural network), the feature information generated by the second neural network is also input into a third neural network, which is used to generate second prediction information. The first loss function indicates the similarity between the second prediction information and second expected information. The second prediction information includes a predicted image of the vehicle's surroundings from a bird's-eye view, and the second prediction information also indicates a predicted category of an object in the predicted image from a bird's-eye view. The second expected information includes a correct image of the vehicle's surroundings from a bird's-eye view, and the second expected information also indicates a correct category of an object in the correct image from a bird's-eye view. That is, both the second prediction information and the second expected information can be represented as images carrying labeled information, where the image refers to an image of the vehicle's surroundings from a bird's-eye view. The difference is that the second prediction information is generated by the third neural network, and the second expected information is correct.

[0017] In this implementation, during the training phase of the second neural network, the feature information generated by the second neural network is also input into the third neural network, and second prediction information is generated by the third neural network. The second prediction information indicates a predicted image of the vehicle's surroundings from a bird's-eye view and a predicted category of objects in the image. The second neural network is then trained using a first loss function, which indicates the similarity between the second prediction information and second expected information. The second expected information includes a correct image of the vehicle's surroundings and the correct category of objects in the image. That is, semantic segmentation of the vehicle's surroundings is performed based on the feature information of the vehicle's surroundings from a bird's-eye view generated by the second neural network, and the correct semantic segmentation results are used as supervision to train the second neural network. Since the better the second feature information collected by the second neural network, the simpler the semantic segmentation is for the third neural network, and the easier it is to generate a better quality semantic segmentation result, the higher the similarity between the second prediction information and the second expected information. Training in this manner is conducive to improving the second neural network's ability to better understand the vehicle's surroundings, that is, it is conducive to ensuring that the first feature information extracted by the trained second neural network contains richer information.

[0018] In one possible implementation, in this method, the executing device may further rasterize the map information within a preset range at the first location in a first format to obtain the map information within the preset range at the first location in a second format, where the first format is a text format or a binary format, and the second format is an image format. The executing device then extracts features from the map information within the preset range at the first location, which may include: the executing device extracts features from the map information within the preset range at the first location in the second format.

[0019] Among them, "rasterizing the map information within the preset range of the first position in the first format" refers to performing a rendering operation based on the map information within the preset range of the first position in the first format to obtain map information in a two-dimensional image format. During the rasterization process, part of all the information included in the map information within the preset range of the first position in the first format will be extracted, so that the map information within the preset range of the first position in the second format only retains part of the map information within the preset range of the first position in the first format. Optionally, three types of information, namely area, line and node, can be extracted during the rasterization process, that is, the map information obtained after rasterization can retain the shape of the objects within the preset range of the first position and the relative positional relationship between different objects.

[0020] In this implementation, when the map information obtained by the vehicle is in text format or binary format, the map information in the first format can be rasterized first to obtain map information in a second format, which is an image format. Feature extraction is then performed on the map information in the second format. Since the map information in the image format can more intuitively display the shapes of objects within the preset range of the first position and the relative positions of different objects, feature extraction on the map information in the second format can more easily obtain rich image information; and after the rasterization process, only part of the important information in the map information within the preset range of the first position is retained, which is conducive to feature extraction of more critical information in the map information within the preset range of the first position, and is also conducive to the second feature information containing more important information, and thus is conducive to the first neural network generating more accurate prediction information with the help of the first feature information and the second feature information.

[0021] In one possible implementation, the executing device extracting features from map information within a preset range of the first location may include: the executing device directly inputting the acquired map information within the preset range of the first location into a fourth neural network, and performing feature extraction via the fourth neural network to obtain the second feature information. Optionally, if the map information within the preset range of the first location acquired in step 302 is in the first format, the fourth neural network may also employ a graph neural network (GNN), an attention-based neural network, or other types of neural networks.

[0022] In this implementation, since the map information in text form includes the coordinate information of multiple coordinate points and the category corresponding to each coordinate point, the map information in text form can be converted into a graph structure. Each vertex in the graph structure can be used to store the coordinate information of a coordinate point and the category information corresponding to the coordinate point. That is, the map information in text form can be well stored with the help of the graph structure, and the use of the graph structure neural network can better extract features of the information in the graph structure form. That is, when the graph neural network is used to extract features of the information in the graph structure form, rich information can be extracted from the information in the graph structure form, and thus second feature information with better quality can be obtained.

[0023] In a second aspect, the present application provides a neural network training method that can apply artificial intelligence technology to the field of intelligent driving. In this method, a training device inputs an image of the vehicle's surrounding environment into a second neural network, and the second neural network extracts features from the image of the vehicle's surrounding environment to obtain first feature information; the first feature information is input into a third neural network to obtain second prediction information generated by the third neural network, the second prediction information including a predicted image of the vehicle's surrounding environment from a bird's-eye view, and the second prediction information also indicates the predicted category of objects in the predicted image from the bird's-eye view; the second neural network is trained using a first loss function to obtain a trained second neural network, the first loss function indicating the similarity between the second prediction information and second expected information, the second expected information including a correct image of the vehicle's surrounding environment from a bird's-eye view, and the second expected information also indicating the correct category of objects in the correct image from the bird's-eye view.

[0024] In one possible implementation, the second expected information is obtained based on a standard precision map and a high-precision map corresponding to the vehicle's surroundings. For example, the second expected information corresponding to the training sample includes the presence of roads and buildings in a correct image of the vehicle's surroundings from a bird's-eye view; wherein the image of the road portion in the correct image of the vehicle's surroundings from a bird's-eye view can be obtained based on the high-precision map, and the image of the building portion in the correct image of the vehicle's surroundings from a bird's-eye view can be obtained based on the standard precision map; the correct category of objects on the road in the vehicle's surroundings can be obtained based on the high-precision map corresponding to the vehicle's surroundings, and the correct category of buildings in the vehicle's surroundings can be obtained based on the standard precision map corresponding to the vehicle's surroundings.

[0025] In this implementation, since the high-precision map carries detailed road information and the standard-precision map carries building information in the environment, the second expected information includes a correct image of the vehicle's surroundings from a bird's-eye view, and the second expected information also indicates the correct category of objects in the correct image from a bird's-eye view, the high-precision map and the standard-precision map are used simultaneously to generate the second expected information, which is conducive to the second expected information containing both high-quality road information and building information in the environment, that is, the second expected information can more comprehensively and accurately reflect the vehicle's surroundings. Using the second expected information as a guide to train the second neural network is conducive to the trained second neural network having a better understanding of the vehicle's surroundings, that is, it is conducive to the first feature information extracted by the trained second neural network covering richer information, and thus it is conducive to the first neural network generating more accurate prediction information with the help of the first feature information and the second feature information.

[0026] In one possible implementation, in this method, the training device may further perform feature extraction on map information within a preset range of the first location to obtain second feature information, where the first location represents the location of the vehicle obtained based on the vehicle's positioning system; based on the first feature information and the second feature information, first prediction information is generated through a first neural network, and the first prediction information is used to determine the predicted actual location of the vehicle. The training device then trains the second neural network using a first loss function to obtain a trained second neural network, which may include: the training device trains the first neural network and the second neural network using a first loss function and a second loss function to obtain a trained first neural network and a trained second neural network, where the second loss function indicates the similarity between the first prediction information and the first expected information, and the first expected information is used to determine the correct actual location of the vehicle.

[0027] In the second aspect of the present application, the training device can also execute the steps performed by the device in the various possible implementation methods of the first aspect. For the meaning of the nouns in the second aspect of the present application and the various possible implementation methods of the second aspect, as well as the beneficial effects brought about by each possible implementation method, please refer to the description of the various possible implementation methods in the first aspect, and will not be repeated here one by one.

[0028] In a third aspect, the present application provides a vehicle positioning device that can apply artificial intelligence technology to the field of intelligent driving. The vehicle positioning device includes: an acquisition module for acquiring an image of the vehicle's surroundings and map information within a preset range of a first position, where the first position is the vehicle's position obtained based on the vehicle's positioning system; a feature extraction module for extracting features from the image of the vehicle's surroundings to obtain first feature information; the feature extraction module is also used to extract features from the map information within the preset range of the first position to obtain second feature information; and a generation module for generating first prediction information through a first neural network based on the first feature information and the second feature information, wherein the first prediction information is used to determine the actual position of the vehicle.

[0029] In one possible implementation, the first neural network includes a Transformer neural network module and a multi-layer perceptron MLP, and a generation module, which is specifically used to: fuse the first feature information and the second feature information to obtain fused feature information; input the fused feature information into the first neural network to obtain first prediction information generated by the first neural network, and the first prediction information indicates the offset between the actual position of the vehicle and the first position.

[0030] In one possible implementation, the image of the vehicle's surroundings includes at least two images of the vehicle's surroundings, and the at least two images include images captured by the vehicle in at least two of the following directions: left front, front, right front, left rear, rear, or right rear.

[0031] In one possible implementation, the first feature information includes feature information of the vehicle's surroundings from a bird's-eye view. The feature extraction module is specifically used to extract features from an image of the vehicle's surroundings through a second neural network. During training of the second neural network using a first loss function, the feature information generated by the second neural network is also input into a third neural network. The third neural network is used to generate second prediction information. The first loss function indicates the similarity between the second prediction information and the second expected information. The second prediction information includes a predicted image of the vehicle's surroundings from a bird's-eye view, and the second prediction information also indicates a predicted category of an object in the predicted image from a bird's-eye view. The second expected information includes a correct image of the vehicle's surroundings from a bird's-eye view, and the second expected information also indicates a correct category of an object in the correct image from a bird's-eye view.

[0032] In one possible implementation, the vehicle positioning device also includes: a rasterization module, used to rasterize the map information within a preset range of the first position in a first format to obtain the map information within the preset range of the first position in a second format, where the first format is a text format or a binary format, and the second format is an image format; and a feature extraction module, specifically used to extract features from the map information within the preset range of the first position in the second format.

[0033] The specific implementation methods, meanings of terms and beneficial effects of the steps in the third aspect can all be referred to the first aspect and will not be repeated here.

[0034] In a fourth aspect, the present application provides a neural network training device that can apply artificial intelligence technology to the field of intelligent driving. The neural network training device includes: an input module for inputting an image of the vehicle's surrounding environment into a second neural network, and extracting features from the image of the vehicle's surrounding environment through the second neural network to obtain first feature information; a generation module for inputting the first feature information into a third neural network to obtain second prediction information generated by the third neural network, the second prediction information including a predicted image of the vehicle's surrounding environment from a bird's-eye view, and the second prediction information also indicating the predicted category of objects in the predicted image from a bird's-eye view; a training module for training the second neural network using a first loss function to obtain a trained second neural network, the first loss function indicating the similarity between the second prediction information and second expected information, the second expected information including a correct image of the vehicle's surrounding environment from a bird's-eye view, and the second expected information also indicating the correct category of objects in the correct image from a bird's-eye view.

[0035] In one possible implementation, the second desired information is obtained based on a standard precision map and a high precision map corresponding to the vehicle's surrounding environment.

[0036] In one possible implementation, the neural network training device also includes: a feature extraction module, which is used to extract features from map information within a preset range of a first position to obtain second feature information, where the first position represents the position of the vehicle obtained based on a vehicle positioning system; a generation module, which is also used to generate first prediction information through a first neural network based on the first feature information and the second feature information, and the first prediction information is used to determine the predicted actual position of the vehicle; a training module, which is specifically used to train the first neural network and the second neural network using a first loss function and a second loss function to obtain a trained first neural network and a trained second neural network, where the second loss function indicates the similarity between the first prediction information and the first expected information, and the first expected information is used to determine the correct actual position of the vehicle.

[0037] In the fourth aspect of this application, the neural network training device is also used to execute the steps performed by the training device in the second aspect and various possible implementation methods of the second aspect. The specific implementation methods, meanings of terms and beneficial effects of the steps in the fourth aspect can all be referred to the second aspect and will not be repeated here.

[0038] In a fifth aspect, an embodiment of the present application provides a device comprising a processor and a memory, wherein the processor is coupled to the memory, the memory is used to store programs, and the processor is used to execute the programs in the memory, so that the device executes the method described in the first or second aspect above.

[0039] In a sixth aspect, an embodiment of the present application provides a vehicle comprising a processor and a memory, wherein the processor is coupled to the memory, the memory being used to store programs; and the processor being used to execute the programs in the memory, so that the vehicle executes the method described in the first aspect above.

[0040] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method described in the first or second aspect above.

[0041] In an eighth aspect, an embodiment of the present application provides a computer program product, which includes a program. When the program runs on a computer, it enables the computer to execute the method described in the first or second aspect above.

[0042] In a ninth aspect, the present application provides a chip system, which includes a processor for supporting the implementation of the functions involved in the above aspects, for example, sending or processing the data and / or information involved in the above methods. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the terminal device or communication device. The chip system can be composed of a chip or can include a chip and other discrete devices.

[0043] The second to eighth aspects of this application correspond to various possible methods of the first aspect and have corresponding beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1 is a schematic diagram of the structure of an artificial intelligence main framework provided in an embodiment of the present application;

[0045] FIG2 is a schematic diagram of an architecture of a vehicle positioning system provided in an embodiment of the present application;

[0046] FIG3 is a schematic diagram of a flow chart of a vehicle positioning method provided in an embodiment of the present application;

[0047] FIG4 is a schematic diagram of an image of the vehicle surrounding environment acquired according to an embodiment of the present application;

[0048] FIG5 is a schematic diagram of extracting features from an image of the vehicle's surrounding environment to obtain first feature information;

[0049] FIG6 is a schematic diagram of extracting features from map information within a preset range of a first location to obtain second feature information according to an embodiment of the present application;

[0050] FIG7 is a schematic diagram of an image of the vehicle's surrounding environment, first feature information, second feature information, and the actual position of the vehicle provided in an embodiment of the present application;

[0051] FIG8 is a schematic diagram of generating first prediction information by a first neural network according to an embodiment of the present application;

[0052] FIG9 is another schematic flow chart of a vehicle positioning method according to an embodiment of the present application;

[0053] FIG10 is a flow chart of a neural network training method according to an embodiment of the present application;

[0054] FIG11 is another flow chart of a neural network training method according to an embodiment of the present application;

[0055] FIG12 is a schematic structural diagram of a vehicle positioning device provided in an embodiment of the present application;

[0056] FIG13 is a schematic diagram of a structure of a neural network training device provided in an embodiment of the present application;

[0057] FIG14 is a schematic structural diagram of a device provided in an embodiment of the present application;

[0058] FIG15 is another schematic diagram of the structure of the device provided in an embodiment of the present application;

[0059] FIG16 is a schematic structural diagram of a vehicle provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present application, rather than all of the embodiments. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0061] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances. This is merely a way of distinguishing when describing objects with the same properties in the embodiments of the present application. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, so that a process, method, system, product or apparatus that includes a series of units is not necessarily limited to those units, but may include other units not expressly listed or inherent to these processes, methods, products or apparatuses.

[0062] In the embodiments of the present application, "indication" may include direct indication and indirect indication, and may also include explicit indication and implicit indication. The information indicated by a certain information (such as the indication information described below) is called information to be indicated. In the specific implementation process, there are many ways to indicate the information to be indicated, such as but not limited to, the information to be indicated can be directly indicated, such as the information to be indicated itself or the index of the information to be indicated. The information to be indicated can also be indirectly indicated by indicating other information, wherein there is an association between the other information and the information to be indicated; it is also possible to indicate only a part of the information to be indicated, while the other parts of the information to be indicated are known or agreed in advance, for example, the indication of specific information can be achieved with the help of the arrangement order of each information agreed in advance (such as predefined by the protocol), thereby reducing the indication overhead to a certain extent. The present application does not limit the specific method of indication. It is understandable that, for the sender of the indication information, the indication information can be used to indicate the information to be indicated, and for the receiver of the indication information, the indication information can be used to determine the information to be indicated.

[0063] First, let's describe the overall workflow of an AI system. See Figure 1, which shows a schematic diagram of the AI ​​framework. This framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0064] (1) Infrastructure

[0065] The infrastructure provides computing power for AI systems, enabling communication with the outside world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by smart chips, which can specifically adopt hardware acceleration chips such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs); the basic platform includes related platform guarantees and support such as distributed computing frameworks and networks, and can include cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to obtain data, which is then provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0066] (2) Data

[0067] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0068] (3) Data processing

[0069] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0070] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0071] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0072] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0073] (4) General ability

[0074] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0075] (5) Smart products and industry applications

[0076] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart manufacturing, smart transportation, smart homes, smart medical care, smart security, smart driving, smart cities, etc.

[0077] The method provided in this application can be applied to various scenarios requiring vehicle positioning. For example, in the field of intelligent driving, vehicles can provide navigation functions, and it is necessary to plan navigation routes for the vehicles based on their actual locations. The vehicles can include cars, trucks, motorcycles, buses, boats, lawn mowers, recreational vehicles, amusement park vehicles, construction equipment, trams, golf carts, trains, airplanes, and helicopters, among others, and the embodiments of this application do not impose any particular limitations.

[0078] Currently, the vehicle's position can be obtained using the global navigation satellite system (GNSS) deployed in the vehicle. However, there is often a large deviation between the vehicle's position information obtained by GNSS and the vehicle's actual position. Therefore, a more accurate positioning solution is urgently needed.

[0079] In order to solve the above problems, the present application provides a vehicle positioning method, which discloses that after the first position of the vehicle is obtained based on the vehicle positioning system, map information within a preset range of the first position can be obtained, and feature extraction is performed on the map information within the preset range of the first position to obtain second feature information. Feature extraction is also performed on the image of the vehicle's surrounding environment to obtain first feature information, and then the first feature information and the second feature information are combined to generate prediction information through a neural network. The prediction information is used to determine the actual position of the vehicle, that is, based on the first position obtained based on the positioning system, more visual information is integrated to finally determine the position of the vehicle, which is conducive to obtaining a more accurate position of the vehicle.

[0080] Before providing a detailed introduction to the method provided herein, the architecture of the vehicle positioning system provided herein will be described. Please refer to Figure 2, which is a schematic diagram of the architecture of a vehicle positioning system provided in an embodiment of the present invention. As shown in Figure 2, vehicle positioning system 200 includes a training device 210, a database 220, an execution device 230, and a data storage system 240. Execution device 230 includes a computing module 231.

[0081] The database 220 stores a training data set. During the training phase of the first neural network 201, the training device 210 generates the first neural network 201 and iteratively trains the first neural network 201 using the training data set, thereby obtaining a trained first neural network 201. The "trained first neural network 201" may also be referred to as the "trained first neural network 201." The first neural network 201 may be specifically a neural network or a non-neural network model. In the embodiments of the present application, the first neural network 201 is described as a neural network.

[0082] The first neural network 201 that has undergone training operations and is obtained by the training device 210 can be deployed to the computing module 231 of the execution device 230. The execution device 230 can access data, code, etc. in the data storage system 240, or store data, instructions, etc. in the data storage system 240. The data storage system 240 can be located within the execution device 230, or it can be an external memory relative to the execution device 230.

[0083] During the application phase of the first neural network 201 , the execution device 230 may determine whether the vehicle deviates when passing a fork in the road based on the image or point cloud data of the vehicle's surroundings through the trained first neural network 201 .

[0084] In some embodiments of the present application, referring to FIG. 2 , the execution device 230 and the client device may be integrated into the same device, so that the user can directly interact with the execution device 230. For example, when the client device is a vehicle, the execution device 230 may be a module in the vehicle's host CPU that uses the first neural network to perform data processing. The execution device 230 may also be a graphics processing unit (GPU) or a neural network processor (NPU) in the vehicle. The GPU or NPU is mounted on the host processor as a coprocessor, and the host processor assigns tasks.

[0085] It is worth noting that FIG2 is only a schematic diagram of the architecture of the vehicle positioning system provided by an embodiment of the present invention, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in other embodiments of the present application, the execution device 230 and the client device can be independent devices, and the execution device 230 is configured with an input / output (I / O) interface to exchange data with the client device. The client device sends an image of the vehicle's surrounding environment to the execution device 230 through the I / O interface. After the execution device 230 determines the actual position of the vehicle through the first neural network 201 in the computing module 231, it can return the actual position of the aforementioned vehicle to the client device through the I / O interface.

[0086] In combination with the above description, the following describes the specific implementation process of the training phase and the application phase in the method provided by this application.

[0087] 1. Application stage

[0088] In the embodiments of the present application, the application phase refers to the process by which the execution device uses the trained first neural network to determine the actual position of the vehicle. The execution device can be specifically manifested as a vehicle or a processor in the vehicle. The subsequent embodiments will only be described using the execution device as a vehicle as an example. When the execution device is manifested as other product forms, it can be understood by reference. The embodiments of the present application will not be described in detail. Specifically, please refer to Figure 3, which is a flow chart of a vehicle positioning method provided in the embodiments of the present application. The vehicle positioning method provided in the embodiments of the present application may include:

[0089] 301. Acquire an image of the vehicle's surroundings.

[0090] In an embodiment of the present application, the vehicle's position information can be determined with the help of at least one image of the vehicle's surroundings; optionally, the aforementioned image of the vehicle's surroundings includes at least two images of the vehicle's surroundings, and the at least two images of the vehicle's surroundings include images collected by the vehicle in at least two of the following directions: left front, front, right front, left rear, rear, right rear or images in other directions, etc.; the specific directions in which the vehicle's images are obtained can be flexibly determined based on actual conditions, and are not limited in this application.

[0091] For a more intuitive understanding of the present solution, please refer to Figure 4, which is a schematic diagram of an image of the vehicle's surrounding environment obtained according to an embodiment of the present application. In Figure 4, the image of the vehicle's surrounding environment obtained includes images of the vehicle's left front, front, right front, left rear, rear and right rear as an example, that is, the image of the vehicle's surrounding environment obtained includes a surround view image of the vehicle. It should be understood that the example in Figure 4 is only for the convenience of understanding the present solution and is not used to limit the present solution.

[0092] In an embodiment of the present application, obtaining images collected from the vehicle in multiple directions is conducive to more accurately reflecting the vehicle's surrounding environment. Determining the vehicle's position information with the help of images collected from the vehicle in multiple directions is conducive to making the final predicted actual position of the vehicle more accurate.

[0093] 302. Obtain map information within a preset range of a first position, where the first position is a position of the vehicle obtained based on a vehicle positioning system, and the map information within the preset range of the first position is derived from a standard map.

[0094] In an embodiment of the present application, when it is necessary to obtain the vehicle's location information, the vehicle's first location can also be obtained based on the vehicle's positioning system, and then map information within a preset range of the first location (hereinafter referred to as the "first preset range" for ease of distinction) can be obtained.

[0095] For example, the vehicle's positioning system may include at least one of the following: a GNSS system, a positioning system using a base station for positioning, or other types of positioning systems deployed in the vehicle, etc., which are not limited in this application. The "first position of the vehicle" can also be understood as the initial position of the vehicle obtained based on the vehicle's positioning system.

[0096] For example, a navigation map may be deployed in the vehicle. After determining the first position, the vehicle may obtain map information within a first preset range of the first position based on the navigation map. For example, the navigation map may be a standard-definition map (SD Map), a high-definition map (HD Map), or other types of maps.

[0097] For example, the accuracy of a standard map can be lower than that of a high-precision map. For example, the accuracy of a standard map can be at the meter level, while the accuracy of a high-precision map can be at the centimeter level. For example, various navigation maps deployed on mobile phones for people are generally standard maps, and the users of high-precision maps are generally computers. For example, a standard map only includes simple road lines, while a high-precision map will carry detailed lane lines, road components or other road information, etc., and a standard map will also include information about buildings in the environment.

[0098] For example, the "first preset range of the first position" can be expressed as a circular area with the first position as the center, and the radius of the circular area can be a first preset length; illustratively, the first preset length can be 40 meters, 50 meters, 60 meters, 80 meters, 100 meters, or other values, etc., which are not limited in this application. For another example, the "first preset range of the first position" can be expressed as a square area with the first position as the center, and the side length of the square area can be a second preset length, which can be 80 meters, 100 meters, 120 meters, 160 meters, 200 meters, or other values, etc., which are not limited in this application.

[0099] 303. Perform feature extraction on the image of the vehicle's surrounding environment to obtain first feature information.

[0100] In an embodiment of the present application, when it is necessary to obtain the location information of the vehicle, the vehicle can extract features from the image of the vehicle's surrounding environment obtained in step 301 through a second neural network to obtain first feature information.

[0101] Exemplarily, the second neural network can be an encoder. For example, the second neural network can specifically adopt a convolutional neural network (CNN), a residual neural network, a neural network based on an attention mechanism, etc. The specific type of neural network to be adopted can be flexibly determined in combination with the actual application scenario.

[0102] Optionally, the first feature information includes feature information of the vehicle surroundings from a top-down perspective, and “feature information of the vehicle surroundings from a top-down perspective” may also be referred to as “feature information of the vehicle surroundings from a bird’s-eye view”.

[0103] Exemplarily, in step 303, the vehicle can perform feature extraction on each image of the vehicle's surroundings through a second neural network to obtain third feature information of each image, where the third feature information refers to feature information obtained without perspective conversion, and "feature information obtained without perspective conversion" can also be understood as feature information of the vehicle's surroundings under a perspective; the third feature information is converted into feature information of each image under a bird's-eye view through the second neural network, and the feature information of all images of the vehicle's surroundings under a bird's-eye view is fused through the second neural network to obtain the first feature information.

[0104] Alternatively, after obtaining the third feature information for each image, the vehicle can convert the third feature information for each image into feature information in the camera coordinate system using a second neural network based on the intrinsic parameters of the camera that captured each image of the vehicle's surroundings. Furthermore, based on the extrinsic parameters of the aforementioned camera, the second neural network can be used to geometrically project the feature information in the camera coordinate system to obtain feature information for each image from a bird's-eye view. Finally, the second neural network can be used to fuse the feature information from a bird's-eye view of all images of the vehicle's surroundings to obtain the first feature information. For example, the camera's extrinsic parameters can include the camera's installation position in the vehicle.

[0105] Alternatively, after obtaining the third feature information of each image, the vehicle can also directly convert the third feature information of each image (that is, the feature information of each image at a perspective perspective) into the feature information of each image at a bird's-eye view through a second neural network based on the internal and external parameters of the camera that captures each image of the vehicle's surrounding environment, and then fuse the feature information of all images of the vehicle's surrounding environment at a bird's-eye view to obtain the first feature information.

[0106] For a more intuitive understanding of this solution, please refer to Figure 5, which is a schematic diagram of extracting features from images of the vehicle's surroundings to obtain first feature information. The images of the vehicle's surroundings obtained in Figure 5 include six images of the vehicle's left front, front, right front, left rear, rear and right rear as an example. As shown in Figure 5, feature extraction can be performed on each image of the vehicle's surroundings through a second neural network to obtain third feature information of each image; the third feature information of each image is then converted into feature information of each image from a bird's-eye view through a second neural network; and the feature information of the six images of the vehicle's surroundings from a bird's-eye view are then fused to obtain first feature information. It should be understood that the first feature information shown in Figure 5 is an image obtained after visualization of the first feature information. It should be understood that the example in Figure 5 is only for the convenience of understanding this solution and is not used to limit this solution.

[0107] Optionally, during the process of training the second neural network using the first loss function, that is, during the training phase of the second neural network, the feature information generated by the second neural network is also input into a third neural network, and the third neural network is used to generate second prediction information. The aforementioned first loss function indicates the similarity between the second prediction information and the second expected information; wherein the second prediction information includes a predicted image of the vehicle's surroundings from a bird's-eye view, and the second expected information also indicates the predicted category of objects in the predicted image from a bird's-eye view; the second expected information includes a correct image of the vehicle's surroundings from a bird's-eye view, and the second expected information also indicates the correct category of objects in the correct image from a bird's-eye view. It should be noted that the specific implementation of the "training phase of the second neural network" will be described in detail in subsequent embodiments and will not be described in detail here.

[0108] Optionally, the vehicle may also acquire at least one point cloud data of the vehicle's surroundings. Optionally, the aforementioned point cloud data of the vehicle's surroundings includes at least two point cloud data of the vehicle's surroundings. The at least two point cloud data of the vehicle's surroundings include point cloud data collected by the vehicle in at least two of the following directions: left front, front, right front, left rear, rear, right rear, or point cloud data in other directions, etc., which are not limited in the embodiments of the present application. Then, in step 303, the vehicle may also perform feature extraction on the at least one point cloud data of the vehicle's surroundings using another neural network to obtain feature information of the aforementioned point cloud data. After the vehicle performs feature extraction on the image of the vehicle's surroundings using the second neural network, feature information of each image of the vehicle's surroundings in a bird's-eye view can be obtained. After the feature information of all images of the vehicle's surroundings in a bird's-eye view is fused, fused feature information can be obtained. After the vehicle further fuses the feature information of the point cloud data with the fused feature information, first feature information can be obtained.

[0109] 304. Perform feature extraction on map information within a first preset range of the first location to obtain second feature information.

[0110] In an embodiment of the present application, when it is necessary to obtain the location information of the vehicle, after obtaining the map information within the first preset range of the first location, the vehicle can also perform feature extraction on the map information within the first preset range of the first location through a fourth neural network to obtain second feature information, where the second feature information includes feature information of the map information within the first preset range of the first location.

[0111] Exemplarily, the fourth neural network can be an encoder. For example, the fourth neural network can specifically adopt a convolutional neural network, a residual neural network, a neural network based on an attention mechanism, a graph neural network, etc., which can be determined in combination with the actual application scenario and is not limited in the embodiments of the present application.

[0112] In one case, the map information within the first preset range of the first location obtained in step 302 is in a first format, which can be a text format, a binary format, or another format; the map information in text format can be a text description that introduces what objects are in the environment and the locations of the objects in the environment, etc. For example, the map information in text format can include coordinate information of multiple coordinate points and the category corresponding to each of the multiple coordinate points. For example, the category corresponding to each coordinate point can be a building, a traffic light pole, a lane line, or another category, etc. Optionally, if the navigation map is a high-precision map, the map information in text format can also include the number of lanes. In another case, the map information within the first preset range of the first location obtained in step 302 can be an image format map information.

[0113] If the vehicle acquires map information within a first preset range at a first location in a first format, the map information within the first preset range at the first location in the first format may optionally be rasterized before step 304 to obtain map information within the first preset range at the first location in a second format, where the second format is an image format. In one implementation, step 304 may include: the vehicle performing feature extraction on the map information within the first preset range at the first location in the second format using a fourth neural network.

[0114] Here, “rasterizing the map information within the first preset range of the first position in the first format” refers to performing a rendering operation based on the map information within the first preset range of the first position in the first format to obtain map information in a two-dimensional image format.

[0115] During the rasterization process, a portion of all information included in the map information within the first preset range of the first position in the first format is extracted, so that the map information within the first preset range of the first position in the second format retains only the portion of the map information within the first preset range of the first position in the first format. Optionally, during the rasterization process, three types of information can be extracted: areas, lanes, and nodes. That is, the resulting map information after rasterization can retain the shapes of objects within the first preset range of the first position and the relative positional relationships between different objects.

[0116] For a more intuitive understanding of this solution, please refer to FIG6 . FIG6 is a schematic diagram of an embodiment of the present application, providing a method for extracting features from map information within a first preset range at a first location to obtain second feature information. As shown in FIG6 , after obtaining map information within a first preset range at a first location in a first format, the vehicle first rasterizes the map information within the first preset range at the first location in the first format to obtain map information within the first preset range at the first location in a second format. FIG6 also shows the map information obtained after visualizing the map information within the first preset range at the first location in the first format. Comparing the map information in the first format (the image obtained after visualization) with the map information in the second format, it can be seen that the rasterization process only retains the shapes of objects within the first preset range at the first location and the relative positional relationships between different objects. The vehicle then extracts features from the map information within the first preset range at the first location in the second format using a fourth neural network to obtain second feature information. FIG6 shows a schematic diagram of the visualization of the second feature information. It should be noted that the second feature information shown in FIG6 is an image obtained after visualizing the second feature information. The example in FIG6 is provided for ease of understanding of this solution and is not intended to limit this solution.

[0117] In an embodiment of the present application, when the map information acquired by the vehicle is in text format or binary format, the map information in the first format can be rasterized first to obtain map information in a second format, where the second format is an image format, and then feature extraction is performed on the map information in the second format. Since the map information in the image format can more intuitively display the shapes of objects within the first preset range of the first position and the relative positions of different objects, feature extraction on the map information in the second format can more easily obtain rich image information; and after the rasterization process, only part of the important information in the map information within the first preset range of the first position is retained, which is conducive to feature extraction of more critical information in the map information within the first preset range of the first position, and is also conducive to the second feature information containing more important information, and thus is conducive to the first neural network generating more accurate prediction information with the help of the first feature information and the second feature information.

[0118] In another implementation, step 304 may include: directly inputting the map information within the first preset range of the first location acquired in step 302 into the fourth neural network, and performing feature extraction through the fourth neural network to obtain second feature information.

[0119] Optionally, if the map information within the first preset range of the first position obtained in step 302 is in the first format, the fourth neural network may also adopt a graph neural network (GNN), a neural network based on an attention mechanism, or other types of neural networks. Since the map information in text form includes the coordinate information of multiple coordinate points and the category corresponding to each coordinate point, the map information in text form can be converted into a graph structure. Each vertex in the graph structure can be used to store the coordinate information of a coordinate point and the category information corresponding to the coordinate point. For example, the category information corresponding to the coordinate point can be an embedded feature obtained by vectorizing the category corresponding to the coordinate point. That is, the map information in text form can be well stored with the help of the graph structure, and the graph structure neural network can better perform feature extraction on the information in the graph structure form. That is, when the graph neural network is used to extract features from the information in the graph structure form, rich information can be extracted from the information in the graph structure form, and thus better quality second feature information can be obtained.

[0120] 305. Based on the first feature information and the second feature information, generate first prediction information through a first neural network, and the first prediction information is used to determine the actual position of the vehicle.

[0121] In the embodiment of the present application, for example, the first neural network may include a transformer neural network module and a multi-layer perceptron (MLP), or the first neural network may include a convolutional neural network and an MLP, or the first neural network may include a graph neural network and an MLP, or the first neural network may specifically adopt an MLP, or the first neural network may also be specifically expressed as other network structures, which is not limited in the embodiment of the present application.

[0122] In one case, the first prediction information indicates a predicted offset between the actual position of the vehicle and the first position.

[0123] For example, the first prediction information can use a first parameter and a second parameter to express the offset between the vehicle's actual position and the first position. In one case, the first parameter refers to the offset between the vehicle's actual position and the first position in the horizontal direction of the map, where "horizontal" can also be understood as the east-west direction, that is, the first parameter can refer to the offset distance between the vehicle's actual position and the first position in the east-west direction; the first parameter refers to the offset between the vehicle's actual position and the first position in the vertical direction of the map, where "longitudinal" can also be understood as the north-south direction, that is, the first parameter can refer to the offset distance between the vehicle's actual position and the first position in the north-south direction. In another case, the first parameter refers to the offset between the vehicle's actual position and the first position in the direction perpendicular to the vehicle's front, and the second parameter refers to the offset between the vehicle's actual position and the first position in the direction the vehicle's front is pointing. The vehicle can then decompose the first parameter into the east-west and north-south directions, and decompose the second parameter into the east-west and north-south directions, thereby obtaining the offset distance between the vehicle's actual position and the first position in the east-west direction, and the offset distance between the vehicle's actual position and the first position in the north-south direction.

[0124] Optionally, the first prediction information may also include a third parameter, which refers to the offset between the actual orientation of the vehicle and the first orientation. The first orientation can also be understood as the default orientation. The first orientation is often regarded as the orientation of due north or due east. The "offset between the actual orientation of the vehicle and the first orientation" can also be understood as the offset angle between the actual orientation of the vehicle and the due north orientation, or the "offset between the actual orientation of the vehicle and the first orientation" can also be understood as the offset angle between the actual orientation of the vehicle and the due east orientation.

[0125] For a more intuitive understanding of the present solution, please refer to Figure 7, which is a schematic diagram of an image of the vehicle's surroundings, first characteristic information, second characteristic information, and the actual position of the vehicle provided in an embodiment of the present application. Figure 7 includes four sub-schematic diagrams (a), (b), (c), and (d). The (a) sub-schematic diagram of Figure 7 represents that the image of the vehicle's surroundings acquired in step 301 includes four images. The (b) sub-schematic diagram of Figure 7 represents an image obtained after visualizing the first characteristic information, that is, an image obtained after visualizing the characteristic information of the vehicle's surroundings from a bird's-eye view. The (c) sub-schematic diagram of Figure 7 represents an image obtained after visualizing the second characteristic information, that is, an image obtained after visualizing the characteristic information of the map information within the first preset range of the first position. The (c) sub-schematic diagram of Figure 7 also shows a box and a black dot. The black dot represents the center point of the vehicle's surroundings, and the box represents the range obtained by mapping the vehicle's surroundings to the first preset range of the first position. In the schematic diagram (d) of Figure 7, the large gray box represents the first preset range of the first position, the small gray box represents the range of the vehicle's surroundings, the gray dot in the lower left corner represents the first position of the vehicle, and the initial direction of the vehicle is due north. The gray dot is located in the lower left corner of the white dotted box; the white dot represents the predicted actual position of the vehicle obtained by the neural network prediction, and the white dotted box represents the predicted orientation of the vehicle. It should be understood that the example in Figure 7 is only for the convenience of understanding this solution and is not used to limit this solution.

[0126] Specifically, in one implementation, step 305 may include: the vehicle fuses the first feature information and the second feature information to obtain fused feature information, and then inputs the fused feature information into the first neural network to obtain the first neural network to generate first prediction information, the first prediction information indicating the offset between the actual position of the vehicle and the first position; the vehicle determines the actual position of the vehicle based on the first prediction information and the first position, and the actual position is the actual position predicted by the neural network.

[0127] For a more intuitive understanding of the present solution, please refer to Figure 8. Figure 8 is a schematic diagram of generating first prediction information through a first neural network provided in an embodiment of the present application. As shown in Figure 8, taking the first neural network including a transformer module and a multi-layer perceptron (MLP) as an example, after obtaining the first feature information and the second feature information, the vehicle first fuses the first feature information and the second feature information to obtain the fused feature information, and inputs the fused feature information into the first neural network to obtain the first prediction information output by the first neural network. The first prediction information indicates the offset between the actual position of the vehicle and the first position. It should be understood that the example in Figure 9 is only for the convenience of understanding the present solution and is not used to limit the present solution.

[0128] In an embodiment of the present application, a specific network structure of the first neural network is provided (that is, the first neural network includes a Transformer neural network module and a multi-layer perceptron MLP), which reduces the difficulty of implementing the present solution; in addition, the use of the Transformer neural network module and the multi-layer perceptron MLP can generate the first prediction information more quickly, which is conducive to shortening the delay in the process of obtaining the actual position of the vehicle, that is, improving the efficiency of the process of obtaining the actual position of the vehicle.

[0129] In another implementation, the vehicle may also directly input the first feature information and the second feature information into the first neural network, so that the first neural network generates first prediction information, where the first prediction information indicates the offset between the actual position of the vehicle and the first position; the vehicle determines the actual position of the vehicle based on the first prediction information and the first position, and the actual position is the actual position predicted by the neural network.

[0130] For a more intuitive understanding of the present solution, please refer to Figure 9, which is a schematic diagram of a vehicle positioning method provided in an embodiment of the present application. As shown in Figure 9, the vehicle can obtain the first position of the vehicle through a positioning system deployed on the vehicle; the vehicle can extract features from an image of the vehicle's surroundings through a second neural network to obtain first feature information; the fourth neural network can extract features from map information within a first preset range of the first position to obtain second feature information; based on the first feature information and the second feature information, first prediction information is generated through the first neural network. In Figure 9, the first prediction information indicates the offset between the first position and the actual position of the vehicle as an example; the vehicle can finally determine the actual position of the vehicle based on the first position and the first prediction information. It should be understood that the example in Figure 9 is only for the convenience of understanding the present solution and is not used to limit the present solution.

[0131] In another embodiment, the first prediction information indicates a predicted actual position of the vehicle. For example, the first prediction information may include coordinate information corresponding to the actual position of the vehicle. Optionally, the first prediction information may also include the actual orientation of the vehicle. In this embodiment, step 305 may include: the vehicle generates the first prediction information using a first neural network based on the first feature information, the second feature information, and the first position.

[0132] Specifically, in one implementation, the vehicle may fuse the first feature information and the second feature information to obtain fused feature information; and input the fused feature information and the first position into the first neural network to obtain first prediction information generated by the first neural network. In another implementation, the vehicle may directly input the first feature information, the second feature information, and the first position into the first neural network to obtain the first prediction information generated by the first neural network.

[0133] Since the accuracy of the vehicle's positioning system (such as the vehicle's position obtained by GNSS) alone is low in scenarios such as urban canyons, tunnels, or under overpasses with high-rise buildings, in an embodiment of the present application, after the vehicle's first position is obtained based on the vehicle's positioning system, map information within a preset range of the first position can be obtained, and feature extraction is performed on the map information within the preset range of the first position to obtain second feature information. Feature extraction is also performed on the image of the vehicle's surroundings to obtain first feature information, and then prediction information is generated through a neural network in combination with the first feature information and the second feature information. The prediction information is used to determine the actual position of the vehicle, that is, more visual information is integrated on the basis of the first position to finally determine the vehicle's position, which is conducive to obtaining a more accurate position of the vehicle. In addition, when the present solution is applied to scenarios such as urban canyons or under overpasses with high-rise buildings, there will be buildings in the vehicle's surroundings, and the image of the vehicle's surroundings will carry images of large buildings, and the standard map will also contain building information. Therefore, the map information within the preset range of the first position is derived from the standard map, which makes it easier to determine the vehicle's actual position.

[0134] 2. Training Phase

[0135] Specifically, refer to FIG10 , which is a flow chart of a neural network training method provided in an embodiment of the present application. The neural network training method provided in an embodiment of the present application may include:

[0136] 1001. Obtain a training sample and second expected information corresponding to the training sample, where the training sample at least includes an image of the vehicle's surroundings, and the second expected information includes a correct image of the vehicle's surroundings from a bird's-eye view. The second expected information also indicates a correct category of an object in the correct image from a bird's-eye view.

[0137] In an embodiment of the present application, before training the second neural network, the training device needs to first obtain at least one training sample from the training data set, and each training sample includes at least one set of images, and the aforementioned set of images includes at least one image of the vehicle's surroundings; optionally, each training sample may also include at least one point cloud data of the vehicle's surroundings. Which images are specifically included in the "at least one image of the vehicle's surroundings" and which point cloud data are specifically included in the "at least one point cloud data of the vehicle's surroundings" can be understood in conjunction with the description in the embodiment corresponding to Figure 3 above, and will not be elaborated here.

[0138] The second expected information corresponding to the training sample can also be referred to as the second label corresponding to the training sample, or can also be referred to as the second true value corresponding to the training sample. Optionally, the second expected information corresponding to the training sample is obtained based on a standard precision map and a high-precision map corresponding to the vehicle's surroundings. Exemplarily, the second expected information corresponding to the training sample includes the presence of roads and buildings in the correct image of the vehicle's surroundings from a bird's-eye view; wherein, the image of the road portion in the correct image of the vehicle's surroundings from a bird's-eye view can be obtained based on the high-precision map, and the image of the building portion in the correct image of the vehicle's surroundings from a bird's-eye view can be obtained based on the standard precision map; the correct category of objects on the road in the vehicle's surroundings can be obtained based on the high-precision map corresponding to the vehicle's surroundings, and the correct category of buildings in the vehicle's surroundings can be obtained based on the standard precision map corresponding to the vehicle's surroundings.

[0139] In an embodiment of the present application, since the high-precision map carries detailed road information and the standard-precision map carries building information in the environment, the second expected information includes a correct image of the vehicle's surroundings from a bird's-eye view, and the second expected information also indicates the correct category of the object in the correct image from a bird's-eye view, then the high-precision map and the standard-precision map are used simultaneously to generate the second expected information, which is beneficial for the second expected information to include both high-quality road information and building information in the environment, that is, the second expected information can more comprehensively and accurately reflect the vehicle's surroundings. Using the second expected information as a guide to train the second neural network is beneficial for the trained second neural network to have a better understanding of the vehicle's surroundings, that is, it is beneficial for the first feature information extracted by the trained second neural network to cover richer information, and thus it is beneficial for the first neural network to generate more accurate prediction information with the help of the first feature information and the second feature information.

[0140] Since subsequent steps 1003 and 1004 are optional, if steps 1003 and 1004 are performed, each training sample may further include a first location, and the training device may then obtain map information within a first preset range of the first location based on the first location. Alternatively, each training sample may further include map information within the first preset range of the first location. The meanings of "first location" and "map information within the first preset range of the first location" can be found in the description of the embodiment corresponding to FIG. 3 above and are not further elaborated here.

[0141] Then in step 1001, the training device also needs to obtain the first expected information corresponding to the training sample, and the first expected information is used to determine the correct actual position of the vehicle; wherein, the first expected information corresponding to the training sample can also be called the first label corresponding to the training sample, or, it can also be called the first true value corresponding to the training sample.

[0142] For example, in one case, the first expected information indicates the offset between the correct actual position of the vehicle and the first position; in another case, the first expected information indicates the correct actual position of the vehicle. The meaning of "first expected information" is similar to that of "first predicted information", the difference being that "predicted" in the content describing the first predicted information is replaced with "correct", that is, the first predicted information is obtained through prediction by a neural network, while the first expected information is correct information.

[0143] Optionally, the correct actual position of the vehicle can be the position of the image of the vehicle's surrounding environment included in the training sample, that is, the training device can use the acquisition position of the image of the vehicle's surrounding environment as the correct actual position of the vehicle; the training device can also randomly generate first expected information, that is, randomly generate an offset between the correct actual position of the vehicle and the first position, and then determine the first position based on the correct actual position of the vehicle and the randomly generated first expected information, so as to obtain map information within a first preset range of the first position.

[0144] 1002. Input an image of the vehicle's surroundings into a second neural network, and perform feature extraction on the image of the vehicle's surroundings through the second neural network to obtain first feature information.

[0145] In the embodiment of the present application, the specific implementation method when the training device executes step 1002 and the specific meaning of the nouns in step 1002 can be found in the description of step 303 in the embodiment corresponding to Figure 3 above, and will not be repeated here.

[0146] 1003. Input the first feature information into a third neural network to obtain second prediction information generated by the third neural network, where the second prediction information includes a predicted image of the vehicle's surroundings from a bird's-eye view, and the second prediction information further indicates a predicted category of an object in the predicted image from a bird's-eye view.

[0147] In an embodiment of the present application, after the training device generates the first feature information through the second neural network, it can also input the first feature information into the third neural network to obtain the second prediction information generated by the third neural network.

[0148] Exemplarily, the third neural network can also be understood as a decoder. For example, the third neural network can be specifically expressed as MLP, convolutional neural network + MLP or other network structures, etc. The specific structure can be determined in combination with the actual application environment and is not limited in the embodiments of this application.

[0149] The second prediction information can be understood as the semantic information of objects in the vehicle's surrounding environment from a bird's-eye view. For example, the second prediction information includes a predicted image of the vehicle's surrounding environment from a bird's-eye view, and the second prediction information also indicates the predicted category of the object in the predicted image from a bird's-eye view. Exemplarily, the second prediction information can be expressed as a predicted image carrying annotation information, and the annotation information is used to indicate the predicted category of the object in the predicted image from a bird's-eye view.

[0150] 1004. Perform feature extraction on map information within a first preset range of the first position to obtain second feature information, where the first position represents a position of the vehicle obtained based on a vehicle positioning system, and the map information within the first preset range of the first position is derived from a precision map.

[0151] 1005. Based on the first feature information and the second feature information, generate first prediction information through a first neural network, and the first prediction information is used to determine the predicted actual position of the vehicle.

[0152] In the embodiment of the present application, steps 1004 and 1005 are optional steps. If steps 1004 and 1005 are performed, illustratively, step 1004 may include: the training device extracts features of map information within a first preset range of the first position through a fourth neural network to obtain second feature information; it should be noted that the specific implementation method of the training device performing steps 1004 and 1005 and the specific meanings of the nouns in steps 1004 and 1005 can be found in the description of steps 304 and 305 in the embodiment corresponding to Figure 3 above, and will not be repeated here.

[0153] 1006. Use the first loss function to train the second neural network to obtain a trained second neural network, where the first loss function indicates the similarity between the second predicted information and the second expected information.

[0154] In an embodiment of the present application, steps 1004 and 1005 are optional steps. If steps 1004 and 1005 are not performed, in step 1006, the training device may use the first loss function to train the second neural network (optionally, also including the third neural network) until the first convergence condition is met, thereby obtaining the trained second neural network (optionally, also including the trained third neural network). The first loss function indicates the similarity between the second prediction information and the second expected information, and the goal of training using the second loss function includes improving the similarity between the second prediction information generated by the second neural network and the second expected information. The first convergence condition may include satisfying the convergence condition of the first loss function, and / or the number of iterative training reaches a preset number.

[0155] Exemplarily, during each training process of the second neural network (optionally, also including the third neural network), the training device can generate a function value of the first loss function based on the second prediction information and the second expected information, and then use the back propagation algorithm based on the function value of the first loss function to update the weight parameters of the second neural network (optionally, also including the third neural network) to achieve one training of the second neural network (optionally, also including the third neural network).

[0156] For example, during the training of the second neural network, the predicted value obtained by the second neural network can be compared with the desired expected value, and the weight vector of each layer of the neural network can be updated according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the second neural network). For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value, and the adjustment is continued until the second neural network can generate the desired expected value or a value very close to the desired expected value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the expected value", which is the loss function or objective function. They are important equations used to measure the difference between the predicted value and the expected value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and the training of the second neural network becomes a process of minimizing this loss as much as possible.

[0157] Neural networks can use the backpropagation algorithm to correct the initial neural network parameters during training, reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the initial neural network parameters, thereby converging the error loss. The backpropagation algorithm is an error-driven backward propagation process aimed at obtaining the optimal neural network parameters.

[0158] For a more intuitive understanding of this solution, please refer to Figure 11. Figure 11 is a schematic diagram of a neural network training method provided in an embodiment of the present application. As shown in Figure 11, after obtaining a training sample, the training device inputs the image of the vehicle's surrounding environment included in the training sample into a second neural network. The second neural network performs feature extraction on the image of the vehicle's surrounding environment to obtain first feature information generated by the second neural network; the first feature information is input into a third neural network to obtain second prediction information generated by the third neural network. The second prediction information includes a predicted image of the vehicle's surrounding environment from a bird's-eye view, and the second prediction information also indicates the predicted category of objects in the predicted image from a bird's-eye view. The training device generates a function value of a first loss function based on the second prediction information and the second expected information, and trains the second and third neural networks based on the function value of the first loss function. It should be understood that the example in Figure 11 is only for the convenience of understanding this solution and is not intended to limit this solution.

[0159] In an embodiment of the present application, during the training phase of the second neural network, the feature information generated by the second neural network is also input into the third neural network, and second prediction information is generated by the third neural network. The second prediction information indicates a predicted image of the vehicle's surroundings from a bird's-eye view and a predicted category of an object in the image. The second neural network is then trained using a first loss function, where the first loss function indicates the similarity between the second prediction information and the second expected information. The second expected information includes a correct image of the vehicle's surroundings and a correct category of an object in the image. That is, semantic segmentation of the vehicle's surroundings is performed based on the feature information of the vehicle's surroundings from a bird's-eye view generated by the second neural network, and the correct semantic segmentation result is used as supervision to train the second neural network. Since the better the second feature information collected by the second neural network, the simpler the semantic segmentation is for the third neural network, and the easier it is to generate a better quality semantic segmentation result, the higher the similarity between the second prediction information and the second expected information. Training in this manner is conducive to improving the second neural network's ability to better understand the vehicle's surroundings, that is, it is conducive to ensuring that the first feature information extracted by the trained second neural network contains richer information.

[0160] Optionally, if steps 1004 and 1005 are performed, step 1006 includes: the training device uses the first loss function and the second loss function to train the first neural network and the second neural network (optionally, also including the third neural network and the fourth neural network) until the second convergence condition is met, and the trained first neural network and the trained second neural network are obtained (optionally, also including the trained third neural network and the trained fourth neural network). The second loss function indicates the similarity between the first prediction information and the first expected information, and the first expected information is used to determine the correct actual position of the vehicle. The second convergence condition may include satisfying the convergence condition of the first loss function and the second loss function, and / or the number of iterative training reaches a preset number.

[0161] Exemplarily, during each training process of the first neural network and the second neural network (optionally, also including the third neural network and the fourth neural network), the training device can generate a function value of the first loss function based on the second prediction information and the second expected information, and generate a function value of the second loss function based on the first prediction information and the first expected information, and then based on the function value of the first loss function and the function value of the second loss function, use the back propagation algorithm to update the weight parameters of the first neural network and the second neural network (optionally, also including the third neural network and the fourth neural network) to achieve one training of the first neural network and the second neural network (optionally, also including the third neural network and the fourth neural network).

[0162] Optionally, if steps 1004 and 1005 are executed, the training device can also obtain sub-feature information from the second feature information, the second feature information includes feature information of map information within a first preset range of the first position, the map information within the first preset range of the first position includes map information within a second preset range of the actual position of the vehicle, the sub-feature information includes feature information of map information within the second preset range of the actual position, the meaning of "second preset range" is similar to that of "first preset range", the difference is that "second preset range" is smaller than "first preset range", for example, the first preset range is a square area with a side length of 256 meters centered on the first position, and the second preset range is a square area with a side length of 128 meters centered on the actual position.

[0163] In step 1006, the training device uses the first loss function, the second loss function, and the third loss function to train the first neural network and the second neural network (optionally, also including the third neural network and the fourth neural network) until the third convergence condition is met, thereby obtaining the trained first neural network and the trained second neural network (optionally, also including the trained third neural network and the trained fourth neural network). The third loss function indicates the similarity between the first feature information and the sub-feature information, and the purpose of training using the third loss function includes improving the similarity between the first feature information and the sub-feature information. The third convergence condition may include satisfying the convergence conditions of the first loss function, the second loss function, and the third loss function, and / or the number of iterative training reaches a preset number.

[0164] In the embodiment of the present application, since the image of the vehicle's surroundings can reflect the vehicle's actual position, theoretically, the similarity between the image of the vehicle's surroundings and the map information within a preset range of the vehicle's actual position is relatively high, that is, the similarity between the feature information of the image of the vehicle's surroundings and the feature information of the map information within a preset range of the vehicle's actual position should also be relatively high. The training device additionally introduces a third loss function in the training stage of the second neural network. The third loss function represents the similarity between the first feature information (that is, the feature information of the image of the vehicle's surroundings) and the sub-feature information (that is, the feature information within the second preset range of the vehicle's actual position), that is, supervising the training of the second neural network in more dimensions, which is conducive to enabling the trained second neural network to learn better feature extraction capabilities, that is, it is conducive to enabling the second neural network to extract more accurate feature information from the image of the vehicle's surroundings, and thus is conducive to more accurate positioning of the vehicle.

[0165] On the basis of the embodiments corresponding to Figures 1 to 11, in order to better implement the above-mentioned scheme of the embodiment of the present application, the following also provides related equipment for implementing the above-mentioned scheme. Please refer to Figure 12 in detail. Figure 12 is a structural schematic diagram of the vehicle positioning device provided in the embodiment of the present application. The vehicle positioning device 1200 includes: an acquisition module 1201, which is used to acquire an image of the vehicle's surroundings and map information within a preset range of a first position, where the first position is the position of the vehicle obtained based on the vehicle's positioning system; a feature extraction module 1202, which is used to extract features from the image of the vehicle's surroundings to obtain first feature information; the feature extraction module 1202 is also used to extract features from the map information within the preset range of the first position to obtain second feature information, where the map information within the preset range of the first position is derived from a precise map; a generation module 1203, which is used to generate first prediction information through a first neural network based on the first feature information and the second feature information, and the first prediction information is used to determine the actual position of the vehicle.

[0166] Optionally, the first neural network includes a Transformer neural network module and a multi-layer perceptron MLP, and a generation module 1203 is specifically used to: fuse the first feature information and the second feature information to obtain fused feature information; input the fused feature information into the first neural network to obtain first prediction information generated by the first neural network, and the first prediction information indicates the offset between the actual position of the vehicle and the first position.

[0167] Optionally, the image of the vehicle's surroundings includes at least two images of the vehicle's surroundings, and the at least two images include images captured by the vehicle in at least two of the following directions: left front, front, right front, left rear, rear, or right rear.

[0168] Optionally, the first feature information includes feature information of the vehicle's surrounding environment from a bird's-eye view. The feature extraction module 1202 is specifically used to extract features from the image of the vehicle's surrounding environment through a second neural network. In the process of training the second neural network using the first loss function, the feature information generated by the second neural network is also input into a third neural network. The third neural network is used to generate second prediction information. The first loss function indicates the similarity between the second prediction information and the second expected information. The second prediction information includes a predicted image of the vehicle's surrounding environment from a bird's-eye view, and the second prediction information also indicates the predicted category of objects in the predicted image from a bird's-eye view. The second expected information includes a correct image of the vehicle's surrounding environment from a bird's-eye view, and the second expected information also indicates the correct category of objects in the correct image from a bird's-eye view.

[0169] Optionally, the vehicle positioning device 1200 also includes: a rasterization module 1204, which is used to rasterize the map information within the preset range of the first position in the first format to obtain the map information within the preset range of the first position in the second format, the first format is a text format or a binary format, and the second format is an image format; a feature extraction module, which is specifically used to extract features from the map information within the preset range of the first position in the second format.

[0170] It should be noted that the information interaction, execution process, etc. between the modules / units in the vehicle positioning device 1200 are based on the same concept as the various method embodiments corresponding to Figures 1 to 11 in this application. For specific contents, please refer to the description in the method embodiments shown above in this application, and will not be repeated here.

[0171] Referring to Figure 13, Figure 13 is a structural schematic diagram of a neural network training device provided in an embodiment of the present application. The neural network training device 1300 includes: an input module 1301, which is used to input an image of the vehicle's surrounding environment into a second neural network, and perform feature extraction on the image of the vehicle's surrounding environment through the second neural network to obtain first feature information; a generation module 1302, which is used to input the first feature information into a third neural network to obtain second prediction information generated by the third neural network, the second prediction information including a predicted image of the vehicle's surrounding environment from a bird's-eye view, and the second prediction information also indicates a predicted category of an object in the predicted image from a bird's-eye view; a training module 1303, which is used to train the second neural network using a first loss function to obtain a trained second neural network, the first loss function indicating the similarity between the second prediction information and second expected information, the second expected information including a correct image of the vehicle's surrounding environment from a bird's-eye view, and the second expected information also indicating a correct category of an object in the correct image from a bird's-eye view.

[0172] Optionally, the second expected information is obtained based on a standard precision map and a high precision map corresponding to the vehicle's surrounding environment.

[0173] Optionally, the neural network training device 1300 also includes: a feature extraction module 1304, which is used to extract features from map information within a preset range of a first position to obtain second feature information, the first position represents the position of the vehicle obtained based on the vehicle positioning system, and the map information within the preset range of the first position comes from the standard map; a generation module 1302, which is also used to generate first prediction information through a first neural network based on the first feature information and the second feature information, and the first prediction information is used to determine the predicted actual position of the vehicle; a training module 1304, which is specifically used to use a first loss function and a second loss function to train the first neural network and the second neural network to obtain a trained first neural network and a trained second neural network, the second loss function indicates the similarity between the first prediction information and the first expected information, and the first expected information is used to determine the correct actual position of the vehicle.

[0174] It should be noted that the information interaction, execution process, etc. between the modules / units in the neural network training device 1300 are based on the same concept as the various method embodiments corresponding to Figures 1 to 11 in this application. For specific contents, please refer to the description in the method embodiments shown above in this application, and will not be repeated here.

[0175] Next, a device provided in an embodiment of the present application is introduced. When the device is specifically an execution device, please refer to Figure 14, which is a schematic structural diagram of a device provided in an embodiment of the present application. Specifically, the device 1400 includes: a receiver 1401, a transmitter 1402, a processor 1403, and a memory 1404 (wherein the number of processors 1403 in the device 1400 can be one or more, and Figure 14 uses one processor as an example). The processor 1403 may include an application processor 14031 and a communication processor 14032. In some embodiments of the present application, the receiver 1401, the transmitter 1402, the processor 1403, and the memory 1404 may be connected via a bus or other means.

[0176] Memory 1404 may include read-only memory and random access memory, and provides instructions and data to processor 1403. A portion of memory 1404 may also include non-volatile random access memory (NVRAM). Memory 1404 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.

[0177] Processor 1403 controls the operation of the device. In specific applications, the various components of the device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.

[0178] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1403. Processor 1403 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in processor 1403. The above processor 1403 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1403 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1404, and processor 1403 reads the information in memory 1404 and, in conjunction with its hardware, completes the steps of the above method.

[0179] Receiver 1401 can be used to receive input digital or character information and generate signal input related to device settings and function control. Transmitter 1402 can be used to output digital or character information through the first interface. Transmitter 1402 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1402 can also include a display device such as a display screen.

[0180] In the embodiments of the present application, processor 1403 is configured to execute the vehicle-executed method described in the embodiments corresponding to Figures 1 to 11. It should be noted that the specific manner in which application processor 14031 in processor 1403 executes the aforementioned steps is based on the same concept as the various method embodiments corresponding to Figures 1 to 11 of the present application, and the technical effects achieved are the same as those of the various method embodiments corresponding to Figures 1 to 11 of the present application. For details, please refer to the description of the method embodiments described above in the present application, and will not be repeated here.

[0181] In the case where the device is specifically manifested as a second device, please refer to Figure 15, which is another structural diagram of the device provided in an embodiment of the present application. Specifically, the device 1500 is implemented by one or more servers. The device 1500 may have relatively large differences due to different configurations or performances. It may include one or more central processing units (CPUs) 1522 (for example, one or more processors) and a memory 1532, and one or more storage media 1530 (for example, one or more mass storage devices) storing application programs 1542 or data 1544. Among them, the memory 1532 and the storage medium 1530 can be temporary storage or permanent storage. The program stored in the storage medium 1530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the device. Furthermore, the central processing unit 1522 can be configured to communicate with the storage medium 1530 to execute a series of instruction operations in the storage medium 1530 on the device 1500.

[0182] The device 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input and output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0183] In the embodiment of the present application, the central processing unit 1522 is used to execute the method performed by the training device in the embodiments corresponding to Figures 1 to 11. It should be noted that the specific manner in which the central processing unit 1522 executes the above-mentioned steps is based on the same concept as the various method embodiments corresponding to Figures 1 to 11 in the present application, and the technical effects brought about are the same as the various method embodiments corresponding to Figures 1 to 11 in the present application. For specific details, please refer to the description of the method embodiments shown above in the present application, and will not be repeated here.

[0184] The present application also provides a vehicle. Please refer to FIG16 , which is a schematic structural diagram of a vehicle provided by the present application. Vehicle 100 is configured for a fully or partially autonomous driving mode. For example, vehicle 100 can control itself while in autonomous driving mode, and can determine the current state of the vehicle and its surrounding environment through human operation, determine the possible behavior of at least one other vehicle in the surrounding environment, and determine the confidence level corresponding to the possibility of the other vehicle performing the possible behavior, and control vehicle 100 based on the determined information. When vehicle 100 is in autonomous driving mode, vehicle 100 can also be set to operate without human interaction.

[0185] The vehicle 100 may include various subsystems, such as a travel system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power source 110, a computer system 112, and a user interface 116. Alternatively, the vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and component of the vehicle 100 may be interconnected via wired or wireless connections.

[0186] Travel system 102 may include components that provide powered movement for vehicle 100. In one embodiment, travel system 102 may include engine 118, power source 119, transmission 120, and wheels / tires 121.

[0187] The engine 118 may be an internal combustion engine, an electric motor, an air compression engine, or a combination of other types of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compression engine. The engine 118 converts the energy source 119 into mechanical energy. Examples of the energy source 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. The energy source 119 may also provide energy for other systems of the vehicle 100. The transmission 120 may transmit the mechanical power from the engine 118 to the wheels 121. The transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 120 may also include other devices, such as a clutch. The drive shaft may include one or more shafts that can be coupled to one or more wheels 121.

[0188] Sensor system 104 may include several sensors that sense information about the environment surrounding vehicle 100. For example, sensor system 104 may include a positioning system 122 (the positioning system may be a global positioning system (GPS), a BeiDou system, or other positioning systems), an inertial measurement unit (IMU) 124, a radar 126, a laser rangefinder 128, and a camera 130. Sensor system 104 may also include sensors for internal systems of monitored vehicle 100 (e.g., an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors may be used to detect objects and their corresponding characteristics (position, shape, direction, speed, etc.). This detection and recognition is a key function for the safe operation of autonomous vehicle 100.

[0189] Among them, the positioning system 122 can be used to estimate the geographic location of the vehicle 100. The IMU 124 is used to sense the position and orientation changes of the vehicle 100 based on inertial acceleration. In one embodiment, the IMU 124 can be a combination of an accelerometer and a gyroscope. The radar 126 can use radio signals to sense objects in the surrounding environment of the vehicle 100, and can specifically be a millimeter wave radar or a lidar. In some embodiments, in addition to sensing objects, the radar 126 can also be used to sense the speed and / or direction of travel of objects. The laser rangefinder 128 can use lasers to sense objects in the environment in which the vehicle 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. The camera 130 can be used to capture multiple images of the surrounding environment of the vehicle 100. The camera 130 can be a still camera or a video camera.

[0190] Control system 106 controls the operation of vehicle 100 and its components. Control system 106 may include various components, including a steering system 132 , a throttle 134 , a brake unit 136 , a computer vision system 140 , a lane control system 142 , and an obstacle avoidance system 144 .

[0191] The steering system 132 is operable to adjust the direction of travel of the vehicle 100. For example, in one embodiment, it may be a steering wheel system. The throttle 134 is used to control the operating speed of the engine 118 and, in turn, the speed of the vehicle 100. The brake unit 136 is used to control the deceleration of the vehicle 100. The brake unit 136 may use friction to slow the wheels 121. In other embodiments, the brake unit 136 may convert the kinetic energy of the wheels 121 into electrical current. The brake unit 136 may also take other forms to slow the rotation speed of the wheels 121 to control the speed of the vehicle 100. The computer vision system 140 is operable to process and analyze images captured by the camera 130 to identify objects and / or features in the environment surrounding the vehicle 100. These objects and / or features may include traffic signs, road boundaries, and obstacles. The computer vision system 140 may use object recognition algorithms, structure from motion (SFM) algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system 140 can be used to map the environment, track objects, estimate their speed, and so on. The route control system 142 is used to determine the route and speed of the vehicle 100. In some embodiments, the route control system 142 may include a lateral planning module 1421 and a longitudinal planning module 1422, which are respectively used to determine the route and speed for the vehicle 100 by combining data from the obstacle avoidance system 144, GPS 122, and one or more predetermined maps. The obstacle avoidance system 144 is used to identify, evaluate, and avoid or otherwise navigate obstacles in the environment of the vehicle 100. The aforementioned obstacles can specifically be represented by actual obstacles and virtual moving objects that may collide with the vehicle 100. In one embodiment, the control system 106 may include additional or alternative components other than those shown and described. Alternatively, some of the components shown above may be reduced.

[0192] Vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users via peripheral devices 108. Peripheral devices 108 may include a wireless communication system 146, an onboard computer 148, a microphone 150, and / or a speaker 152. In some embodiments, peripheral devices 108 provide a means for the user of vehicle 100 to interact with user interface 116. For example, onboard computer 148 may provide information to the user of vehicle 100. User interface 116 may also operate onboard computer 148 to receive user input. Onboard computer 148 may be operated via a touchscreen. In other cases, peripheral devices 108 may provide a means for vehicle 100 to communicate with other devices located within the vehicle. For example, microphone 150 may receive audio (e.g., voice commands or other audio input) from the user of vehicle 100. Similarly, speaker 152 may output audio to the user of vehicle 100. Wireless communication system 146 may wirelessly communicate with one or more devices directly or via a communication network. For example, the wireless communication system 146 may utilize 3G cellular communications, such as CDMA, EVDO, GSM / GPRS, or 4G cellular communications, such as LTE. Or 5G cellular communications. The wireless communication system 146 may utilize wireless local area network (WLAN) communications. In some embodiments, the wireless communication system 146 may utilize infrared links, Bluetooth, or ZigBee to communicate directly with devices. Other wireless protocols, such as various vehicle communication systems, may include one or more dedicated short range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0193] Power source 110 can provide power to various components of vehicle 100. In one embodiment, power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more battery packs of such batteries can be configured as a power source to provide power to various components of vehicle 100. In some embodiments, power source 110 and energy source 119 can be implemented together, such as in some all-electric vehicles.

[0194] Some or all functions of vehicle 100 are controlled by computer system 112. Computer system 112 may include at least one processor 113 that executes instructions 115 stored in a non-transitory computer-readable medium, such as memory 114. Computer system 112 may also be a plurality of computing devices that control individual components or subsystems of vehicle 100 in a distributed manner. Processor 113 may be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, processor 113 may be a specialized device, such as an application-specific integrated circuit (ASIC) or other hardware-based processor. Although FIG. 16 functionally illustrates the processor, memory, and other components of computer system 112 in the same block, those skilled in the art will appreciate that the processor or memory may actually include multiple processors or memories that are not stored in the same physical housing. For example, memory 114 may be a hard drive or other storage medium located in a different housing than computer system 112. Therefore, references to processor 113 or memory 114 should be understood to include references to a collection of processors or memories that may or may not operate in parallel. Rather than using a single processor to perform the steps described herein, some components, such as the steering assembly and the retarding assembly, may each have its own processor that performs only calculations related to the functionality of the component specific component.

[0195] In various aspects described herein, the processor 113 may be located remotely from the vehicle 100 and in wireless communication with the vehicle 100. In other aspects, some of the processes described herein are performed on the processor 113 disposed within the vehicle 100 while others are performed by the remote processor 113, including taking the necessary steps to perform a single maneuver.

[0196] In some embodiments, memory 114 may contain instructions 115 (e.g., program logic) that are executable by processor 113 to perform various functions of vehicle 100, including those described above. Memory 114 may also contain additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of travel system 102, sensor system 104, control system 106, and peripherals 108. In addition to instructions 115, memory 114 may also store data such as road maps, route information, the vehicle's location, direction, speed, and other such vehicle data, as well as other information. This information may be used by vehicle 100 and computer system 112 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes. A user interface 116 is provided for providing information to or receiving information from a user of vehicle 100. Optionally, user interface 116 may include one or more input / output devices within the set of peripherals 108, such as wireless communication system 146, onboard computer 148, microphone 150, and speaker 152.

[0197] Computer system 112 may control functions of vehicle 100 based on input received from various subsystems (e.g., travel system 102, sensor system 104, and control system 106) and from user interface 116. For example, computer system 112 may utilize input from control system 106 to control steering system 132 to avoid obstacles detected by sensor system 104 and obstacle avoidance system 144. In some embodiments, computer system 112 may be operable to provide control over many aspects of vehicle 100 and its subsystems.

[0198] Alternatively, one or more of the above components may be installed or associated separately from the vehicle 100. For example, the memory 114 may be partially or completely separate from the vehicle 100. The above components may be communicatively coupled together in a wired and / or wireless manner.

[0199] Optionally, the above components are just an example. In actual applications, the components in the above modules may be added or deleted according to actual needs. Figure 16 should not be understood as limiting the embodiments of the present application. A vehicle traveling on a road, such as vehicle 100 above, can identify objects in its surrounding environment to determine adjustments to the current speed. The objects can be other vehicles, traffic control devices, or other types of objects. In some examples, each identified object can be considered independently, and based on the respective characteristics of the object, such as its current speed, acceleration, distance from the vehicle, etc., it can be used to determine the speed to be adjusted for the vehicle.

[0200] Optionally, the vehicle 100 or a computing device associated with the vehicle 100, such as the computer system 112, computer vision system 140, and memory 114 of Figure 16, can predict the behavior of the identified objects based on the characteristics of the identified objects and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of each other, so all identified objects can be considered together to predict the behavior of a single identified object. The vehicle 100 can adjust its speed based on the predicted behavior of the identified objects. In other words, the vehicle 100 can determine what stable state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the objects. In this process, other factors can also be considered to determine the speed of the vehicle 100, such as the lateral position of the vehicle 100 on the road it is traveling on, the curvature of the road, the proximity of static and dynamic objects, etc. In addition to providing instructions to adjust the speed of the vehicle, the computing device may also provide instructions to modify the steering angle of the vehicle 100 so that the vehicle 100 follows a given trajectory and / or maintains a safe lateral and longitudinal distance from objects near the vehicle 100 (e.g., cars in adjacent lanes on the road).

[0201] In the embodiment of the present application, the processor 113 in the vehicle 100 is used to execute the method executed by the vehicle in the embodiments corresponding to Figures 1 to 11. It should be noted that the specific manner in which the processor 113 executes the aforementioned steps is based on the same concept as the various method embodiments corresponding to Figures 1 to 11 in this application, and the technical effects it brings are the same as the various method embodiments corresponding to Figures 1 to 11 in this application. For specific details, please refer to the description of the method embodiments shown above in this application, and will not be repeated here.

[0202] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a program, which, when executed on a computer, enables the computer to execute the steps executed by the vehicle in the method described in the embodiments shown in Figures 1 to 11 above, or enables the computer to execute the steps executed by the training device in the method described in the embodiments shown in Figures 1 to 11 above.

[0203] Also provided in an embodiment of the present application is a computer program product, which includes a program. When the computer program product is run on a computer, it enables the computer to execute the steps executed by the vehicle in the method described in the embodiments shown in Figures 1 to 11 above, or enables the computer to execute the steps executed by the training device in the method described in the embodiments shown in Figures 1 to 11 above.

[0204] An embodiment of the present application further provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the embodiments shown in Figures 1 to 11 above.

[0205] The execution device, training device, or vehicle positioning device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin, or a circuit. The processing unit may execute computer-executable instructions stored in the storage unit so that the chip executes the method described in the embodiments shown in Figures 1 to 11 above. Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0206] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first aspect method.

[0207] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0208] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general-purpose hardware, and of course can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CLUs, dedicated memories, dedicated components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, disk or optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0209] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0210] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, a computer, a server, or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium can be a magnetic medium, (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive (SSD)).

Claims

1. A vehicle positioning method, characterized in that: The method comprises: Acquire an image of the vehicle's surroundings and map information within a preset range of a first location, where the first location is a location of the vehicle obtained based on a vehicle positioning system; Extracting features from the image of the vehicle's surrounding environment to obtain first feature information; performing feature extraction on map information within a preset range of the first location to obtain second feature information, wherein the map information within the preset range of the first location is derived from a standard map; Based on the first feature information and the second feature information, first prediction information is generated through a first neural network, and the first prediction information is used to determine the actual position of the vehicle.

2. The method according to claim 1, characterized in that The first neural network includes a Transformer neural network module and a multi-layer perceptron MLP. The first prediction information is generated by the first neural network based on the first feature information and the second feature information, including: fusing the first feature information and the second feature information to obtain fused feature information; The fused feature information is input into the first neural network to obtain the first prediction information generated by the first neural network, where the first prediction information indicates an offset between the actual position of the vehicle and the first position.

3. The method according to claim 1 or 2, characterized in that The image of the vehicle surroundings includes at least two images of the vehicle surroundings, and the at least two images include images captured by the vehicle in at least two of the following directions: left front, front, right front, left rear, rear, or right rear.

4. The method according to claim 1 or 2, characterized in that The first feature information includes feature information of the vehicle surrounding environment from a bird's-eye view, and the feature extraction of the image of the vehicle surrounding environment includes: extracting features from the image of the vehicle's surroundings using a second neural network, wherein during training the second neural network using a first loss function, feature information generated by the second neural network is also input into a third neural network, the third neural network being used to generate second prediction information, and the first loss function indicating a similarity between the second prediction information and second expected information; The second prediction information includes a predicted image of the vehicle's surroundings from a bird's-eye view, and the second prediction information further indicates a predicted category of an object in the predicted image from a bird's-eye view; The second expected information includes a correct image of the vehicle's surroundings from a bird's-eye view, and the second expected information further indicates a correct category of an object in the correct image from a bird's-eye view.

5. The method according to claim 1 or 2, characterized in that The method further comprises: rasterizing the map information within a preset range of the first position in a first format to obtain the map information within the preset range of the first position in a second format, wherein the first format is a text format or a binary format and the second format is an image format; The extracting features from the map information within the preset range of the first position includes extracting features from the map information within the preset range of the first position in the second format.

6. A method for training a neural network, characterized in that: The method comprises: Inputting an image of the vehicle's surroundings into a second neural network, and performing feature extraction on the image of the vehicle's surroundings through the second neural network to obtain first feature information; inputting the first feature information into a third neural network to obtain second prediction information generated by the third neural network, wherein the second prediction information includes a predicted image of the vehicle's surroundings from a bird's-eye view, and the second prediction information further indicates a predicted category of an object in the predicted image from a bird's-eye view; The second neural network is trained using a first loss function to obtain a trained second neural network, wherein the first loss function indicates a similarity between the second predicted information and second expected information, wherein the second expected information includes a correct image of the vehicle's surroundings from a bird's-eye view, and the second expected information further indicates a correct category of an object in the correct image from a bird's-eye view.

7. The method according to claim 6, characterized in that The second expected information is obtained based on a standard precision map and a high precision map corresponding to the vehicle surrounding environment.

8. The method according to claim 6 or 7, characterized in that The method further comprises: performing feature extraction on map information within a preset range of a first position to obtain second feature information, wherein the first position represents a position of the vehicle obtained based on a vehicle positioning system, and the map information within the preset range of the first position is derived from a standard map; generating first prediction information through a first neural network based on the first feature information and the second feature information, wherein the first prediction information is used to determine a predicted actual position of the vehicle; The step of training the second neural network using the first loss function to obtain a trained second neural network includes: The first neural network and the second neural network are trained using the first loss function and the second loss function to obtain the trained first neural network and the trained second neural network, wherein the second loss function indicates the similarity between the first predicted information and the first expected information, and the first expected information is used to determine the correct actual position of the vehicle.

9. A vehicle positioning device, characterized in that: The device comprises: an acquisition module, configured to acquire an image of the vehicle's surroundings and map information within a preset range of a first position, where the first position is a position of the vehicle obtained based on a vehicle positioning system; a feature extraction module, configured to extract features from an image of the vehicle's surrounding environment to obtain first feature information; The feature extraction module is further configured to extract features from map information within a preset range of the first position to obtain second feature information, wherein the map information within the preset range of the first position is derived from a standard map; A generation module is used to generate first prediction information through a first neural network based on the first feature information and the second feature information, where the first prediction information is used to determine the actual position of the vehicle.

10. A neural network training device, characterized in that: The device comprises: an input module, configured to input an image of the vehicle's surroundings into a second neural network, and extract features from the image of the vehicle's surroundings through the second neural network to obtain first feature information; The input module is further configured to input the first feature information into a third neural network to obtain second prediction information generated by the third neural network, wherein the second prediction information includes a predicted image of the vehicle's surroundings from a bird's-eye view, and the second prediction information further indicates a predicted category of an object in the predicted image from a bird's-eye view; A training module is used to train the second neural network using a first loss function to obtain a trained second neural network, wherein the first loss function indicates the similarity between the second predicted information and second expected information, the second expected information including a correct image of the vehicle surroundings from a bird's-eye view, and the second expected information further indicates a correct category of an object in the correct image from a bird's-eye view.

11. A device, characterized in that The method comprises a processor coupled to a memory, wherein the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method according to any one of claims 1 to 8 is implemented.

12. A vehicle, characterized in that: The method comprises a processor coupled to a memory, wherein the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method according to any one of claims 1 to 5 is implemented.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a program, and when the program is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 8.

14. A computer program product, characterized in that The computer program product comprises a program, which, when run on a computer, causes the computer to perform the method according to any one of claims 1 to 8 .

Images