Vehicle positioning method, device, equipment, medium and program product

Abstract

The invention discloses a vehicle positioning method and device, equipment, a medium and a program product. The method comprises the steps of obtaining a perception image in a scene where a vehicle is located and initial positioning data of the vehicle; determining a standard map in a scene where the vehicle is located according to the initial positioning data; inputting the perception image and a standard map into a preset model, and generating a local map and a first mapping relation between the local map and the standard map through the preset model; and determining a positioning result of the vehicle according to the local map and the first mapping relation. Thus, the local map is established in real time through the perception image of the vehicle and the low-cost standard map, and the initial positioning data is corrected based on the local map and the first mapping relation to obtain the positioning result of the vehicle, so that the requirement of intelligent driving for high-precision positioning can be met, and the positioning precision of the vehicle is improved.

Description

Technical Field

[0001] The present application relates to the field of positioning technology, and in particular to a vehicle positioning method, device, equipment, medium and program product. Background Art

[0002] With the development of connected cars, high-precision positioning technology has been applied to intelligent driving to improve vehicle safety, which can greatly improve the perception of vehicles during driving. However, when the vehicle is driving in a complex environment or the signal of the satellite positioning system is weak, the accuracy of vehicle positioning is low, which makes it difficult to meet the requirements of intelligent driving for high-precision positioning. Summary of the invention

[0003] The present application provides a vehicle positioning method, device, equipment, medium and program product, which can meet the requirements of intelligent driving for high-precision positioning, thereby improving the vehicle positioning accuracy.

[0004] In a first aspect, an embodiment of the present application provides a vehicle positioning method, the method comprising:

[0005] Obtaining a perception image of the scene where the vehicle is located and initial positioning data of the vehicle;

[0006] Determine a standard map of the scene where the vehicle is located based on the initial positioning data;

[0007] Inputting the perceived image and the standard map into a preset model, generating a local map and a first mapping relationship between the local map and the standard map through the preset model;

[0008] A positioning result of the vehicle is determined according to the local map and the first mapping relationship.

[0009] Through the above technical means, a standard map is obtained based on the initial positioning data, and a local map and a first mapping relationship are obtained by inputting the perceived image and the standard map into a preset model, and then the position of the vehicle in the local map is mapped to the standard map using the first mapping relationship, and finally the positioning result of the vehicle can be obtained. In this way, a local map is established in real time through the perceived image of the vehicle and a low-cost standard map, and the initial positioning data is corrected based on the local map and the first mapping relationship to obtain the positioning result of the vehicle, thereby meeting the requirements of intelligent driving for high-precision positioning, improving the positioning accuracy of the vehicle, and further improving the safety of vehicle driving; and the quality of the local map established in real time is improved by utilizing the fusion of multi-source information (perceived images and standard maps, etc.).

[0010] In some embodiments, based on the initial positioning data, determining the standard map in the scene where the vehicle is located includes: obtaining a preset standard map in the scene where the vehicle is located; taking the initial positioning data as the center, obtaining a local standard map within a preset range in the preset standard map, and determining the local standard map as the standard map in the scene where the vehicle is located.

[0011] Through the above-mentioned technical means, with the initial positioning data as the center, a standard map within a preset range is obtained in the preset standard map, which can more finely display the map information of the scene where the vehicle is located, thereby improving the positioning accuracy; and can reduce the amount of data loaded and improve operational efficiency.

[0012] In some embodiments, the preset model includes a feature sub-model and a generation sub-model; the perceived image and the standard map are input into the preset model, and a local map and a first mapping relationship between the local map and the standard map are generated through the preset model, including: the perceived image and the standard map are input into the feature sub-model, and a fused visual feature is generated through the feature sub-model; the fused visual feature is input into the generation sub-model, and a local map and a first mapping relationship are generated through the generation sub-model.

[0013] Through the above technical means, the feature sub-model in the preset model performs feature processing on the perceived image and the standard map to generate fused visual features, so as to generate features that are more suitable for the prediction of the preset model, and can remove irrelevant or redundant features, reduce the complexity of the preset model, and improve the generalization ability of the model; and the fused visual features are input into the generation sub-model in the preset model, and the local map and the first mapping relationship are generated through the generation sub-model. In this way, the accuracy of the preset model prediction is improved, thereby improving the vehicle positioning accuracy.

[0014] In some embodiments, the feature sub-model includes a first feature extraction module, a second feature extraction module and a feature fusion module; the perceived image and the standard map are input into the feature sub-model, and a fused visual feature is generated through the feature sub-model, including: performing feature extraction on the perceived image through the first feature extraction module to determine the first visual feature corresponding to the perceived image; and performing feature extraction on the standard map through the second feature extraction module to determine the second visual feature corresponding to the standard map; performing feature fusion on the first visual feature and the second visual feature through the feature fusion module to determine the fused visual feature of the vehicle.

[0015] Through the above-mentioned technical means, feature extraction is performed on the perceived image through the first feature extraction module, and feature extraction is performed on the standard map through the second feature extraction module. Targeted extraction can be performed according to different situations of the perceived image and the standard map, thereby improving the flexibility of feature extraction and the accuracy of scene features; and the first visual feature and the second visual feature are fused through the feature fusion module. By fusing different features, the generated sub-model model can understand the data more comprehensively and capture more complex patterns and information, thereby improving the accuracy of prediction.

[0016] In some embodiments, the positioning result of the vehicle is determined based on the local map and the first mapping relationship, including: determining the coordinate data corresponding to the vehicle in the standard map based on the local map and the first mapping relationship; determining the target positioning data of the vehicle based on the coordinate data and the second mapping relationship between the standard map and the satellite positioning system coordinate system; determining the positioning result of the vehicle based on the target positioning data.

[0017] Through the above technical means, the position of the vehicle in the current scene can be accurately mapped to the standard map through the first mapping relationship between the local map and the standard map; and the second mapping relationship between the standard map and the satellite positioning system coordinate system enables the standard map coordinates of the vehicle to be accurately converted to the satellite positioning system coordinate system. In this way, the accumulation of errors can be effectively reduced and the accuracy of vehicle positioning can be improved by updating the vehicle position and map information in real time.

[0018] In some embodiments, the method further includes: obtaining at least one training image and at least one training standard map corresponding to the training image; inputting at least one training image and the corresponding training standard map into the initial network model for model training to obtain a preset model.

[0019] Through the above-mentioned technical means, the initial network model is trained through at least one training image and a corresponding training standard map to obtain a preset model, which can improve the accuracy of the preset model prediction and thus improve the accuracy of vehicle positioning.

[0020] In some embodiments, at least one training image and a corresponding training standard map are input into an initial network model for model training to obtain a preset model, including: inputting at least one training image and a corresponding training standard map into the initial network model, generating at least one training local map and at least one training first mapping relationship through the initial network model; determining a first loss value of the initial network model according to at least one training local map, at least one training first mapping relationship, a local map true value corresponding to at least one training image, and at least one first mapping relationship true value; adjusting model parameters of the initial network model according to the first loss value to obtain a preset model.

[0021] Through the above-mentioned technical means, the initial network model is supervised and trained by the local map truth value and the first mapping relationship truth value, and its performance is evaluated by comparing the prediction results with the labels (i.e., the true value data), and the parameters are adjusted accordingly, so that the initial network model can learn the potential rules and features in the local map truth value and the first mapping relationship, so that the preset model shows higher accuracy in prediction, thereby improving the accuracy of vehicle positioning.

[0022] In some embodiments, the method further includes: inputting at least one training image into a pre-trained model, generating at least one intermediate local map through the pre-trained model; determining a second loss value of the pre-trained model based on the at least one intermediate local map and the local map true value corresponding to the at least one training image; and adjusting the model parameters of the pre-trained model based on the second loss value to obtain an initial network model.

[0023] Through the above technical means, the initial network model is based on the trained pre-trained model. The online mapping capability of the trained pre-trained model can be applied to the training of the initial network model with new input, which can accelerate the convergence of the initial network model.

[0024] In a second aspect, an embodiment of the present application provides a vehicle positioning device, the vehicle positioning device comprising an acquisition unit, a determination unit and a prediction unit, wherein:

[0025] An acquisition unit, used to acquire a perception image of the scene where the vehicle is located and initial positioning data of the vehicle;

[0026] A determination unit, used to determine a standard map in a scene where the vehicle is located based on the initial positioning data;

[0027] A prediction unit, configured to input the perceived image and the standard map into a preset model, and generate a local map and a first mapping relationship between the local map and the standard map through the preset model;

[0028] The determination unit is further used to determine the positioning result of the vehicle according to the local map and the first mapping relationship.

[0029] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method described in the first aspect is implemented.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the method described in the first aspect.

[0032] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a vehicle positioning method provided in an embodiment of the present application Figure 1 ;

[0034] Figure 2 A schematic diagram of a composition structure of a perceived image provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of the composition structure of a local map provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the composition structure of a standard map provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of a vehicle positioning process provided by an embodiment of the present application;

[0038] Figure 6 A schematic diagram of a vehicle positioning method provided in an embodiment of the present application Figure 2 ;

[0039] Figure 7 A schematic diagram of a vehicle positioning method provided in an embodiment of the present application Figure 3 ;

[0040] Figure 8 A schematic diagram of an enhancement process of a training standard map provided in an embodiment of the present application;

[0041] Fig. 9 A schematic diagram of a training process of a pre-training model provided in an embodiment of the present application;

[0042] Fig.10 A schematic diagram of a training process of an initial network model provided in an embodiment of the present application;

[0043] Fig.11 A schematic diagram of the structure of a vehicle navigation interface layout provided in an embodiment of the present application;

[0044] Fig.12 A schematic diagram of the structure of a vehicle positioning device provided in an embodiment of the present application;

[0045] Fig.13 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0047] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0048] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0049] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0051] With the development of connected cars, high-precision positioning technology has been applied to intelligent driving to improve driving safety, which can greatly enhance the perception ability of vehicles during driving. However, when the vehicle is driving in a complex environment or the signal of the satellite positioning system (such as the Global Positioning System (GPS)) is weak, the accuracy of vehicle positioning is low, which makes it difficult to meet the requirements of intelligent driving for high-precision positioning.

[0052] For example, in human driving or automatic driving, if the vehicle is traveling on complex road conditions (such as mountainous areas) or the GPS signal is weak, the positioning accuracy of the vehicle is low, which is not conducive to the safety of vehicle driving.

[0053] With the rapid development of autonomous driving technology, the mapless mode has become the technical direction of autonomous driving. Since its dependence on high-definition maps (HDM) is far less than that of the map-based mode, it can control the cost of autonomous driving deployment and has significant advantages in terms of rapid updates. In the mapless mode, the fusion of online bird's eye view (BEV) mapping and standard definition map (SDMap) has become a key technology to improve navigation accuracy and environmental understanding capabilities without relying on high-precision maps. Here, the standard definition map can be referred to as "standard map".

[0054] Among them, BEV online mapping technology is a method of fusing and converting vehicle surrounding environment information (such as sensor data such as cameras, radars, and lidars) into a bird's-eye view representation. This technology can construct the precise location of elements such as roads, obstacles, and traffic signs around the vehicle in real time, thereby helping the autonomous driving system to better understand the current environment. Compared with traditional forward-looking monocular or binocular perception solutions, multi-camera solutions based on BEV input can obtain field of vision (FOV) information far exceeding 360 degrees. This panoramic perception capability enables the autonomous driving system to have a more comprehensive understanding of the environment around the vehicle, thereby constructing a more robust perception system with a wider range of perception. BEV images not only provide real-time environmental information around the vehicle, but also simplify environmental understanding through a bird's-eye view, allowing the autonomous driving system to more easily identify lane lines, traffic participants, and potential obstacles. SDMap is a simpler and more general map form than high-precision maps. It contains basic road information, such as road networks, intersections, lane directions, etc., but usually does not include detailed elements in high-precision maps (such as lane width, precise location of obstacles, etc.). In mapless mode, SDMap can serve as the basic navigation framework of the autonomous driving system, providing basic route planning and navigation information for the vehicle. By integrating BEV online mapping with SDMap, autonomous driving can achieve accurate perception and navigation of the environment without relying on high-precision maps.

[0055] Based on this, the embodiment of the present application provides a vehicle positioning method, which first obtains a perceived image of the scene where the vehicle is located and the initial global positioning system positioning data of the vehicle, obtains a standard map based on the initial positioning data, obtains a local map and a first mapping relationship by inputting the perceived image and the standard map into a preset model, and then uses the first mapping relationship to map the position of the vehicle in the local map to the standard map, and finally obtains the positioning result of the vehicle. In this way, a local map is established in real time through the perceived image of the vehicle and a low-cost standard map, and the initial positioning data is corrected based on the local map and the first mapping relationship to obtain the positioning result of the vehicle, thereby meeting the requirements of intelligent driving for high-precision positioning, improving the positioning accuracy of the vehicle, and further improving the safety of vehicle driving; and the quality of the local map established in real time is improved by using the fusion of multi-source information (perceived images and standard maps, etc.).

[0056] The present application is further described in detail below through the accompanying drawings and specific embodiments.

[0057] In one embodiment of the present application, Figure 1 A schematic diagram of a vehicle positioning method provided in an embodiment of the present application Figure 1 .like Figure 1 As shown, the method may include:

[0058] S101, obtaining a perception image of a scene where a vehicle is located and initial positioning data of the vehicle.

[0059] In an embodiment of the present application, the perceived image is acquired by a sensor (such as a camera) carried by the vehicle. Here, with the vehicle as the center, a perceived image of the scene where the vehicle is located is acquired, that is, a panoramic view of the scene where the vehicle is located can be obtained. It should be noted that the perceived image may include at least one sub-image, and processing the perceived image can obtain map information of the scene where the vehicle is located, such as 360-degree map information or generate 180-degree map information. Here, when the vehicle uses a monocular or binocular camera, due to the limitations of the camera FOV, it is necessary to use continuous frame historical information to generate 360-degree map information or only generate the first 180-degree map information, and there is no specific limitation on this.

[0060] For example, Figure 2 A schematic diagram of the composition structure of a perceived image provided by an embodiment of the present application. Figure 2 As shown, the perceived image may include 6 sub-images; among which, Figure 2 (a) in the figure is the sub-image of the left front of the vehicle. Figure 2 (b) in the figure is the sub-image directly in front of the vehicle. Figure 2 (c) in the figure is the sub-image of the right front of the vehicle. Figure 2 (d) in the figure is the sub-image of the left rear of the vehicle. Figure 2(e) in the figure is the sub-image directly behind the vehicle. Figure 2 (f) in the figure is the sub-image of the right rear of the vehicle; Figure 2 The six sub-images in can obtain a panoramic view of the vehicle's environment. Here, Figure 2 The perception image shown can be obtained by the vehicle's multi-camera. Figure 2 The sensory image shown may also be referred to as a surrounding view, which provides a view of the scene surrounding the vehicle.

[0061] In an embodiment of the present application, the initial positioning data may be obtained by a satellite positioning system; here, the satellite positioning system may be GPS, or it may be a Beidou satellite navigation system, or it may be other satellite positioning systems, without specific limitation. Exemplarily, the satellite positioning system may be GPS, and the vehicle may include a built-in GPS function for obtaining the initial positioning data of the vehicle, where the initial positioning data may also be referred to as initial GPS data. Wherein, the initial GPS data may be expressed as P = [lat, lon], where P is the coordinate of the initial GPS data, lat represents the latitude of the vehicle, and lon represents the longitude of the vehicle. Exemplarily, the initial GPS data is obtained by the vehicle's own Global Navigation Satellite System (GNSS) sensor, and the initial GPS data may have a deviation of several meters to tens of meters due to weather, cost and surrounding environment. In other words, the initial positioning data may be erroneous positioning data, and the initial positioning data needs to be corrected to obtain high-precision positioning.

[0062] S102, determining a standard map in the scene where the vehicle is located according to the initial positioning data.

[0063] In some embodiments, based on the initial positioning data, determining the standard map in the scene where the vehicle is located includes: obtaining a preset standard map in the scene where the vehicle is located; taking the initial positioning data as the center, obtaining a local standard map within a preset range in the preset standard map, and determining the local standard map as the standard map in the scene where the vehicle is located.

[0064] In the embodiment of the present application, the preset standard map is a large-size map, which contains data such as roads, lane lines, and intersections. After obtaining the initial positioning data, a local standard map of a preset range is obtained in the preset standard map with the initial positioning data as the center, and this local standard map is determined as the standard map in the scene. In other words, the standard map is a local map in the preset standard map, which is a small-size map. Here, the standard map is represented by MAP SD Indicates that its range can be expressed as [x SD ,y SD ,hSD ,w SD ] means, where x SD and SD represents the center of the vehicle, h SD and w SD Indicates the height and width of the preset range centered on the vehicle. For example, the preset range is 100 meters (m) before and after and 100 meters left and right centered on the initial positioning data (i.e., the coordinates corresponding to the initial positioning data). The preset range is determined according to actual conditions.

[0065] It can be understood that, with the initial positioning data as the center, obtaining a standard map within a preset range in the preset standard map can more finely display the map information of the scene where the vehicle is located, thereby improving the positioning accuracy; and can reduce the amount of data loaded and improve operating efficiency.

[0066] S103: Input the perceived image and the standard map into a preset model, and generate a local map and a first mapping relationship between the local map and the standard map through the preset model.

[0067] In an embodiment of the present application, after the perceived image and the standard map are acquired, they are input into a preset model, and a local map and a first mapping relationship between the local map and the standard map can be generated through the preset model. The positioning result of the vehicle can be determined through the local map and the first mapping relationship. It should be noted that the local map may also be referred to as an online map, which may be an online map from the perspective of the BEV, or may also be referred to as a scene map, which includes map elements within a certain range around the vehicle. For example, the online map may include road elements (such as lane lines, curbs, crosswalks) and basic road information such as road topology. Exemplarily, Figure 3 A schematic diagram of the composition structure of a local map provided in an embodiment of the present application; Figure 3 As shown, Figure 3 The location of vehicle a1 is the center of the local map. Figure 3 The solid and dotted lines in the figure represent different lane line information. Figure 3 The arrows in the figure represent the topological relationship of the road. Figure 3 The circle a2 in the figure represents an upstream starting point and another downstream end point of the topological relationship. For example, the range of the local map is 60m in front and behind and 30m in left and right with the vehicle a1 as the center.

[0068] As another example, Figure 4 A schematic diagram of the composition structure of a standard map provided in an embodiment of the present application. Figure 4 As shown, the standard map may include road information such as main roads, lane lines and intersections. Figure 4 The rectangular box b1 and the triangle b2 represent Figure 2 and Figure 3The local map range and vehicle center corresponding to the vehicle shown in .

[0069] S104: Determine a positioning result of the vehicle according to the local map and the first mapping relationship.

[0070] In some embodiments, the positioning result of the vehicle is determined based on the local map and the first mapping relationship, including: determining the coordinate data corresponding to the vehicle in the standard map based on the local map and the first mapping relationship; determining the target positioning data of the vehicle based on the coordinate data and the second mapping relationship between the standard map and the satellite positioning system coordinate system; determining the positioning result of the vehicle based on the target positioning data.

[0071] In the embodiment of the present application, according to the first mapping relationship, the center coordinates of the vehicle in the local map are mapped to the coordinate data corresponding to the vehicle in the standard map, and then the target positioning data corresponding to the coordinate data in the satellite positioning system coordinate system is obtained through the second mapping relationship between the standard map and the satellite positioning system coordinate system, thereby determining the positioning result of the vehicle to correct the initial positioning data. Here, if the satellite positioning system is GPS, then the target positioning data can also be referred to as target GPS data. Exemplarily, Figure 5 A schematic diagram of a vehicle positioning process provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, Figure 5 The left sub-image (a) is the local map output by the preset model. Figure 5 The right sub-image (b) in is the standard map. Figure 5 The rectangular box c1 is the range of the local map in the standard map, and the positioning result in the standard map is obtained by the first mapping relationship output by the preset model and returned to the satellite positioning system coordinates. The circle c2 in the rectangular box represents the initial positioning data, and the triangle c3 represents the target positioning data after correction, which is located at the center of the local map.

[0072] It can be understood that, through the first mapping relationship between the local map and the standard map, the position of the vehicle in the current scene can be accurately mapped to the standard map; and the second mapping relationship between the standard map and the satellite positioning system coordinate system enables the standard map coordinates of the vehicle to be accurately converted into satellite positioning system coordinates. In this way, the accumulation of errors can be effectively reduced and the accuracy of vehicle positioning can be improved by updating the vehicle position and map information in real time.

[0073] The embodiment of the present application provides a vehicle positioning method, which first obtains a perceived image of the scene where the vehicle is located and the initial positioning data of the vehicle; then obtains a standard map based on the initial positioning data, obtains a local map and a first mapping relationship by inputting the perceived image and the standard map into a preset model, and then uses the first mapping relationship to map the position of the vehicle in the local map to the standard map, and finally obtains the positioning result of the vehicle. In this way, a local map is established in real time through the perceived image of the vehicle and a low-cost standard map, and the initial positioning data is corrected based on the local map and the first mapping relationship to obtain the positioning result of the vehicle, thereby meeting the requirements of intelligent driving for high-precision positioning, thereby improving the positioning accuracy of the vehicle.

[0074] In another embodiment of the present application, the preset model may include a feature sub-model and a generation sub-model. When the perceived image and the standard map are input into the preset model, a local map and a first mapping relationship between the local map and the standard map are generated by the preset model, such as Figure 6 As shown, the method may include:

[0075] S601, input the perceived image and the standard map into the feature sub-model, and generate fused visual features through the feature sub-model.

[0076] In an embodiment of the present application, the feature sub-model can be used to extract features from the perceived image and the standard map, and to fuse at least two extracted features to obtain fused features.

[0077] In some embodiments, the feature sub-model includes a first feature extraction module, a second feature extraction module and a feature fusion module; for inputting the perceived image and the standard map into the feature sub-model, generating fused visual features through the feature sub-model can include: performing feature extraction on the perceived image through the first feature extraction module to determine the first visual feature corresponding to the perceived image; performing feature extraction on the standard map through the second feature extraction module to determine the second visual feature corresponding to the standard map; performing feature fusion on the first visual feature and the second visual feature through the feature fusion module to determine the fused visual feature of the vehicle.

[0078] In an embodiment of the present application, the first visual feature may be a basic feature, which may include multiple features, and the second visual feature is integrated into a feature branch of the first visual feature to form a fused visual feature. In addition, the first feature extraction model and the second feature extraction model may be the same or different. Exemplarily, the first feature extraction model and the second feature extraction model are different. The first feature extraction model is a convolutional neural network (CNN), and the second feature extraction model is a deep neural network model (Transformer) based on a self-attention mechanism. It should be noted that no matter what network structure the first feature extraction model and the second feature extraction model adopt, they can extract features from the input data to obtain a feature image corresponding to the input data.

[0079] S602, inputting the fused visual features into a generated sub-model, and generating a local map and a first mapping relationship through the generated sub-model.

[0080] In the embodiment of the present application, after obtaining the fused visual features, the fused visual features can be input into the generation sub-model to generate the local map and the first mapping relationship. Here, the generation sub-model can be a multilayer perceptron (Multilayer Perceptron, MLP) or a Transformer encoder-decoder (Encoder-Decoder), which is not specifically limited.

[0081] The embodiment of the present application provides a vehicle positioning method, which performs feature processing on the perceived image and the standard map through the feature sub-model in the preset model to generate fused visual features, so as to generate features that are more suitable for the prediction of the preset model, and can remove irrelevant or redundant features, reduce the complexity of the preset model, and improve the generalization ability of the model; and input the fused visual features into the generation sub-model in the preset model, and generate a local map and a first mapping relationship through the generation sub-model. In this way, the accuracy of the preset model prediction is improved, thereby improving the vehicle positioning accuracy.

[0082] In another embodiment of the present application, Figure 7 A schematic diagram of a vehicle positioning method provided in an embodiment of the present application Figure 3 .like Figure 7 As shown, for the construction of the preset model, the method may include:

[0083] S701, obtaining at least one training image and at least one training standard map corresponding to the training image.

[0084] In the embodiment of the present application, for a training image, it includes at least one sub-image, and the training image is a perception image of the scene where the vehicle is located. By processing the training image, a local map of the scene where the vehicle is located can be obtained. For the acquisition of the training standard map, it can be obtained based on the training initial positioning data corresponding to the acquisition of the training image by the vehicle. Here, if the training initial positioning data is obtained by GPS, then the training initial positioning data can also be called training initial GPS data.

[0085] S702: Input at least one training image and a corresponding training standard map into an initial network model for model training to obtain a preset model.

[0086] In an embodiment of the present application, after obtaining a training image and a corresponding training standard map, the training image and the corresponding training standard map are input into the initial network model. The initial network model first performs feature processing on the training image and the corresponding training standard map respectively, and then fuses the two features to obtain a training fused visual feature, which can be processed by a model related to feature processing in the initial network model, such as a Transformer model, and then the training fused visual feature is input into a loop iterative optimization model in the initial network model, and the loop optimization is iterated to generate the final preset model. Here, the model related to feature processing is also continuously learned during the training process to obtain a trained model related to feature processing, which is the feature sub-model in the above embodiment; the loop iterative optimization model is trained during the training process to obtain a trained loop iterative optimization model, which is the generation sub-model in the above embodiment.

[0087] In some embodiments, for step S702, the method may include: inputting at least one training image and a corresponding training standard map into an initial network model, generating at least one training local map and at least one training first mapping relationship through the initial network model; determining a first loss value of the initial network model based on at least one training local map, at least one training first mapping relationship, a local map true value corresponding to at least one training image, and at least one first mapping relationship true value; adjusting model parameters of the initial network model according to the first loss value to obtain a preset model.

[0088] In the embodiment of the present application, the first loss value can be determined by the following formula:

[0089]

[0090] Among them, L M is the first loss value, which includes the local map related loss and the supervision loss related to the first mapping relationship, To train the local map, Z_MAP onlineis the local map truth value, α is the loss function balance coefficient, To train the first mapping relationship, Z_A is the true value of the first mapping relationship.

[0091] Here, the training process will last for 2 to 7 days depending on the size of the data set. After the training is completed, the initial network model after training can be obtained. The data related to the true value used in the training process can understand the data with labels, and can be used for true value supervised training with the training samples to obtain the preset model.

[0092] In some embodiments, obtaining a local map truth value corresponding to at least one training image may include: obtaining high-precision map data corresponding to at least one training image; and determining a local map truth value corresponding to at least one training image based on at least one training image and the high-precision map data.

[0093] In the embodiment of the present application, after acquiring at least one training image, high-precision map data corresponding to the at least one training image can be acquired, and the at least one training image and the corresponding high-precision map data can be processed to generate a local map truth value corresponding to the at least one training image from the perspective of the BEV. Here, the local map truth value may include road elements and road topology information. The local map truth value can be expressed as Z_MAP online Indicates that Z_MAP online The coordinate data is recorded as [Z_x m ,Z_y m ,Z_h m ,Z_w m ], where Z_x m , Z_y m represents the center of the vehicle, Z_h m , Z_w m Represents the range centered on the vehicle, for example, 60m in front and behind and 30m in left and right around the vehicle, i.e. Z_h m =60m, Z_w m =30m.

[0094] In some embodiments, obtaining the true value of the first mapping relationship may include: enhancing at least one training standard map to determine at least one enhanced training standard map; determining the true value of the first mapping relationship based on the at least one enhanced training standard map and the corresponding local map true value.

[0095] In an embodiment of the present application, after obtaining the local map truth value and the training standard map corresponding to at least one training image, the training standard map is enhanced, which can also be called random data enhancement. For example, the training standard map is randomly translated, cropped, and scaled within a certain range to obtain the enhanced training standard map; then, the first mapping relationship truth value is determined based on the enhanced training standard map and the local map truth value. It should be understood that when the training standard map is not enhanced, the training standard image is aligned with the center of the local map truth value. Since the training standard map is processed, the first mapping relationship is required to convert the local map to the standard map. Specifically, through the relationship: coord(Z_MAP online )=Z_A(coord(X1_MAP SD )) to obtain the first mapping relationship true value Z_A, where X1_MAP SD represents the training standard map after enhancement processing, coord represents the matching key points of the extracted image, and the key matching points are used to represent the local map truth value Z_MAP online With training standard Figure X _MAP SD to associate with a specific area in the

[0096] In the embodiments of the present application, Figure 8 A schematic diagram of an enhancement process of a training standard map provided in an embodiment of the present application. Figure 8 As shown, Figure 8 The left sub-image (a) in the middle shows the training standard map pulled out from the preset standard map according to the preset range with the training initial positioning data as the center. Figure 8 The right sub-image (b) is the training standard map after the training standard map is enhanced by random transformations such as cropping, rotation and scaling. Figure 8 The middle rectangular box d1 represents the range covered by the training local map. Figure 8 The circle d2 in the figure is the location of the vehicle’s initial training positioning data. Figure 8 The middle triangle shape d3 represents the center of the vehicle in the training local map, where the coordinate position of the training initial positioning data is at the geometric center of the training standard map. Figure 8 In the left sub-figure (a), we can see that the two centers overlap (i.e., circle b2 and triangle b3). Figure 8 The right sub-figure (b) shows that the two centers are deviated, so the first mapping relationship truth value is needed to map the vehicle center in the training local map to the training standard map after the enhanced transformation. The first mapping relationship truth value Z_A can be expressed in the form of a matrix, as shown in the following formula:

[0097]

[0098] Among them, t x and t y represents the translation distance of width and height; a, b, c and d represent the relevant parameters of the rotation angle θ, for example, a is cosθ, b is -sinθ, c is sinθ, and d is cosθ.

[0099] For example, t x =W / 2-w / 2, t y =H / 2-h / 2, a=d=1, b=c=0; here, the image width and height of the training standard map are W and H respectively, and the image width and height of the training local map are w and h respectively.

[0100] That is to say, the coordinates of any pixel p in the training local map can be mapped to the training standard map after the first mapping relationship true value Z_A, thereby obtaining the specific positioning true value coordinates of the pixel p.

[0101] It can be understood that by supervising the initial network model through the local map truth value and the first mapping relationship truth value, and evaluating its performance by comparing the prediction results with the labels (i.e., data related to the true value), and adjusting the parameters accordingly, the initial network model can learn the potential rules and features in the local map truth value and the first mapping relationship, thereby showing higher accuracy in prediction, thereby improving the accuracy of vehicle positioning.

[0102] In some embodiments, the method further includes: inputting at least one training image into a pre-trained model, generating at least one intermediate local map through the pre-trained model; determining a second loss value of the pre-trained model based on the at least one intermediate local map and the local map true value corresponding to the at least one training image; and adjusting the model parameters of the pre-trained model based on the second loss value to obtain an initial network model.

[0103] In an embodiment of the present application, after acquiring the training image, the training image is input into the pre-training model. The pre-training model first performs feature processing on the training image, which may be processed by a model related to feature processing in the initial network model, such as a Transformer model. The obtained visual features are then input into a loop iterative optimization model in the initial network model, and the loop optimization iteration is performed to obtain a trained pre-trained model. Here, the model parameters of the trained pre-trained model are used as the model parameters of the initial network model before training. This enables the initial network model to have the ability to generate a local map, simplifying the training. In other words, the initial network model uses the trained pre-trained model as the basis, and the online mapping capability of the trained pre-trained model is applied to the training of the newly input initial network model to accelerate the convergence of the initial network model. Here, during the loop optimization iteration process, the loop iteration parameter can be set to 3.

[0104] In the embodiment of the present application, the second loss value can be determined by the following formula:

[0105]

[0106] Among them, L M0 is the second loss value, which includes the supervised loss of the local map midpoint loss, the line segment classification loss, and the topological relationship loss. To train the local map, Z_MAP online is the local map truth value.

[0107] It can be understood that the initial network model is based on the trained pre-trained model, and the online mapping capability of the trained pre-trained model is applied to the training of the initial network model with new input, which can speed up the convergence of the initial network model.

[0108] An embodiment of the present application provides a vehicle positioning method, which trains an initial network model through at least one training image and a corresponding training standard map to obtain a preset model, thereby improving the accuracy of the preset model prediction and thus improving the accuracy of vehicle positioning.

[0109] In another embodiment of the present application, based on the vehicle positioning method described in the above embodiment, the purpose of the present application is to improve the positioning accuracy and positioning robustness during vehicle driving. A vehicle positioning method provided in the embodiment of the present application is specifically called a vehicle positioning method integrating real-time mapping and GPS, which uses real-time mapping technology based on visual input to restore the road topology and road element structure within a fixed range around the vehicle, and obtains a standard map as a priori map and reference template based on the surrounding range corresponding to the initial GPS data of the vehicle (i.e., the initial positioning data in the above embodiment), and outputs a refined local map and a matching mapping (i.e., a first mapping relationship) with the reference template through a designed cyclic iterative regression method; then, after the matching mapping coordinate parameter conversion, the current vehicle is finely positioned in the standard map. In this way, the dependence on high-precision maps can be completely eliminated in the entire vehicle positioning process, and only the vehicle visual input (i.e., the perceived image), the initial GPS data, and the low-cost standard map can be relied on, so as to achieve high positioning accuracy and high coverage.

[0110] In a possible implementation, a vehicle positioning method integrating real-time mapping and GPS is provided. The detailed steps are as follows:

[0111] Step 1: Calibrate the sensor, using visual sensors such as inertial navigation system (INS), lidar and camera, and use calibration tools to calibrate the internal and external parameters of the sensors in the vehicle. For example, the visual sensor can be a monocular camera, a binocular camera or a panoramic multi-camera;

[0112] In the embodiment of the present application, the laser radar is used to collect lane lines and the like in the process of collecting local map truth values. In the process of model training, the data collected by these sensors will be used to produce the training set and the true value used by the training model. Here, the laser radar is specifically needed to obtain the local map truth value. In actual use, sensors and visual sensors that obtain GPS data are needed. In addition, calibrating the internal and external parameters related to the sensors in the vehicle can reduce the error to ensure the accuracy of the data obtained by the sensors during the use of the vehicle. It should be noted that for the calibration step, it is performed before the vehicle leaves the factory, and calibration is not required during actual use.

[0113] Step 2: Use camera calibration parameters and GPS data in combination with high-precision map data to collect true data, and match the standard map with the high-precision map to produce true training data for local maps, GPS data, high-precision maps, and standard maps.

[0114] In the embodiments of the present application, if there is a public data set of true value training data, the public data set can also be used directly. In addition, for high-precision map data, it can contain detailed information on lane lines, such as width, arrows, etc., which can be provided by manufacturers, etc. After acquiring the GPS data, the local map true value data can be acquired in combination with the high-precision map data and the on-board image data. Here, the on-board image data is the training image in the above embodiment during the training process and the perception image in the above embodiment during actual use; for a training image or perception image, it can include six sub-images, such as the above Figure 2 The form shown may also include sub-images of other data, depending on actual conditions.

[0115] Step 3: The obtained vehicle image data and the corresponding high-precision map data are used to generate the simplified online map truth value from the BEV perspective. The online map truth value is the local map truth value in the above embodiment, which can be expressed as Z_MAP online Indicates that Z_MAP online The coordinate data is recorded as [Z_x m ,Z_y m ,Z_h m ,Z_w m ], where Z_x m and Z_y mrepresents the center of the vehicle, Z_h m and Z_w m Indicates the range centered on the vehicle. For example, the range centered on the vehicle is 60m in front and behind and 30m in left and right, i.e. Z_h m =60m, Z_w m = 30m. The online map truth value may include basic road information such as road elements (such as lane lines, curbs and crosswalks) and road topology. A pre-trained model (such as a feature encoder) can be used to encode the vehicle images (i.e., training images) obtained by a monocular or multi-camera, and the supervised pre-trained model can be trained to output map elements consistent with the online map truth value. For example, the online map range is 60m front and rear and 30m left and right centered on the vehicle, i.e., h m =60m,w m =30m, a Transformer-based feature extraction and online map generation network (such as an MLP model) can be used to finally obtain an online mapping model M0, that is, a pre-trained model after training.

[0116] Step 4: After obtaining the online mapping model M0 in step 3, use M0 as the initial network model, and introduce the vehicle image data, the online map truth from the BEV perspective, and the training standard map within a certain range corresponding to the corresponding GPS data at the corresponding time to construct the training data. Here, the training standard map is denoted as X_MAP SD , whose range can be expressed as [X_x SD ,X_y SD ,X_h SD ,X_w SD ] means, where X_x SD and X_y SD represents the center of the vehicle, X_h SD and X_w SD Indicates the range height and width). Generally, the range of the training standard map is 100m before and after and 100m left and right of the vehicle.

[0117] Step 5: Use the online map truth MAP obtained in step 4 online and training standards Figure X _MAP SD , used to obtain the true value of the matching mapping parameter relationship, where the matching mapping relationship is the first mapping relationship in the above embodiment. Before training, random data enhancement is used to enhance X_MAP SD Perform enhancement processing to generate the enhanced training standard map (denoted as X1_MAP SD ), Z_MAP online With X1_MAP SDThe processing obtains the true value of the matching mapping parameter. For example, the enhancement method is to convert X_MAP SD Do a certain range of random translation, cropping and scaling to get X1_MAP SD , so that the Z_MAP originally aligned with the center point online and X_MAP SD , a matching mapping is needed to convert the local map to the standard map, which can be done through the relationship coord(Z_MAP online )=Z_A(coord(X1_MAP SD )) to obtain the true value of the matching mapping parameter Z_A, where coord represents the matching key points of the extracted image, and the key matching points are used to represent the true value of the online map Z_MAP online With training standard Figure X _MAP SD to associate with a specific area in the

[0118] In the embodiment of the present application, after the training standard map is obtained by the training initial GPS data, the training initial GPS data is at the center of the training standard map, and each pixel in the image corresponds to a point in the Mercator coordinate system (UTM). The training standard map after random transformation is used to produce the training matching relationship mapping truth value, and the training standard map X1_MAP after enhancement processing SD The corresponding UTM coordinates of any pixel in the image can also be calculated through matching relationship mapping.

[0119] It should be noted that the coordinates of each map in the embodiment of the present application need to be consistent, so as to reduce the amount of calculation for conversion between different coordinate systems and avoid errors caused by conversion between different coordinate systems, thereby improving accuracy. Here, the coordinate system of each map can be in units of m, which is not specifically limited.

[0120] Step 6: Obtain the vehicle image (referred to as image I) and the online map truth value Z_MAP online , training standard Figure X _MAP SD After matching the true value of the mapping parameters, the image I and the training standard Figure X _MAP SD As the initial network model input for the fusion of online mapping and standard map, the initial network model may include a model related to feature processing and a loop iteration optimization model. Among them, the model related to feature processing may be a Transformer, which is used to transform the image I and X_MAP respectively. SDPerform feature extraction, and input the extracted features into the loop iterative optimization model, update the online map and matching mapping parameters in the loop iterative optimization model, and finally generate a training online map (i.e., the training local map in the above embodiment can be used Represents) and training matching mapping parameters and the online map truth value Z_MAP online The training initial network model is obtained by constraining the true value Z_A of the matching mapping parameter, calculating the loss, and performing cyclic optimization. Here, the training initial network model can be called the matching model M of the fusion of online mapping and standard map, which is the preset model in the above embodiment.

[0121] In the embodiment of the present application, in the loop iteration optimization model, when the number of loop iterations is set to 1, the input image I and the training standard image X_MAP SD The training online map and training matching mapping parameters are directly output through the loop iteration optimization model. When the number of loop iterations is set to a parameter greater than 1, the training online map and training matching mapping parameters output by the i-1th loop iteration optimization model will be used as the input of the i-th loop iteration optimization model, and after being fused with the fused visual features, the output result of the i-th loop iteration optimization model iteration is obtained; illustratively, the number of loop iterations can be set to 3.

[0122] Step 7: After obtaining the matching model M of the online map and the standard map, the obtained image data (i.e., the perceived image) is preprocessed during the vehicle driving process and input into the matching model M. At the same time, based on the initial GPS data of the current vehicle, a standard map MAP within a certain range around the initial GPS data is pulled from the preset standard map. SD , and input into the matching model M, and output the online map MAP around the current vehicle through the matching model M online And the current vehicle position corresponds to MAP online Matching mapping parameters A with the standard map. Here, the online map MAP online It is the local map described in the above embodiment.

[0123] Step 8: After obtaining the online map and matching mapping parameters, based on the correspondence between the existing pixels of the standard map and the GPS coordinate system (i.e., the second mapping relationship), the matching mapping parameters are used to apply the position of the vehicle in the online map to the standard map coordinate system, and the target GPS data corresponding to the GPS coordinate system is calculated and returned to finally obtain the vehicle positioning result.

[0124] It should be noted that when building an online map, vehicle-mounted image data collected by a monocular or multi-camera can be used as input. Among them, when using vehicle-mounted image data collected by a multi-camera as input, the process of building an online map will be more stable due to the wider camera FOV. In addition, the online map can be represented by semantic segmentation or by a polyline segment based on a fixed number of points. For example, in order to facilitate the generation of road topological relationships for downstream tasks, a polyline segment based on a fixed number of points is used, and an adjacency matrix is ​​used to represent the connection topological relationship between center lines.

[0125] That is to say, in the embodiment of the present application, the vehicle positioning using real-time mapping and integrated GPS can use low-cost standard maps instead of high-precision maps to match the online maps obtained by the vehicle image sensor, so as to calculate the specific position of the online map in the standard map and correct the initial GPS data to obtain accurate GPS data. This avoids the problem of vehicle positioning deviation caused by deviation of initial GPS data due to GPS signal attenuation or road network matching errors, thereby improving positioning accuracy and meeting the requirements of high positioning accuracy.

[0126] In some embodiments, Fig. 9 A schematic diagram of a training process of a pre-training model provided in an embodiment of the present application. Fig. 9 As shown, here, the pre-training model may include a model related to feature processing and a loop optimization iteration model (also referred to as an online map generation model here), and the training process may include:

[0127] S901, start.

[0128] S902, vehicle-mounted image.

[0129] Here, the vehicle-mounted images, that is, the perception images required for training, can be obtained through a monocular or multi-camera.

[0130] S903, feature coding.

[0131] Here, a model related to feature processing may be used to perform feature encoding on the vehicle-mounted image to obtain a first image feature corresponding to the vehicle-mounted image.

[0132] S904, online map generation model.

[0133] Here, the first image feature is input into the online map generation model, and then a training local map is generated by the online map generation model.

[0134] S905, training a local map.

[0135] Here, the training local map is input into the online map generation model, and the training local map truth value corresponding to the training local map supervises the training of the online map generation model.

[0136] S906, the pre-trained model after training.

[0137] Here, when the number of loop iterations reaches the requirement, or the corresponding loss value reaches the requirement, the trained pre-trained model can be obtained.

[0138] In the embodiment of the present application, it is necessary to first train an online map generation model, i.e., a pre-trained model, by inputting a vehicle image and outputting a training local map using a loop iterative optimization method, and then performing supervised training by comparing the local map true value generated by a high-precision map to obtain a trained pre-trained model.

[0139] In some embodiments, the model parameters of the initial network model before training are the model parameters of the pre-trained model after training. Fig.10 A schematic diagram of the training process of an initial network model provided in an embodiment of the present application. The initial network model includes a model related to feature processing and a loop iteration optimization model (also referred to as an online map generation model here). It should be noted that the online map generation model in the initial network model has added some new branches at the feature level compared to the online map generation in the pre-trained model, namely, the features related to the standard map. Fig.10 As shown, the training process may include:

[0140] S1001, start.

[0141] S1002, vehicle-mounted image.

[0142] S1003, feature coding.

[0143] Here, the operations of S1002 to S1003 are similar to those of S902 and S903, and are not further explained.

[0144] S1004, training initial GPS data.

[0145] Here, the initial training GPS data corresponding to the feature image can be obtained to obtain the training standard map.

[0146] S1005, training standard map.

[0147] Here, taking the training initial GPS data as the center, a training standard map of a preset range is obtained in the preset standard map.

[0148] S1006, feature coding.

[0149] Here, a model related to feature processing may be used to perform feature encoding on the training standard map to obtain second image features corresponding to the training standard map.

[0150] S1007, online map generation model.

[0151] Here, the first image feature and the second image feature are fused and input into the online map generation model, and then the training local map and the training first mapping relationship are generated by the online map generation model.

[0152] S1008, training a local map.

[0153] S1009, training a first mapping relationship.

[0154] Here, the first mapping relationship may be referred to as a matching mapping parameter or an affine transformation parameter, which may be represented in the form of a matrix. Exemplarily, the first mapping relationship is a matrix of two rows and three columns.

[0155] Here, the training local map and the training first mapping relationship are input into the online map generation model, and the training local map truth value and the first mapping relationship truth value corresponding to the training local map supervise the training of the online map generation model.

[0156] S1010, initial network model after training.

[0157] Here, when the number of loop iterations reaches the requirement, or the corresponding loss value reaches the requirement, the trained pre-trained model can be obtained.

[0158] That is to say, after the pre-trained model is trained, the initial network model reuses the weights (i.e., model parameters) of the pre-trained model after training, introduces the initial GPS data for training and the training standard map in the relevant surrounding area, transforms the standard definition map by random data enhancement, obtains the matching mapping parameters between the training standard map and the local map truth value, and thus returns the transformed training standard map. During the training process, the vehicle-mounted image and the training standard map are used as the input of the initial network model, and the online map and matching mapping parameters are output through a cyclic iterative optimization method, and the model is supervised for training with the local map truth value and the first mapping relationship truth value.

[0159] It can be understood that the embodiment of the present application proposes a vehicle positioning method that integrates real-time mapping and GPS. The method uses the visual image of the vehicle and the standard map obtained through the initial GPS data of the vehicle as the input of the preset model. It does not rely on high-precision maps. It is a flexible and low-cost vehicle positioning method that can be applied to positioning methods in multiple scenarios such as urban areas and rural areas. In addition, the cyclic iterative optimization method proposed by the method can improve the generation quality of the local map and the first mapping relationship in a coarse-to-fine manner, and use the first mapping relationship to match the current vehicle position to the specific position in the standard map to improve the positioning accuracy.

[0160] In another possible implementation, a vehicle positioning method integrating real-time mapping and GPS is provided, which includes the following parts:

[0161] In the embodiment of the present application, during the model training process, it can be implemented under the Ubuntu 18.04 system and the Pytorch deep learning framework (where the deployment process is implemented on a chip equipped with the NVIDIA Orin architecture). Here, the computer is configured with an Intel (R) Core (TM) i5-13600K processor, a main frequency of 4.0GHz, a memory of 128GB, a graphics card of 8 NVIDIARTX 4090, and a video memory of 24GB. The steps of data collection and processing of vehicle-mounted images, training initial GPS, and training standard maps can be obtained using the vehicle-side data collection platform, or public data sets (such as Nuscenes) and open street maps (OpenStreetMap, OSM) can be directly used to obtain the data required for the model algorithm to obtain the training process.

[0162] 1. Data processing

[0163] The image data source of the vehicle-mounted image can be images obtained by monocular, binocular or panoramic cameras, and the cameras used need to go through a series of sensor calibration to ensure the correctness of the internal and external parameters of the camera, so that they can be correctly matched with the high-precision map. Figure 2 The six sub-images shown are the surrounding images of the vehicle. The surrounding images are used as input to generate the training online map. Among them, the six images can cover the 360-degree range around the vehicle, and the information is rich enough. If the images collected by a monocular or binocular camera are used as input to generate the training online map, due to the limitation of the camera FOV, it is necessary to use the continuous frame history information to generate 360-degree map information or only generate the first 180-degree map information.

[0164] The true value of the online map can be obtained by cropping, projecting, and abstract modeling of the high-precision map. Its unit is meter. During use, it is uniformly fixed to a certain range (for example, 60 meters in front and behind the center of the vehicle and 30 meters to the left and right). The center of the vehicle is always the center of the range, and the visualization is a fixed-resolution image. The true value of the online map can be as follows: Figure 3 In the form shown, the type of map elements modeled here can be determined according to the downstream tasks, such as zebra crossings, curbs, lane lines, center lines, and topological relationships. The initial GPS data for training and the initial GPS data can come from the vehicle's own GNSS sensor. Affected by weather, cost, and surrounding environment, the positioning data accuracy may have a deviation of several meters to tens of meters.

[0165] For training standard maps and standard maps in actual use, they usually cover a fixed range around the vehicle, for example, 100 meters in front and behind the vehicle, which can be as follows: Figure 4 For training standard maps and standard maps, it can include data such as main roads, lane lines and intersections, for example Figure 4 The rectangular box b1 and the triangle b2 in the figure represent Figure 2 and Figure 3 The local map range and vehicle center corresponding to the vehicle in the map. The GPS coordinate system here needs to be consistent with the coordinate system in the standard map to avoid the problem of difficulty in calculation and error caused by the conversion of different coordinate systems due to the different encryption methods of different coordinate systems. Due to the scale of the coordinates of the GPS and the standard map, the one-to-one mapping relationship between each pixel in the standard map and the GPS coordinates can be calculated, that is, the second mapping relationship, which can map the image coordinates of any point in the standard map back to the GPS coordinates.

[0166] The data required for the training of the pre-trained model are vehicle images and online map truth values. The training process of the initial network model requires the use of vehicle images, online map truth values, enhanced training standard maps, and the matching mapping parameter truth values ​​between online maps and enhanced training standard maps. The training standard map corresponding to the initial GPS data of the vehicle is made into an enhanced training standard map. For example Figure 8 As shown, Figure 8 The left sub-image (a) in the middle shows the training standard map pulled out according to the preset range. Figure 8 The right sub-image (b) is the training standard map after the training standard map is enhanced by random transformations such as cropping, rotation and scaling. The coordinate position of the initial GPS data for training is at the geometric center of the training standard map. Figure 8 In the left sub-image (a), we can see that the two centers overlap. Figure 8The right sub-image (b) shows that the two centers are deviated, so the first mapping relationship truth value is needed to map the vehicle center in the training local map to the training standard map after the enhanced transformation. At this time, the matching mapping parameter can be expressed by the above formula (2). After the matching mapping parameter mapping, any pixel coordinate in the training local map can be mapped to the training standard map, and then the coordinates of the initial GPS data of each pixel are obtained.

[0167] 2. Network training.

[0168] In the initial network model, two variables are output by the model, namely, a training online map and a training matching mapping parameter. The training online map output uses polyline segments based on a fixed number of points to represent road elements, and uses an adjacency matrix to represent the topological relationship between center lines. The matching mapping parameter can be a matrix of 2 rows and 3 columns.

[0169] Obtaining the preset model is divided into two stages. The first stage is to train the pre-trained model, and the second stage is to train the initial network model (i.e., the matching model M of the fusion of online mapping and standard map). The first stage can be the process shown in 9, using the vehicle image as input and extracting the vehicle image features through feature encoding. The feature encoding can be based on the structure of the CNN network or the structure of the Transformer network. It should be noted that no matter which network model is used, the encoded first image feature F will be output. I The first image feature F after encoding I The second image feature F M The training online map is output in the loop iteration optimization module. Here, the loop iteration optimization module can be MLP, convolutional neural network, or Transformer (Encoder-Decoder). The training process supervises the output training online map and the true value of the online map, and calculates the second loss value through the above formula (3), thereby obtaining the online mapping model M0.

[0170] The online mapping capability (i.e., model parameters) of the online mapping model M0 obtained in the first stage is applied to the initial network model training of the newly added input to accelerate the model convergence. The matching mapping parameter truth value Z_A between the newly added enhanced training standard map and the online map truth value - the enhanced transformed training map is used as the model input, and the second image feature F is obtained through feature encoding based on the convolutional neural network or the Transformer network. M , and the first image feature F I Fusion together as loop iterative optimization model input, and output training online map Matching mapping parameters with training The training process uses the output line map true value and the matching mapping relationship true value for supervised training, and calculates the first loss value using the above formula (1) to obtain the matching model M.

[0171] 3. Reasoning and verification.

[0172] In actual use, when positioning the vehicle, the vehicle's visual sensor collects the perceived image in real time, the GNSS sensor obtains the initial GPS data (coordinates P = [lat, lon]), and pulls the standard map within a fixed range around the current initial GPS data from the preset standard map database. At this time, the coordinates P of the initial GPS data have a certain degree of deviation. The perceived image is encoded through the feature sub-model in the preset model to obtain the first visual feature, and the standard map is encoded through the feature sub-model in the preset model to obtain the second visual feature. The two visual features are fused and input into the generated sub-model in the preset model. Data online map MAP online (i.e., local map) and the first mapping relationship (i.e., matching mapping parameter A). Since each pixel in the standard map has a corresponding relationship G (i.e., the second mapping relationship) with the GPS coordinates, the image coordinates of the vehicle center coordinates in the online map in the standard definition map are obtained through the output matching mapping parameter A, and then the target GPS data corresponding to the coordinates are obtained through the corresponding relationship G, i.e., the corresponding coordinates of the real GPS data (i.e., the target positioning data). The schematic diagram of the process of obtaining the coordinates corresponding to the real GPS data is as mentioned above. Figure 5 As shown, Figure 5 The left sub-image in the middle is the local map output by the preset model. Figure 5 The right sub-image in is the standard map. Figure 5 The rectangular box in the figure is the range of the local map in the standard map. The positioning result in the standard map is obtained by the first mapping relationship output by the preset model and returned to the GPS coordinates. The circle c2 in the rectangular box represents the initial GPS data, and the triangle c3 represents the target GPS data after correction, which is located at the center of the local map. Fig.11 A schematic diagram of the structure of a vehicle navigation interface layout provided in an embodiment of the present application is shown as follows: Fig.11 As shown, the vehicle's navigation interface 110 may include a vehicle perception establishment interface 1101 and a map navigation interface 1102. The output online map can be used to provide road topology and road structure perception priors to downstream intelligent driving related modules, and can also provide input to the vehicle perception establishment interface 1101. The corrected GPS data can be provided to the vehicle navigation service, and can also provide data to the map navigation interface to reduce the occurrence of vehicle logo drift and improve vehicle positioning accuracy.

[0173] That is to say, during automatic driving or manual driving, when the vehicle is traveling in complex road conditions, the positioning method provided in the embodiment of the present application can be used to obtain accurate positioning, thereby improving the positioning accuracy of vehicle navigation.

[0174] Here, through the specific elaboration of the above-mentioned embodiments, it can be seen that by using the cyclic iterative regression of the features corresponding to the fused on-board image and the standard map, a multi-scale feature extraction module (i.e., a feature extraction model) can be used to extract the deep features of the on-board image and the standard map respectively, and merge them with the online map and the matching mapping parameters. The module is trained in an iterative regression supervision manner to achieve the effect of improving the accuracy of the online map and the matching mapping parameters; in addition, the matching mapping parameters of the local map and the standard map established in real time are calculated to locate the current position of the vehicle at a specific position in the standard map, and the corrected corresponding GPS coordinates are calculated, and finally an accurate positioning result is obtained. In this way, the dependence on high-precision maps can be completely eliminated, and only the perception images obtained by the vehicle visual sensor, the initial GPS signals and the low-cost standard maps can be relied on to achieve high positioning accuracy and high coverage, and the perception results can be used more efficiently and accurately to improve the accuracy of online mapping and vehicle positioning.

[0175] Based on the above embodiments, the present application also provides a vehicle positioning device. Fig.12 This is a schematic diagram of the structure of a vehicle positioning device provided in an embodiment of the present application. Fig.12 As shown, the vehicle positioning device 120 includes an acquisition unit 1001, a determination unit 1002, and a prediction unit 1203, wherein:

[0176] An acquisition unit 1201 is used to acquire a perception image of a scene where a vehicle is located and initial positioning data of the vehicle;

[0177] A determination unit 1202, configured to determine a standard map in a scene where the vehicle is located according to the initial positioning data;

[0178] The prediction unit 1203 is used to input the perceived image and the standard map into a preset model, and generate a local map and a first mapping relationship between the local map and the standard map through the preset model;

[0179] The determination unit 1202 is further configured to determine a positioning result of the vehicle according to the local map and the first mapping relationship.

[0180] In some embodiments, the determination unit 1202 is also used to obtain a preset standard map in the scene where the vehicle is located; with the initial positioning data as the center, obtain a local standard map within a preset range in the preset standard map, and determine the local standard map as the standard map in the scene where the vehicle is located.

[0181] In some embodiments, the preset model includes a feature sub-model and a generation sub-model; the perceived image and the standard map are input into the preset model, and a local map and a first mapping relationship between the local map and the standard map are generated through the preset model, including: the perceived image and the standard map are input into the feature sub-model, and a fused visual feature is generated through the feature sub-model; the fused visual feature is input into the generation sub-model, and a local map and a first mapping relationship are generated through the generation sub-model.

[0182] In some embodiments, the feature sub-model includes a first feature extraction module, a second feature extraction module and a feature fusion module; the determination unit 1202 is also used to perform feature extraction on the perceived image through the first feature extraction module to determine the first visual feature corresponding to the perceived image; and to perform feature extraction on the standard map through the second feature extraction module to determine the second visual feature corresponding to the standard map; and to perform feature fusion on the first visual feature and the second visual feature through the feature fusion module to determine the fused visual feature of the vehicle.

[0183] In some embodiments, the determination unit 1202 is also used to determine the coordinate data corresponding to the vehicle in the standard map based on the local map and the first mapping relationship; determine the target positioning data of the vehicle based on the coordinate data and the second mapping relationship between the standard map and the satellite positioning system coordinate system; and determine the positioning result of the vehicle based on the target positioning data.

[0184] In some embodiments, see Fig.12 The vehicle positioning device 120 also includes a training unit 1204; wherein the acquisition unit 1201 is also used to acquire at least one training image and at least one training standard map corresponding to the training image; the training unit 1204 is used to input at least one training image and the corresponding training standard map into the initial network model for model training to obtain a preset model.

[0185] In some embodiments, the training unit 1204 is further used to input at least one training image and a corresponding training standard map into the initial network model, generate at least one training local map and at least one training first mapping relationship through the initial network model; determine a first loss value of the initial network model according to at least one training local map, at least one training first mapping relationship, a local map true value corresponding to at least one training image, and at least one first mapping relationship true value; and adjust model parameters of the initial network model according to the first loss value to obtain a preset model.

[0186] In some embodiments, the training unit 1204 is further used to input at least one training image into a pre-trained model, generate at least one intermediate local map through the pre-trained model; determine a second loss value of the pre-trained model based on the at least one intermediate local map and the local map true value corresponding to the at least one training image; and adjust the model parameters of the pre-trained model based on the second loss value to obtain an initial network model.

[0187] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0188] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0189] In yet another embodiment of the present application, the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the above method is implemented when the processor executes the computer program.

[0190] In yet another embodiment of the present application, the present application further provides a computer-readable storage medium on which a computer program is stored, and the computer program implements the above method when executed by a processor. The computer-readable storage medium may be transient or non-transient.

[0191] In yet another embodiment of the present application, the present application further provides a computer program product, including a computer program or an instruction, which, when executed by a processor, implements some or all of the steps in the above method. The computer program product can be implemented specifically by hardware, software, or a combination thereof. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0192] It should be noted that Fig.13 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Fig.13 As shown, the hardware entity of the electronic device 130 includes: a processor 1301, a communication interface 1302 and a memory 1303, wherein:

[0193] The processor 1301 generally controls the overall operation of the electronic device 130 .

[0194] The communication interface 1302 may enable the electronic device 130 to communicate with other terminals or servers via a network.

[0195] The memory 1303 is configured to store instructions and applications executable by the processor 1301, and can also cache data to be processed or processed by the processor 1301 and each module in the electronic device 130 (such as image data, GPS data, and map data, etc.), which can be implemented by flash memory (FLASH) or RAM. Data can be transmitted between the processor 1301, the communication interface 1302, and the memory 1303 through the bus 1304.

[0196] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0197] In yet another embodiment of the present application, the present application further provides a vehicle, which includes the electronic device described in the above embodiment.

[0198] Here, the electronic device may be a vehicle computer in a vehicle.

[0199] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0200] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0201] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the above-mentioned units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0202] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0203] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0204] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, read-only memories, magnetic disks or optical disks.

[0205] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0206] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A vehicle positioning method, characterized in that: The method comprises: Acquire a perception image of a scene in which a vehicle is located and initial positioning data of the vehicle; Determining a standard map in the scene where the vehicle is located according to the initial positioning data; Inputting the perceived image and the standard map into a preset model, and generating a local map and a first mapping relationship between the local map and the standard map through the preset model; A positioning result of the vehicle is determined according to the local map and the first mapping relationship.

2. The vehicle positioning method according to claim 1, characterized in that: The step of determining a standard map in a scene where the vehicle is located according to the initial positioning data includes: Obtaining a preset standard map in the scene where the vehicle is located; Taking the initial positioning data as the center, a local standard map within a preset range is obtained in the preset standard map, and the local standard map is determined as the standard map in the scene where the vehicle is located.

3. The vehicle positioning method according to claim 1, characterized in that: The preset model includes a feature sub-model and a generation sub-model; the step of inputting the perceived image and the standard map into the preset model, and generating a local map and a first mapping relationship between the local map and the standard map through the preset model includes: Inputting the perceived image and the standard map into the feature sub-model, and generating a fused visual feature through the feature sub-model; The fused visual features are input into the generation sub-model, and the local map and the first mapping relationship are generated through the generation sub-model.

4. The vehicle positioning method according to claim 3, characterized in that: The feature sub-model includes a first feature extraction module, a second feature extraction module and a feature fusion module; the step of inputting the perceived image and the standard map into the feature sub-model and generating fused visual features through the feature sub-model includes: Extracting features from the perceived image by the first feature extraction module to determine a first visual feature corresponding to the perceived image; and extracting features from the standard map by the second feature extraction module to determine a second visual feature corresponding to the standard map; The feature fusion module performs feature fusion on the first visual feature and the second visual feature to determine a fused visual feature of the vehicle.

5. The vehicle positioning method according to any one of claims 1 to 4, characterized in that: Determining the positioning result of the vehicle according to the local map and the first mapping relationship includes: Determining coordinate data corresponding to the vehicle in the standard map according to the local map and the first mapping relationship; Determining target positioning data of the vehicle according to the coordinate data and a second mapping relationship between the standard map and a satellite positioning system coordinate system; A positioning result of the vehicle is determined according to the target positioning data.

6. The vehicle positioning method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire at least one training image and a training standard map corresponding to the at least one training image; The at least one training image and the corresponding training standard map are input into the initial network model for model training to obtain the preset model.

7. The vehicle positioning method according to claim 5, characterized in that: The step of inputting the at least one training image and the corresponding training standard map into the initial network model for model training to obtain the preset model comprises: Inputting the at least one training image and the corresponding training standard map into the initial network model, and generating at least one training local map and at least one training first mapping relationship through the initial network model; Determine a first loss value of the initial network model according to the at least one training local map, the at least one training first mapping relationship, a local map true value corresponding to the at least one training image, and at least one first mapping relationship true value; The model parameters of the initial network model are adjusted according to the first loss value to obtain the preset model.

8. The vehicle positioning method according to claim 7, characterized in that: The method further comprises: Inputting the at least one training image into a pre-trained model, and generating at least one intermediate local map through the pre-trained model; Determining a second loss value of the pre-trained model according to the at least one intermediate local map and a local map true value corresponding to the at least one training image; The model parameters of the pre-trained model are adjusted according to the second loss value to obtain the initial network model.

9. A vehicle positioning device, characterized in that: The vehicle positioning device comprises an acquisition unit, a determination unit and a prediction unit, wherein: The acquisition unit is used to acquire a perception image of a scene where a vehicle is located and initial positioning data of the vehicle; The determining unit is used to determine a standard map in the scene where the vehicle is located according to the initial positioning data; The prediction unit is configured to input the perceived image and the standard map into a preset model, and generate a local map and a first mapping relationship between the local map and the standard map through the preset model; The determination unit is further used to determine the positioning result of the vehicle according to the local map and the first mapping relationship.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program executable on the processor, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the method according to any one of claims 1 to 8 are implemented.

Images