A method and device for constructing an online vector map based on a pre-trained world model

Abstract

The present invention discloses a method and device for constructing an online vector map based on a pre-trained world model, and belongs to the field of autonomous driving perception technology. The method includes: obtaining a multi-perspective image at the current moment; outputting prediction data for multiple future moments for the multi-perspective image at the current moment based on a pre-trained world model; inputting the multi-perspective image at the current moment and the prediction data for multiple future moments into a trained online vector map model, the online vector map model is used to generate a first bird's-eye view feature for the multi-perspective image at the current moment, and to generate a corresponding second bird's-eye view feature for each prediction data at a future moment, and by fusing the second bird's-eye view feature into the first bird's-eye view feature, a global bird's-eye view feature with future compensation information is obtained, and an online vector map is generated using the global bird's-eye view feature. The present invention can improve the accuracy of online vector maps.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving perception technology, and in particular to an online vector map construction method and device based on a pre-trained world model. Background Art

[0002] With the rapid development of artificial intelligence technology, the requirements for the safety and accuracy of autonomous driving technology have become more stringent. Autonomous driving systems must have high-performance environmental perception capabilities in complex driving environments. Vehicles obtain driving environment information in real time through perception systems, understand road semantics, traffic signals, people and other targets, and lay a foundation for subsequent autonomous driving planning and decision-making. Therefore, the ability to understand the semantics of the driving environment, especially the ability to build semantic vector maps, has become a key cornerstone for the practical application of autonomous driving technology.

[0003] In the related art, vector maps are constructed using perception data acquired in real time. However, due to the existence of prediction bias, the accuracy of online vector maps is poor. Summary of the invention

[0004] The present invention provides a method and device for constructing an online vector map based on a pre-trained world model. The technical solution is as follows:

[0005] In one aspect, a method for constructing an online vector map based on a pre-trained world model is provided, the method comprising:

[0006] Get the multi-view image at the current moment;

[0007] Based on the pre-trained world model, it outputs prediction data for multiple future moments based on the multi-view images at the current moment.

[0008] The multi-view image at the current moment and the predicted data at multiple future moments are input into a trained online vector map model, the online vector map model is used to generate a first bird's-eye view feature for the multi-view image at the current moment, and to generate a corresponding second bird's-eye view feature for the predicted data at each future moment, and to obtain a global bird's-eye view feature with future compensation information by fusing the second bird's-eye view feature into the first bird's-eye view feature, and to generate an online vector map using the global bird's-eye view feature.

[0009] On the other hand, a device for constructing an online vector map based on a pre-trained world model is provided, the device comprising:

[0010] A first acquisition unit, used to acquire a multi-view image at a current moment;

[0011] A second acquisition unit is used to output prediction data at multiple future moments for the multi-view image at the current moment based on the pre-trained world model;

[0012] A map construction unit is used to input the multi-view image at the current moment and the prediction data at multiple future moments into a trained online vector map model, wherein the online vector map model is used to generate a first bird's-eye view feature for the multi-view image at the current moment, and to generate a corresponding second bird's-eye view feature for the prediction data at each future moment, and to obtain a global bird's-eye view feature with future compensation information by fusing the second bird's-eye view feature into the first bird's-eye view feature, and to generate an online vector map using the global bird's-eye view feature.

[0013] On the other hand, a computer device is provided, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the steps of the above-mentioned method for constructing an online vector map based on a pre-trained world model.

[0014] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned method for constructing an online vector map based on a pre-trained world model are implemented.

[0015] On the other hand, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for constructing an online vector map based on a pre-trained world model.

[0016] The technical solution provided by the present invention can at least bring the following beneficial effects:

[0017] By pre-training the world model, the world model can be used to output prediction data for multiple future moments based on the multi-view images obtained at the current moment. By compensating the prediction data for the future moments to the current moment, the information of the current online vector map can be compensated, which can improve the accuracy of the online vector map and provide a basis for subsequent autonomous driving planning and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1It is a flow chart of a method for constructing an online vector map based on a pre-trained world model provided by an embodiment of the present invention;

[0020] Figure 2 It is a structural diagram of an online vector map construction device based on a pre-trained world model provided by an embodiment of the present invention;

[0021] Figure 3 It is a hardware architecture diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] Please refer to Figure 1 , an embodiment of the present invention provides an online vector map construction method based on a pre-trained world model, the method comprising:

[0024] Step 100, obtaining a multi-view image at the current moment;

[0025] Step 102, based on the pre-trained world model, output prediction data for multiple future moments for the multi-view image at the current moment;

[0026] Step 104: input the multi-view image at the current moment and the predicted data at multiple future moments into the trained online vector map model, wherein the online vector map model is used to generate a first bird's-eye view feature for the multi-view image at the current moment, and to generate a corresponding second bird's-eye view feature for the predicted data at each future moment, and to obtain a global bird's-eye view feature with future compensation information by fusing the second bird's-eye view feature into the first bird's-eye view feature, and to generate an online vector map using the global bird's-eye view feature.

[0027] In an embodiment of the present invention, by pre-training a world model, the world model can be used to output prediction data for multiple future moments based on the multi-view images obtained at the current moment. By compensating the prediction data for the future moments to compensate for the current moment, thereby compensating for the information of the current online vector map, the accuracy of the online vector map can be improved, which is more helpful to provide a basis for subsequent autonomous driving planning and decision-making.

[0028] Described below Figure 1 How the various steps are performed.

[0029] First, with respect to step 100, a multi-view image at the current moment is acquired.

[0030] In the embodiment of the present invention, a plurality of cameras are arranged on the vehicle at intervals to obtain multi-view images in the driving environment in real time by the plurality of cameras, wherein the multi-view images are two-dimensional images.

[0031] Then, for step 102, based on the pre-trained world model, prediction data for multiple future moments are output for the multi-view images at the current moment.

[0032] In order to obtain information at a future time, it is necessary to use the multi-view images at the current time to predict the information at the future time to obtain prediction data. In the embodiment of the present invention, the prediction data can be a multi-view image or point cloud data.

[0033] Considering that a bird's-eye view feature is needed when constructing a vector map, and point cloud data contains more three-dimensional features, more accurate bird's-eye view features can be obtained, and in order to reduce the workload of the online vector map model and improve the efficiency of online vector map construction, preferably, the predicted data is point cloud data.

[0034] The following uses point cloud data as an example to illustrate the training method of the world model.

[0035] The training method of the world model includes the following steps S1-S8:

[0036] S1: Acquire multiple first sample pairs based on historical data; the first sample pairs include a first sample multi-view image at time t0 as input and a first sample multi-view image at time t1-t2 as output. n The first sample point cloud data corresponding to each time; n is an integer greater than 1;

[0037] The multiple first sample pairs may be obtained for the same vehicle or for different vehicles.

[0038] When obtaining the first sample pair for training the world model, since the predicted data is point cloud data, in order to ensure the accuracy of the training results, the vehicle used to obtain the first sample pair is equipped with a laser radar in addition to multiple cameras for interval viewing. The laser radar is used to obtain the point cloud data of the vehicle at each moment.

[0039] After obtaining the historical data of the vehicle, the historical data includes the multi-view images and point cloud data corresponding to each historical moment. Then, according to the value of n, continuous (n+1) historical moments can be selected from the historical moments, and the farthest historical moment among the selected (n+1) historical moments is taken as the t0 moment, and the nearest n historical moments are taken as t1-t2. n time, and based on t0-t n The historical data at time instant obtains the first sample pair.

[0040] S2: Using the first backbone network to extract features from the first sample multi-view image at time t0, to obtain the first sample multi-view two-dimensional features at time t0;

[0041] Among them, the first backbone network can sample a Resnet network, such as a Resnet-101 network.

[0042] S3: using a first bird's-eye view feature extractor to project the first sample multi-view two-dimensional feature into the bird's-eye view space to obtain the first sample bird's-eye view feature at time t0;

[0043] The first bird's-eye view feature extractor can be implemented based on convolution, and is used to obtain the first sample bird's-eye view feature through a spatial cross-attention mechanism and a feed-forward neural network.

[0044] S4: using the imitation rendering module to imitate the rendering operation of the bird's-eye view feature of the first sample in the latent space, and calculating the geometric embedding feature at time t0;

[0045] In one embodiment of the present invention, the simulated rendering operation can be implemented in the following manner:

[0046] S41: based on the bird's-eye view grid corresponding to the sample bird's-eye view feature, emitting multiple light rays outward from the origin, the multiple light rays extending from the origin to multiple corresponding bird's-eye view grids;

[0047] S42: for each light ray, performing the following steps: uniformly sampling along the light ray and calculating the bird's-eye view feature of each sampling point; and calculating the light ray feature of the light ray based on the bird's-eye view feature of each sampling point and its conditional probability;

[0048] S43: Calculate the geometric embedding features using the light features of each light ray and the conditional probability matrix.

[0049] The origin is the location of the vehicle. Since the bird's-eye view feature corresponds to multiple bird's-eye view grids, multiple rays are emitted from the origin in accordance with the rendering operation, so that the rays correspond to the bird's-eye view grids. The bird's-eye view feature of each sampling point can be obtained through uniform sampling. The light feature of the i-th ray can be calculated using the following feature expectation function:

[0050]

[0051] in, is the light characteristic of the i-th light, is the conditional probability corresponding to the kth sampling point when uniform sampling is performed along the i-th ray, n is the total number of sampling points on the i-th ray, is the bird's-eye view feature corresponding to the kth sampling point.

[0052] After obtaining the light features of each ray, the geometric embedding features are calculated using the following formula:

[0053]

[0054] in, is the geometric embedding feature, is a conditional probability matrix, each element in which is used to characterize the conditional probability of the i-th bird's-eye view grid, and B'0 is a light feature matrix, each element in which is used to characterize the light feature of the i-th light.

[0055] S5: Use the Transformer module to combine the geometric embedding features at time t0 with the expected ego-vehicle motion conditions at time t1 to predict the geometric embedding features at time t1;

[0056] In the embodiment of the present invention, at each moment, the vehicle corresponds to a self-vehicle motion condition, which may include vehicle speed, whether the accelerator is pressed, whether the steering is turned, etc. The self-vehicle motion condition at the current moment can be used to obtain the expected self-vehicle motion condition at the next moment.

[0057] The multi-layer perceptron in the Transformer module can encode the ego-vehicle motion conditions into high-dimensional embedded features, which are added to the bird's-eye view feature query by aligning the coordinate system of the ego-vehicle motion conditions. Then, the deformable self-attention mechanism is used as the query vector. The geometric embedding features at the current moment are used as keys and weights together with the query vector through the spatiotemporal cross-attention mechanism and the feedforward neural network to obtain the refined bird's-eye view features as the input of the next layer of Transformer. After multiple layers of superposition, the predicted geometric embedding features at the next moment are obtained.

[0058] S6: Using the Transformer module, combined with the predicted tj The geometric embedding features and expected t at the moment j+1 The vehicle motion condition at time t is predicted j+1 The geometric embedding feature at the moment; j is an integer in [1, n-1];

[0059] Each time, the geometric embedding features of the current moment and the vehicle motion conditions of the current moment are used as inputs to the Transformer module, so that the geometric embedding features of the next moment can be predicted. By continuously iterating, the geometric embedding features of each moment in the future can be obtained.

[0060] S7: The predicted t1-t n The geometric embedding features corresponding to each time are used to generate a 3D voxel grid using the projection layer, and the center point of the voxel grid is used as the representative point to generate t1-t n Prediction point cloud data corresponding to each moment;

[0061] S8: Using t1-t n The loss function between the first sample point cloud data and the predicted point cloud data corresponding to each time moment is used to complete the training of the world model.

[0062] Among them, the loss function can be implemented by the cross entropy loss of light.

[0063] Normally, when acquiring three-dimensional features, the depth of a two-dimensional multi-view image is estimated to obtain the three-dimensional features. In the embodiment of the present invention, after the three-dimensional features are obtained, they are projected to two dimensions to obtain bird's-eye view features, and then the bird's-eye view features are projected back to three dimensions to obtain point cloud data. Compared with the three-dimensional features obtained only by depth estimation, the three-dimensional point cloud data obtained by using the bird's-eye view features will have more feature content. In addition, by performing a simulation rendering operation on the bird's-eye view features, key features can be emphasized, making the point cloud data obtained by projecting the bird's-eye view features back to three dimensions more accurate. Furthermore, by performing a simulation rendering operation on the bird's-eye view features, it can be observed that there are only a few peaks with higher responses on specific light rays, indicating that there are objects in the environmental scene. By uniformly sampling to calculate the light features of each light ray, and then calculating the geometric embedding features, it can effectively promote a more accurate and consistent understanding of the three-dimensional environmental scene.

[0064] Next, for step 104, the multi-view image at the current moment and the predicted data at multiple future moments are input into the trained online vector map model, and the online vector map model is used to generate a first bird's-eye view feature for the multi-view image at the current moment, and generate a corresponding second bird's-eye view feature for the predicted data at each future moment, and obtain a global bird's-eye view feature with future compensation information by fusing the second bird's-eye view feature into the first bird's-eye view feature, and generate an online vector map using the global bird's-eye view feature.

[0065] In the embodiment of the present invention, the online vector map model needs to be trained first. When the prediction data is point cloud data, the training method of the online vector map model may include A1-A7:

[0066] A1: Acquire multiple second sample pairs; the second sample pairs include: a second sample multi-view image at the current moment and second sample point cloud data corresponding to multiple future moments as input, and a sample online vector map at the current moment as output; wherein the second sample point cloud data corresponding to multiple future moments are predicted by the world model for the second sample multi-view image at the current moment;

[0067] After the pre-trained world model has been obtained, the second sample point cloud data in the sample pair can be obtained through the world model. Whenever the second sample multi-view image at the current moment is obtained, the world model is used to predict the second sample multi-view image at the current moment to obtain the second sample point cloud data corresponding to multiple future moments, and then the real online vector map at the current moment is obtained as the sample online vector map.

[0068] Similarly, when obtaining the second sample pair, it can be obtained for the same vehicle or for different vehicles.

[0069] A2: Using the second backbone network to extract features from the second sample multi-view image at the current moment, obtaining the second sample multi-view two-dimensional features at the current moment; using the second bird's-eye view feature extractor to project the second sample multi-view two-dimensional features to the bird's-eye view space, obtaining the second sample bird's-eye view features at the current moment;

[0070] Similar to the first backbone network, the second backbone network can sample a Resnet network, such as a Resnet-101 network.

[0071] Similar to the first bird's-eye view feature extractor, the first bird's-eye view feature extractor can be implemented based on convolution, and is used to obtain the first sample bird's-eye view feature through a spatial cross-attention mechanism and a feedforward neural network.

[0072] A3: Using a two-dimensional feature extraction module, two-dimensionally project the second sample point cloud data corresponding to multiple future moments to obtain bird's-eye view point cloud features at multiple future moments;

[0073] Specifically, the second sample point cloud data is voxelized, projected to the ground top-view plane after coordinate transformation to obtain two-dimensional features, and then a convolutional network with a 1×1 convolution kernel is used to extract features. Finally, maximum pooling is used to retain key feature information to obtain the bird's-eye view point cloud features.

[0074] A4: Using the foreground feature extraction module to extract the foreground bird's-eye view features related to the current moment from the bird's-eye view point cloud features of multiple future moments;

[0075] In an embodiment of the present invention, a deformable self-attention mechanism is specifically used to align the bird's-eye view point cloud features at a future moment with the current moment, wherein the aligned bird's-eye view feature query is used to extract foreground bird's-eye view features related to the current moment from the bird's-eye view point cloud features at the future moment.

[0076] Extracting the foreground bird's-eye view features related to the current moment from the bird's-eye view point cloud features of multiple future moments may specifically include:

[0077] For the bird's-eye view point cloud features at each of multiple future moments, the following is performed: based on the vehicle's motion state at the current moment, the motion states of other objects in the vehicle's field of view, the road condition semantics in the vehicle's field of view, and the vehicle's environment, determine whether there is a valid feature point of the second sample bird's-eye view feature at the current moment in the current bird's-eye view point cloud features; if so, extract the valid feature point to obtain a foreground bird's-eye view feature extracted from the current bird's-eye view point cloud features.

[0078] For example, if the point cloud features of the bird's-eye view at a future moment include an empty road, two walls, and a large tree, this background information is not very meaningful at the current moment. However, if there is a car or pedestrian in the future moment, and it also exists at the current moment, and the current moment and the future moment have different states, it will have a certain impact on the current moment and affect the future planning at the current moment. Therefore, it is necessary to extract the foreground information that exists at both the current moment and the future moment.

[0079] Which foreground information is related to the current moment can be trained by pre-marking the future moments. After the foreground feature extraction module is pre-trained, the foreground feature extraction module takes the second sample bird's-eye view feature of the current moment and the bird's-eye view point cloud feature of the future moment as input to output the foreground bird's-eye view feature related to the current moment extracted from the bird's-eye view point cloud feature of the future moment. The foreground feature extraction module can be extracted by the following formula:

[0080]

[0081] Among them, F t bev ' is the foreground bird's-eye view feature related to the current moment extracted from the bird's-eye view point cloud feature at the future time t, F t bev is the point cloud feature of the bird's-eye view at the future moment, Ego is the vehicle motion state, Obj(h) is the motion state of the hth other object in the vehicle's perspective at the current moment, m is the total number of other objects in the vehicle's perspective at the current moment, Sem is the semantics of the road condition in the vehicle's perspective, and Env is the environment in which the vehicle is located.

[0082] A5: Using a feature fusion module, the foreground bird's-eye view features of multiple future moments are fused with the second sample bird's-eye view features of the current moment to obtain a predicted global bird's-eye view feature with future compensation information;

[0083] When performing feature fusion, the foreground bird's-eye view features of multiple future moments and the second sample bird's-eye view features of the current moment can form a feature map of (n+1) layers, and then the features of all layers can be fused into a feature map of one layer through the maximum pooling operation, thereby obtaining a predicted global bird's-eye view feature with future compensation information.

[0084] A6: Use the map decoder to predict the global bird's-eye view feature output and predict the online vector map;

[0085] Among them, the map decoder can be implemented based on the Transformer model.

[0086] A7: Use the sample online vector map and the loss function of the predicted online vector map to complete the training of the online vector map model.

[0087] As described above, after obtaining the trained online vector map model, the online vector map model and the world model can be set on each vehicle. The vehicle only needs to set multiple cameras at intervals to obtain multi-view images in real time using multiple cameras, and then use the world model and the online vector map model to generate an online vector map.

[0088] In an embodiment of the present invention, by pre-training a world model, the computing resources required for training an online vector map can be effectively reduced; and the world model can be used to predict point cloud data at future times, thereby projecting the bird's-eye view features at future times, and integrating the bird's-eye view features at future times into the bird's-eye view features at the current time, thereby compensating the current time for future information, thereby constructing a more accurate online vector map, and laying a solid foundation for subsequent autonomous driving planning and decision-making.

[0089] Please refer to Figure 2 , an embodiment of the present invention provides an online vector map construction device based on a pre-trained world model, the device comprising:

[0090] A first acquisition unit 200 is used to acquire a multi-view image at a current moment;

[0091] A second acquisition unit 202 is used to output prediction data at multiple future moments for the multi-view image at the current moment based on the pre-trained world model;

[0092] The map construction unit 204 is used to input the multi-view image at the current moment and the prediction data at multiple future moments into the trained online vector map model, wherein the online vector map model is used to generate a first bird's-eye view feature for the multi-view image at the current moment, and to generate a corresponding second bird's-eye view feature for the prediction data at each future moment, and to obtain a global bird's-eye view feature with future compensation information by fusing the second bird's-eye view feature into the first bird's-eye view feature, and to generate an online vector map using the global bird's-eye view feature.

[0093] In one embodiment of the present invention, the prediction data is point cloud data; and the training method of the world model includes:

[0094] S1: Acquire multiple first sample pairs based on historical data; the first sample pairs include a first sample multi-view image at time t0 as input and a first sample multi-view image at time t1-t2 as output. n The first sample point cloud data corresponding to each time; n is an integer greater than 1;

[0095] S2: Using the first backbone network to extract features from the first sample multi-view image at time t0, to obtain the first sample multi-view two-dimensional features at time t0;

[0096] S3: using a first bird's-eye view feature extractor to project the first sample multi-view two-dimensional feature into the bird's-eye view space to obtain the first sample bird's-eye view feature at time t0;

[0097] S4: using the imitation rendering module to imitate the rendering operation of the bird's-eye view feature of the first sample in the latent space, and calculating the geometric embedding feature at time t0;

[0098] S5: Use the Transformer module to combine the geometric embedding features at time t0 with the expected ego-vehicle motion conditions at time t1 to predict the geometric embedding features at time t1;

[0099] S6: Using the Transformer module, combined with the predicted t j The geometric embedding features and expected t at the moment j+1 The vehicle motion condition at time t is predicted j+1 The geometric embedding feature at the moment; j is an integer in [1, n-1];

[0100] S7: The predicted t1-t n The geometric embedding features corresponding to each time are used to generate a 3D voxel grid using the projection layer, and the center point of the voxel grid is used as the representative point to generate t1-t n Prediction point cloud data corresponding to each moment;

[0101] S8: Using t1-t n The loss function between the first sample point cloud data and the predicted point cloud data corresponding to each time moment is used to complete the training of the world model.

[0102] In one embodiment of the present invention, S4 specifically includes:

[0103] Based on the bird's-eye view grids corresponding to the sample bird's-eye view features, emitting multiple rays outward from the origin, the multiple rays extending from the origin to multiple corresponding bird's-eye view grids;

[0104] For each ray, the following steps are performed: uniform sampling is performed along the ray, and the bird's-eye view feature of each sampling point is calculated; the ray feature of the ray is calculated based on the bird's-eye view feature of each sampling point and its conditional probability;

[0105] The geometric embedding features are calculated using the light features of each ray and the conditional probability matrix.

[0106] In one embodiment of the present invention, the prediction data is point cloud data; the training method of the online vector map model includes:

[0107] Acquire multiple second sample pairs; the second sample pairs include: a second sample multi-view image at the current moment and second sample point cloud data corresponding to multiple future moments as input, and a sample online vector map at the current moment as output; wherein the second sample point cloud data corresponding to multiple future moments are predicted by the world model for the second sample multi-view image at the current moment;

[0108] Using the second backbone network to extract features from the second sample multi-view image at the current moment, obtaining the second sample multi-view two-dimensional features at the current moment; using the second bird's-eye view feature extractor to project the second sample multi-view two-dimensional features to the bird's-eye view space, obtaining the second sample bird's-eye view features at the current moment;

[0109] Using a two-dimensional feature extraction module, two-dimensionally project the second sample point cloud data corresponding to the multiple future moments respectively to obtain the bird's-eye view point cloud features of the multiple future moments;

[0110] A foreground feature extraction module is used to extract foreground bird's-eye view features related to the current moment from the bird's-eye view point cloud features at multiple future moments;

[0111] A feature fusion module is used to fuse the foreground bird's-eye view features of multiple future moments with the second sample bird's-eye view features of the current moment to obtain a predicted global bird's-eye view feature with future compensation information;

[0112] Using the map decoder to predict the global bird's-eye view feature output, the predicted online vector map is output;

[0113] The online vector map model is trained using the sample online vector map and the loss function for predicting the online vector map.

[0114] In one embodiment of the present invention, extracting the foreground bird's-eye view features related to the current moment from the bird's-eye view point cloud features at multiple future moments includes:

[0115] For the bird's-eye view point cloud features at each of multiple future moments, the following is performed: based on the vehicle's motion state at the current moment, the motion states of other objects in the vehicle's field of view, the road condition semantics in the vehicle's field of view, and the vehicle's environment, determine whether there is a valid feature point of the second sample bird's-eye view feature at the current moment in the current bird's-eye view point cloud features; if so, extract the valid feature point to obtain a foreground bird's-eye view feature extracted from the current bird's-eye view point cloud features.

[0116] In one embodiment of the present invention, the foreground feature extraction module is pre-trained.

[0117] It should be noted that the online vector map construction device based on the pre-trained world model provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the online vector map construction device based on the pre-trained world model provided in the above embodiment and the online vector map construction method embodiment based on the pre-trained world model belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0118] The embodiment of the present application also provides a computer device, please refer to Figure 3 The computer device includes a processor and a memory, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the memory, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the online vector map construction method based on the pre-trained world model provided in the above-mentioned method embodiments.

[0119] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the online vector map construction method based on the pre-trained world model provided in the above-mentioned method embodiments.

[0120] An embodiment of the present application also provides a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the online vector map construction method based on a pre-trained world model described in any of the above embodiments.

[0121] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0122] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0123] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0124] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for constructing an online vector map based on a pre-trained world model, characterized in that: The method comprises: Get the multi-view image at the current moment; Based on the pre-trained world model, it outputs prediction data for multiple future moments based on the multi-view images at the current moment. Inputting the multi-view image at the current moment and the prediction data at multiple future moments into the trained online vector map model, the online vector map model is used to generate a first bird's-eye view feature for the multi-view image at the current moment, and respectively generate a corresponding second bird's-eye view feature for the prediction data at each future moment, and obtain a global bird's-eye view feature with future compensation information by fusing the second bird's-eye view feature into the first bird's-eye view feature, and generate an online vector map using the global bird's-eye view feature; The prediction data is point cloud data; the training method of the world model includes: S1: Acquire multiple first sample pairs based on historical data; the first sample pairs are based on t0-t n The historical data of the moment is obtained. Among the n+1 historical moments, the farthest historical moment is taken as the moment t0, and the nearest n historical moments are taken as the moments t1-t n moment; the first sample pair comprises: a first sample multi-view image at time t0 as input, and, as output t1-t n The first sample point cloud data corresponding to each time; n is an integer greater than 1; S2: Using the first backbone network to extract features from the first sample multi-view image at time t0, to obtain the first sample multi-view two-dimensional features at time t0; S3: using a first bird's-eye view feature extractor to project the first sample multi-view two-dimensional feature into the bird's-eye view space to obtain the first sample bird's-eye view feature at time t0; S4: using the imitation rendering module to imitate the rendering operation of the bird's-eye view feature of the first sample in the latent space, and calculating the geometric embedding feature at time t0; S5: Use the Transformer module to combine the geometric embedding features at time t0 with the expected ego-vehicle motion conditions at time t1 to predict the geometric embedding features at time t1; S6: Using the Transformer module, combined with the predicted t j The geometric embedding features and expected t at the moment j+1 The vehicle motion condition at time t is predicted j+1 The geometric embedding feature at the moment; j is an integer in [1, n-1]; S7: The predicted t1-t n The geometric embedding features corresponding to each time are used to generate a 3D voxel grid using the projection layer, and the center point of the voxel grid is used as the representative point to generate t1-t n Prediction point cloud data corresponding to each moment; S8: Using t1-t n The loss function between the first sample point cloud data and the predicted point cloud data corresponding to each time moment is used to complete the training of the world model.

2. The method according to claim 1, characterized in that Step S4 includes: Based on the bird's-eye view grid corresponding to the sample bird's-eye view feature, emitting multiple rays outward from the origin, the multiple rays extending from the origin to multiple corresponding bird's-eye view grids; For each ray, the following steps are performed: uniform sampling is performed along the ray, and the bird's-eye view feature of each sampling point is calculated; the ray feature of the ray is calculated based on the bird's-eye view feature of each sampling point and its conditional probability; The geometric embedding features are calculated using the light features of each ray and the conditional probability matrix.

3. The method according to claim 1, characterized in that The predicted data is point cloud data; The training method of the online vector map model includes: Acquire multiple second sample pairs; the second sample pairs include: a second sample multi-view image at the current moment and second sample point cloud data corresponding to multiple future moments as input, and a sample online vector map at the current moment as output; wherein the second sample point cloud data corresponding to multiple future moments are predicted by the world model for the second sample multi-view image at the current moment; Using the second backbone network to extract features from the second sample multi-view image at the current moment, obtaining the second sample multi-view two-dimensional features at the current moment; using the second bird's-eye view feature extractor to project the second sample multi-view two-dimensional features to the bird's-eye view space, obtaining the second sample bird's-eye view features at the current moment; Using a two-dimensional feature extraction module, two-dimensionally project the second sample point cloud data corresponding to the multiple future moments respectively to obtain the bird's-eye view point cloud features of the multiple future moments; A foreground feature extraction module is used to extract foreground bird's-eye view features related to the current moment from the bird's-eye view point cloud features at multiple future moments; A feature fusion module is used to fuse the foreground bird's-eye view features of multiple future moments with the second sample bird's-eye view features of the current moment to obtain a predicted global bird's-eye view feature with future compensation information; Using the map decoder to predict the global bird's-eye view feature output, the predicted online vector map is output; The online vector map model is trained using the sample online vector map and the loss function for predicting the online vector map.

4. The method according to claim 3, characterized in that The step of extracting the foreground bird's-eye view features related to the current moment from the bird's-eye view point cloud features at multiple future moments includes: For the bird's-eye view point cloud features at each of multiple future moments, the following is performed: based on the vehicle's motion state at the current moment, the motion states of other objects in the vehicle's field of view, the road condition semantics in the vehicle's field of view, and the vehicle's environment, determine whether there is a valid feature point of the second sample bird's-eye view feature at the current moment in the current bird's-eye view point cloud features; if so, extract the valid feature point to obtain a foreground bird's-eye view feature extracted from the current bird's-eye view point cloud features.

5. The method according to claim 4, characterized in that The foreground feature extraction module is pre-trained.

6. An online vector map construction device based on a pre-trained world model, characterized in that: The device comprises: A first acquisition unit, used to acquire a multi-view image at a current moment; A second acquisition unit is used to output prediction data at multiple future moments for the multi-view image at the current moment based on the pre-trained world model; a map construction unit, configured to input the multi-view image at the current moment and the prediction data at multiple future moments into a trained online vector map model, wherein the online vector map model is configured to generate a first bird's-eye view feature for the multi-view image at the current moment, and to generate a corresponding second bird's-eye view feature for the prediction data at each future moment, and to obtain a global bird's-eye view feature with future compensation information by fusing the second bird's-eye view feature into the first bird's-eye view feature, and to generate an online vector map using the global bird's-eye view feature; The prediction data is point cloud data; the training method of the world model includes: S1: Acquire multiple first sample pairs based on historical data; the first sample pairs are based on t0-t n The historical data of the moment is obtained. Among the n+1 historical moments, the farthest historical moment is taken as the moment t0, and the nearest n historical moments are taken as the moments t1-t n moment; the first sample pair comprises: a first sample multi-view image at time t0 as input, and, as output t1-t n The first sample point cloud data corresponding to each time; n is an integer greater than 1; S2: Using the first backbone network to extract features from the first sample multi-view image at time t0, to obtain the first sample multi-view two-dimensional features at time t0; S3: using a first bird's-eye view feature extractor to project the first sample multi-view two-dimensional feature into the bird's-eye view space to obtain the first sample bird's-eye view feature at time t0; S4: using the imitation rendering module to imitate the rendering operation of the bird's-eye view feature of the first sample in the latent space, and calculating the geometric embedding feature at time t0; S5: Use the Transformer module to combine the geometric embedding features at time t0 with the expected ego-vehicle motion conditions at time t1 to predict the geometric embedding features at time t1; S6: Using the Transformer module, combined with the predicted t j The geometric embedding features and expected t at the moment j+1 The vehicle motion condition at time t is predicted j+1 The geometric embedding feature at the moment; j is an integer in [1, n-1]; S7: The predicted t1-t n The geometric embedding features corresponding to each time are used to generate a 3D voxel grid using the projection layer, and the center point of the voxel grid is used as the representative point to generate t1-t n Prediction point cloud data corresponding to each moment; S8: Using t1-t n The loss function between the first sample point cloud data and the predicted point cloud data corresponding to each time moment is used to complete the training of the world model.

7. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the steps of any one of the methods described in claims 1-5.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

9. A computer program product, characterized in that The method comprises a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5.

Images