Road prediction method and device and vehicle

Abstract

The invention provides a road prediction method, device and system. The method comprises the following steps: acquiring road data, wherein the road data is road condition data of a road acquired by at least one sensor; performing feature extraction on the road data to obtain a first feature; inputting the first feature into a first prediction model to obtain a first road element, the first road element being a road element of the road in a first range; wherein the first prediction model encodes the first feature in the first range to obtain a second feature; encoding the first feature in a second range to obtain a third feature, the second range being smaller than the first range; and decoding based on the second feature and the third feature to obtain the first road element. According to the scheme, the perception distance and prediction accuracy of the road elements are improved, and road element prediction in a larger range is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent driving, and more particularly, to a road prediction method, device and vehicle. BACKGROUND

[0002] Road element prediction is an important task of an advanced driver assistance system (ADAS) and is a key technology for realizing adaptive cruise control (ACC), a lane departure warning system (LDWS) and the like. In research on intelligent vehicles or unmanned vehicles, a vehicle needs to acquire road structure information to determine a drivable area and then realize planning and control of the ego vehicle.

[0003] With the development of deep learning technology, current road element prediction technology mostly constructs a local map in real time based on sensor information at the vehicle end. Specifically, in related technology, a bird’s eye view (BEV) paradigm is mainly used to convert road condition data features acquired by a sensor to a BEV feature space, and road structure is predicted based on the BEV features. However, when road element prediction is performed based on explicit BEV features, only near-end elements in the BEV feature space can be identified, and far-end road elements outside the BEV feature space cannot be effectively identified.

[0004] Therefore, how to realize road prediction in a larger range is a problem to be solved. SUMMARY

[0005] The present application provides a road prediction method, device and system, which can effectively utilize all information collected by a sensor and realize road prediction in a larger range.

[0006] In a first aspect, a road prediction method is provided, and the method comprises: acquiring road data, the road data being road condition data of a road collected by at least one sensor; performing feature extraction on the road data to obtain first features; inputting the first features into a first prediction model to obtain first road elements, the first road elements being road elements of the road in a first range; wherein the first prediction model encodes the first features in the first range to obtain second features; encodes the first features in a second range to obtain third features, the second range being smaller than the first range; and decodes the second features and the third features to obtain the first road elements.

[0007] Based on the above scheme, different coding features corresponding to different ranges are used to avoid the feature truncation problem caused by the implementation of road element prediction in a single range, improve the perception distance and prediction accuracy of the road element, and realize the prediction of road elements in a larger range. In addition, by using multiple encoding methods for feature processing and fusion, the feature points are provided with their feature representations in different spatial perspectives, so that the position information of each feature point can be more comprehensively represented, the information amount of the input features at the decoding end is further expanded, and the prediction accuracy is improved, which provides more development space for other downstream tasks based on the road element prediction results.

[0008] In combination with the first aspect, in some implementations of the first aspect, the decoding based on the second feature and the third feature to obtain the first road element includes: inputting the third feature into a second decoder, the second decoder being configured to decode the input feature in the second range to obtain a second road element, the second road element being a road element of the road in the second range; and inputting the second feature and the second road element into a first decoder, the first decoder being configured to decode the input feature in the first range to obtain the first road element.

[0009] Based on the above scheme, the second feature and the third feature are decoded using two decoders, and the first road element prediction result with a larger range and the second road element prediction result with higher accuracy can be obtained respectively. By inputting the second road element as prior information into the first decoder, the input and output of the first decoder are associated, and the prediction result in a small range is used as a reference for realizing prediction in a large range, which can effectively improve the calculation convergence speed of the first decoder and further improve the prediction accuracy of the first road element.

[0010] In combination with the first aspect, in some implementations of the first aspect, the decoding based on the second feature and the third feature to obtain the first road element includes: inputting the second feature and the third feature into a first decoder, the first decoder being configured to decode the input feature in the first range to obtain the first road element.

[0011] Based on the above scheme, the coding results of multiple encoders are decoded using one decoder, and the decoder can also obtain the potential association of the same feature point in different representation methods based on a larger number of feature information using different encoding representations, so as to further improve the prediction accuracy while reducing the time and resource overhead of using multiple decoders for training and inference.

[0012] In some implementations of the first aspect, the encoding the first feature in the first range to obtain a second feature comprises: inputting the first feature into a first encoder to obtain the second feature, the first encoder being configured to add a position encoding to each feature point in the first feature, the position encoding being configured to describe a position of each feature point in the first feature in the first range in a three-dimensional (3D) space.

[0013] According to the above scheme, since the first encoder adds the position encoding to each feature point in the first feature, decoding based on the second feature can represent all road elements in the road range covered by the road data, thereby achieving prediction of road elements in a larger range.

[0014] In some implementations of the first aspect, the encoding the first feature in the second range to obtain a third feature comprises: inputting the first feature into a second encoder to obtain the third feature, the second encoder being configured to project the first feature to a bird's eye view (BEV) space, the second range being a range corresponding to the BEV space.

[0015] According to the above scheme, the second encoder is configured to convert the first feature to the third feature in the BEV space, thereby presenting the first feature in the second range more intuitively and achieving prediction of road elements in a near range.

[0016] In some implementations of the first aspect, the method further comprises: inputting a query vector into the first decoder and / or the second decoder.

[0017] In some implementations of the first aspect, the adding the position encoding to each feature point in the first feature comprises: splicing each feature point in the first feature and the corresponding position encoding; or superimposing each feature point in the first feature and the corresponding position encoding.

[0018] In some implementations of the first aspect, the position encoding is determined according to a first matrix, the first matrix being a projection transformation matrix corresponding to the sensor.

[0019] In some implementations of the first aspect, the first matrix, each feature point in the first feature, and the position encoding satisfy:

[0020] z ij ·[x ij ,y ij ,1] T =Ti • [x', y', z', 1] T

[0021] wherein, T i denotes a projection transformation matrix corresponding to the i th sensor, j denotes the j th feature point in the first feature, [x ij , y ij , z ij ] denotes the position coordinates of the j th feature point in the first feature collected by the i th sensor, [x', y', z'] denotes the position encoding of the j th feature point, and i and j are positive integers.

[0022] With reference to the first aspect, in some implementations of the first aspect, the sensor comprises at least one of a camera, a laser radar, or a millimeter wave radar; and the road data comprises at least one of image data, video data, point cloud data, or a map containing the road.

[0023] Based on the above scheme, a suitable sensor can be selected according to the specific application scenario of the processing device, thereby improving the flexibility of road prediction.

[0024] With reference to the first aspect, in some implementations of the first aspect, the road element comprises at least one of a lane line, a lane center line, a road, or an obstacle.

[0025] Based on the above scheme, prediction for a single road element can be realized, or common prediction of multiple road elements can also be realized, thereby improving the flexibility of road prediction.

[0026] With reference to the first aspect, in some implementations of the first aspect, the first prediction model is implemented based on a transformation neural network (Transformer) architecture or a convolutional neural network (CNN) architecture.

[0027] With reference to the first aspect, in some implementations of the first aspect, the first range is a road range covered by the road data.

[0028] Based on the above scheme, prediction can be performed according to all information of the road condition data features obtained by the sensor, thereby obtaining a larger range of road element prediction results.

[0029] With reference to the first aspect, in some implementations of the first aspect, the method further comprises: the first prediction model is obtained by training sample road data and first label data, the sample road data comprises road data of a sample road, and the first label data is a road element of the sample road within the first range.

[0030] In some implementations of the first aspect, the method further includes: the first prediction model is trained according to sample road data, first label data and second label data, the sample road data includes road data of a sample road, the first label data is a road element of the sample road within the first range, and the second label data is a road element of the sample road within the second range.

[0031] In some implementations of the first aspect, the first prediction model is trained according to the following manner: the sample road data and an initial query vector are input into a first initial model to obtain a first prediction result, the initial query vector is a vector generated by random initialization, and the first prediction result is a road element prediction result of the sample road within the first range; and parameters of the first initial model and parameters of the initial query vector are adjusted according to a difference between the first prediction result and the first label data, so as to train the first prediction model and the query vector.

[0032] In some implementations of the first aspect, the first prediction model is trained according to the following manner: the sample road data and an initial query vector are input into a first initial model to obtain a second prediction result, the initial query vector is a vector generated by random initialization, and the second prediction result is a road element prediction result of the sample road within the second range; a first prediction result of the sample road within the first range is obtained according to the sample road data and the second prediction result; and parameters of the first initial model and parameters of the query vector are adjusted according to a difference between the first prediction result and the first label data and a difference between the second prediction result and the second label data, so as to train the first prediction model and the query vector.

[0033] In a second aspect, a device for road prediction is provided, which includes an acquisition module, a processing module and a prediction module.

[0034] The acquisition module is configured to acquire road data, the road data being road condition data of a road collected by at least one sensor.

[0035] The processing module is configured to perform feature extraction on the road data to obtain first features.

[0036] The prediction module is configured to input the first features into a first prediction model to obtain first road elements, the first road elements being road elements of the road within a first range.

[0037] The first prediction model encodes the first features to obtain second features in the first range, encodes the first features to obtain third features in a second range, the second range is smaller than the first range, and decodes the second features and the third features to obtain the first road element.

[0038] With reference to the second aspect, in some implementations of the second aspect, the prediction module is configured to input the third features into a second decoder, the second decoder is configured to decode the input features in the second range to obtain the second road element, the second road element being a road element of the road in the second range, and input the second features and the second road element into a first decoder, the first decoder being configured to decode the input features in the first range to obtain the first road element.

[0039] With reference to the second aspect, in some implementations of the second aspect, the prediction module is configured to input the second features and the third features into a first decoder, the first decoder being configured to decode the input features in the first range to obtain the first road element.

[0040] With reference to the second aspect, in some implementations of the second aspect, the prediction module is configured to input the first features into a first encoder to obtain the second features, the first encoder being configured to add a position encoding to each feature point in the first features, the position encoding being used to describe a position of each feature point in the first features in the first range in a 3D space.

[0041] With reference to the second aspect, in some implementations of the second aspect, the prediction module is configured to input the first features into a second encoder to obtain the third features, the second encoder being configured to project the first features to a bird's eye view (BEV) space, and the second range being a range corresponding to the BEV space.

[0042] With reference to the second aspect, in some implementations of the second aspect, the prediction module is further configured to input a query vector into the first decoder and / or the second decoder.

[0043] With reference to the second aspect, in some implementations of the second aspect, the first range is a road range covered by the road data.

[0044] It should be understood that the beneficial effects of the second aspect and the various implementations of the second aspect can refer to the beneficial effects of the first aspect and the various implementations of the first aspect, which will not be repeated here.

[0045] In a third aspect, a device for road prediction is provided, the device comprising: an obtaining module, a training module.

[0046] The obtaining module is configured to obtain sample road data and first label data, the sample road data comprising road data of a sample road, and the first label data comprising road elements of the sample road within a first range;

[0047] The training module is configured to train a first initial model based on the sample road data and the first label data to obtain the first prediction model.

[0048] With reference to the third aspect, in some implementations of the third aspect, the obtaining module is further configured to obtain second label data, the second label data comprising road elements of the sample road within a second range; and the training module is further configured to train the first initial model based on the sample road data, the first label data, and the second label data to obtain the first prediction model.

[0049] With reference to the third aspect, in some implementations of the third aspect, the training module is further configured to input the sample road data and an initial query vector into the first initial model to obtain a first prediction result, the initial query vector being a vector generated by random initialization, and the first prediction result being a road element prediction result of the sample road within the first range; and adjust parameters of the first initial model and parameters of the initial query vector based on a difference between the first prediction result and the first label data to train the first prediction model and the query vector.

[0050] With reference to the third aspect, in some implementations of the third aspect, the training module is further configured to input the sample road data and an initial query vector into the first initial model to obtain a second prediction result, the initial query vector being a vector generated by random initialization, and the second prediction result being a road element prediction result of the sample road within the second range; obtain the first prediction result based on the sample road data and the second prediction result, the first prediction result being a road element prediction result of the sample road within the first range; and adjust parameters of the first initial model and parameters of the query vector based on a difference between the first prediction result and the first label data and a difference between the second prediction result and the second label data to train the first prediction model and the query vector.

[0051] It should be understood that the beneficial effects of the third aspect and the various implementations of the third aspect are the same as those of the first aspect and the various implementations of the first aspect, which will not be repeated here.

[0052] In a fourth aspect, a computing device is provided, comprising a processor and a memory, and optionally, an input / output interface. The processor is configured to control the input / output interface to transmit and receive information, and the memory is configured to store a computer program. The processor is configured to invoke and run the computer program stored in the memory, so as to execute the method in the first aspect or any possible implementation of the first aspect.

[0053] Optionally, the processor can be a general-purpose processor, which can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, or the like. When implemented by software, the processor can be a general-purpose processor, which reads software codes stored in the memory to implement the processor. The memory can be integrated in the processor or exist independently of the processor.

[0054] In a fifth aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so as to execute the method in the first aspect or any possible implementation of the first aspect.

[0055] In a sixth aspect, a chip is provided, which acquires instructions and executes the instructions to implement the method in the first aspect or any possible implementation of the first aspect.

[0056] Optionally, as an implementation, the chip comprises a processor and a data interface. The processor reads instructions stored in the memory through the data interface, and executes the method in the first aspect or any possible implementation of the first aspect.

[0057] Optionally, as an implementation, the chip can further comprise a memory, which stores instructions. The processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to execute the method in the first aspect or any possible implementation of the first aspect.

[0058] In a seventh aspect, a computer program product comprising instructions is provided, which, when executed by a computing device, causes the computing device to execute the method in the first aspect or any possible implementation of the first aspect.

[0059] In an eighth aspect, a computer program product comprising instructions is provided, which, when executed by a computing device cluster, causes the computing device cluster to execute the method in the first aspect or any possible implementation of the first aspect.

[0060] In a ninth aspect, a computer-readable storage medium is provided, including computer program instructions, when the computer program instructions are executed by a computing device, the computing device performs the method in the first aspect and any one of the implementation manners of the first aspect.

[0061] By way of example, such computer-readable storage can include one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and hard drive.

[0062] Optionally, as an implementation manner, the storage medium can be a non-volatile storage medium.

[0063] In a tenth aspect, a computer-readable storage medium is provided, including computer program instructions, when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method in the first aspect and any one of the implementation manners of the first aspect.

[0064] By way of example, such computer-readable storage can include one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), Flash memory, electrically EPROM (EEPROM), and hard drive.

[0065] Optionally, as an implementation manner, the storage medium can be a non-volatile storage medium. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is a schematic diagram of the training phase and the inference phase of the model.

[0067] Figure 2 is a schematic diagram of a system architecture 200 suitable for embodiments of the present application.

[0068] Figure 3 is a functional schematic diagram of a vehicle 300 suitable for embodiments of the present application.

[0069] Figure 4 is a schematic diagram of the BEV feature coverage.

[0070] Figure 5 is a schematic diagram of a road prediction method 500 suitable for embodiments of the present application.

[0071] Figure 6 is a schematic block diagram of a road prediction method 600 suitable for embodiments of the present application.

[0072] Figure 7 is a schematic block diagram of a road prediction method 700 suitable for embodiments of the present application.

[0073] Figure 8 is a schematic block diagram of a road prediction method 800 suitable for embodiments of the present application.

[0074] Figure 9 is a schematic block diagram of a road prediction method 900 suitable for embodiments of the present application.

[0075] Figure 10 is a schematic diagram of a model training method 1000 suitable for a first prediction model provided by embodiments of the present application.

[0076] Figure 11 is a schematic block diagram of a prediction device 1100 provided by embodiments of the present application.

[0077] Figure 12 is a schematic block diagram of another prediction device 1200 provided by embodiments of the present application.

[0078] Figure 13 is an architectural schematic diagram of a computing device 1300 provided by embodiments of the present application.

[0079] Figure 14 is an architectural schematic diagram of a computing device cluster provided by embodiments of the present application.

[0080] Figure 15 is a schematic diagram of a connection between computing devices 1300A and 1300B over a network provided by embodiments of the present application. DETAILED DESCRIPTION

[0081] The technical solutions in the present application will be described below with reference to the accompanying drawings.

[0082] The technical solutions provided in the present application can be applied to the Internet of Vehicles, such as vehicle to everything (V2X), long term evolution-vehicle (LTE-V), vehicle to vehicle (V2V), and the like. For example, the technical solutions can be applied to a vehicle with a driving mobile function, or other devices with a driving mobile function in the vehicle. The other devices include, but are not limited to, a vehicle terminal, a vehicle controller, a vehicle module, a vehicle module, a vehicle component, a vehicle chip, a vehicle unit, a vehicle radar, a vehicle camera, and other sensors. The vehicle can implement the road prediction method provided in the embodiments of the present application through the other devices. Of course, the technical solutions provided in the present application can also be used in other intelligent terminals with a mobile control function except for vehicles, or be arranged in other intelligent terminals with a mobile control function except for vehicles, or be arranged in components of the intelligent terminals. The intelligent terminals can be intelligent transportation equipment, intelligent home equipment, robots, and the like. For example, the intelligent terminals include, but are not limited to, a controller, a chip, a radar, a camera, and other sensors in the intelligent terminals, and other components.

[0083] Specifically, the technical solutions provided in the present application can be an application program, which can be installed or run in a chip or a component of a vehicle, or a smart device such as a mobile phone or a tablet computer on the vehicle. Alternatively, the technical solutions can be a software module, which can be deployed in any electronic control unit (ECU) of the vehicle. Alternatively, the technical solutions can also be a newly added hardware module in the vehicle, which can be configured with related judgment logic or algorithms. For example, the technical solutions can be an ECU in the vehicle, which can perform information transmission with other ECUs or various sensors through an automobile bus, and implement road prediction. The embodiments of the present application do not limit the product form or deployment manner of the measuring device.

[0084] Artificial intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. The research in the field of artificial intelligence includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, AI basic theory, and the like.

[0085] The basic principle of AI is to combine massive data with super strong operation processing capability and intelligent algorithm to establish an AI model for solving a specific problem, so that the AI model can automatically induce and learn potential patterns or features from data, thereby realizing a thinking mode close to that of humans.

[0086] An AI model, i.e., an AI algorithm (or an AI operator), is a general term for a mathematical algorithm constructed based on the principle of artificial intelligence, and is also the basis for solving a specific problem by using AI. According to different specific methods and / or technologies for realizing artificial intelligence, the AI model can also be referred to as a machine learning model, a deep learning model, or a reinforcement learning model.

[0087] For the convenience of understanding the embodiments of the present application, the terms involved in the present application are first explained simply.

[0088] It should be understood that the relevant concept explanation may be limited due to the specific circumstances of the embodiments of the present application, but it does not mean that the present application can only be limited to this specific situation, and there may be differences in the specific circumstances of different embodiments, which are not limited here.

[0089] 1. Neural network

[0090] A neural network is a model, which can be composed of neural units, and a neural unit can refer to an operation unit with X s and intercept 1 as input, and the output of the operation unit can be:

[0091]

[0092] wherein s = 1, 2, … n, n is a natural number greater than 1, W s is the weight of X s , b is the bias of the neural unit. f is an activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network and convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a Relu function. The neural network is a network formed by connecting a plurality of the above single neural units, i.e., the output of one neural unit can be the input of another neural unit, and the input of each neural unit can be connected to the local receptive field of the previous layer, thereby extracting the features of the local receptive field, wherein the local receptive field can be a region composed of a plurality of neural units.

[0093] Based on the above, a neural network can be specifically understood as a network composed of neurons, with input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The input to a neural network module can be a single frame of image or an image sequence. A neural network with many hidden layers is called a deep neural network (DNN). The operation of each layer in a neural network can be expressed using mathematical expressions. To describe it, from a physical perspective, the work of each layer in a neural network can be understood as transforming the input space (the set of input vectors) to the output space (i.e., from the row space to the column space of a matrix) through five operations on the input space. These five operations include: 1. Dimensionality increase / decrease; 2. Magnification / scaling; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are performed by [a], operation 4 is performed by "+b", and operation 5 is performed by "a()". The reason for using the word "space" is that the object being classified is not a single thing, but a class of things, and space refers to the set of all individuals of this class of things. Here, W is the weight matrix of each layer of the neural network, and each value in this matrix represents the weight value of a neuron in that layer. This matrix W determines the spatial transformation from the input space to the output space mentioned above, that is, W of each layer of the neural network controls how the space is transformed. The purpose of training a neural network is to finally obtain the weight matrices of all layers of the trained neural network. Therefore, the training process of a neural network is essentially learning how to control the spatial transformation, more specifically, learning the weight matrix.

[0094] Neural networks can include models such as feedforward neural networks (FNN), convolutional neural networks (CNN), or Transformers, which will be introduced below.

[0095] 2. Feedforward Neural Network

[0096] Feedforward Neural Networks (FNNs) were among the earliest invented simple artificial neural networks. In a feedforward neural network, each neuron belongs to a different layer. Neurons in each layer receive signals from neurons in the previous layer and output signals to the next layer. There is no feedback throughout the network; signals propagate unidirectionally from the input layer to the output layer.

[0097] 3. Convolutional Neural Networks

[0098] A CNN (Convolutional Neural Network) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as performing convolution between the same trainable filter and an input image or a convolutional feature map. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature maps, each composed of rectangularly arranged neural units. Neural units within the same feature map share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of an image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all locations in the image. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0099] Convolutional kernels can be initialized as matrices of random size, and during CNN training, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, thus lowering the risk of overfitting.

[0100] 4. Transformer

[0101] Transformer is a feature extraction network based on an attention mechanism, which includes an encoder and a decoder.

[0102] Encoder: Used to encode an input sequence into a series of consecutive hidden states, which contain semantic information about the input sequence. The encoder consists of multiple identical layers, each including an attention mechanism and a function-native neural network (FNN). The attention mechanism computes a representation for each position in the input sequence, while the FNN further processes these representations.

[0103] Decoder: Used to generate the target sequence based on the encoder's output and the generated partial sequence. The encoder also consists of multiple identical layers. In addition to the attention mechanism and FNN, the decoder also includes an encoder-decoder attention mechanism to associate the encoder's output with the generated partial sequence.

[0104] The attention mechanism is described below: The attention mechanism (also known as attention) occurs between the encoder and decoder (or between the input sequence and the generated sequence). It can quickly extract important features from sparse data, enabling the model to dynamically calculate the correlation between each position in the input sequence and other positions when processing sequence data.

[0105] In simple terms, attention uses the similarity between features (e.g., inner product) as weights to integrate the queried features as the updated values ​​of the current features. Assuming the input is Q(query), and the context is stored as key-value pairs (K(key), V(value)), then the attention mechanism is essentially a mapping function from the query to a series of (key, value) pairs. The Attention function, which implements this mapping, describes the mapping relationship between a query and a series of key-value pairs (key, value). Essentially, the Attention function assigns a weight coefficient to each element in the sequence (which can also be understood as soft addressing). If each element in the sequence is stored in the form of (K, V), then the Attention function performs addressing by calculating the similarity between Q and K. The similarity calculated by Q and K reflects the importance of the extracted V values, i.e., the weights, and then the weighted sum is used to obtain the final feature value.

[0106] Attention calculation mainly consists of three steps. The first step is to calculate the similarity between the query and each key to obtain weights. Common similarity functions include dot product, concatenation, and perceptron. The second step typically involves normalizing these weights using the softmax function (which can normalize the data, resulting in a probability distribution where the sum of all weight coefficients is 1, and also highlight the weights of important elements). Finally, the weights and their corresponding values ​​are weighted and summed to obtain the final feature value. The specific calculation formula is as follows:

[0107]

[0108] Where d is the dimension of the QK matrix.

[0109] Attention includes self-attention and cross-attention. Self-attention can be understood as a special type of attention, where the input to QKV is consistent. The self-attention mechanism in a self-attention model occurs within the input or output sequence, extracting connections between distant words within the same sentence, such as syntactic features (phrase structure). Self-attention is attention extracted based on the feature map itself, providing an effective modeling method for capturing global contextual information through QKV. In contrast, the input to QKV in cross-attention is inconsistent.

[0110] It should be understood that the above description is based only on the most basic structure of Transformer. Its encoder and decoder can be implemented in other ways, or it may include other structures besides encoders and decoders. For specific details regarding Transformer and its derived models, please refer to relevant descriptions in the prior art; this application does not limit them.

[0111] 5. Model training and inference

[0112] In the embodiments of this application, the models used for road prediction are essentially neural networks, or a portion of the model's structure is a neural network. Model application generally includes two stages: training and inference. Model training refers to using a specified initial model to calculate training data, and then adjusting the parameters of the initial model based on the calculation results, so that the model gradually learns certain patterns and acquires specific functions. Once trained and possessing stable functionality, the model can be used for inference. Model inference is the process of using the trained AI model to calculate input data and obtain the predicted inference result.

[0113] For example, the model is trained on a training set containing static road elements from different perspectives to obtain the trained model; the inference stage is used to make road predictions on real unlabeled instances (i.e. real target images to be processed) using the trained model, and the accuracy of the road prediction results is one of the important indicators for measuring the quality of a model's training.

[0114] Figure 1 This is a diagram illustrating the training and inference phases of a model.

[0115] like Figure 1As shown, during the training phase, a training set for the deep learning model needs to be constructed based on the objective. The training set includes multiple training data points, each labeled. The label of a training data point represents the correct answer to a specific question, and the label can indicate the objective of training the deep learning model using the training data. For example, to train a deep learning model that can identify different static road features, the training set can include multiple images of different static road features (i.e., training data). Each image can have a label identifying the type of static road feature it contains, such as lane lines, lane center lines, roads, or obstacles. In this example, the type of static road feature corresponding to each image is the label of that training data.

[0116] When training a deep learning model, training data can be input into the model in batches after parameter initialization. The deep learning model performs calculations (i.e., inference) on the training data to obtain prediction results based on the training data. The prediction results obtained through inference, along with the corresponding labels of the training data, are used as data to calculate the loss based on the loss function. The loss function is used during the model training phase to calculate the difference (i.e., the loss value) between the model's prediction results on the training data and the labels of that training data. The loss function can be implemented using different mathematical functions; commonly used expressions for loss functions include: mean squared error loss function, logarithmic loss function, least squares method, etc.

[0117] The loss value calculated based on the loss function can be used to update the parameters of a deep learning model. Gradient descent is a common method for updating these parameters. Model training is an iterative process where each iteration infers from different training data and calculates the loss value. The goal of these multiple iterations is to continuously update the parameters of the deep learning model and find the parameter configuration that minimizes or stabilizes the loss value of the loss function.

[0118] It should be understood that a loss function is a function that maps the values ​​of a random event or its related random variables to non-negative real numbers to represent the "risk" or "loss" of that random event. In applications, loss functions are often used as learning criteria and associated with optimization problems; that is, the model is solved and evaluated by minimizing the loss function. For example, in machine learning, loss functions are used for parameter estimation of the model, and the loss value obtained based on the loss function can be used to describe the degree of difference between the model's predictions and the actual values. Common loss functions include the mean squared error loss function, the support vector machine (SVM) hinge loss function, and the cross-entropy loss function.

[0119] During the training phase, to improve training efficiency and post-training model performance, it's necessary to set appropriate hyperparameters. Hyperparameters in deep learning models refer to parameters that cannot be obtained through learning from training data or that cannot be changed by training data; they are a concept relative to the parameters in the model. Hyperparameters of deep learning models are typically set manually based on experience or experiments. These hyperparameters include: learning rate, batch size, and network structure hyperparameters (e.g., number of layers (also called depth), interaction methods between layers, number and size of convolutional kernels, activation functions, etc.). Among these, the learning rate, as a hyperparameter, controls the magnitude of parameter weight updates during training, significantly impacting training speed and accuracy.

[0120] like Figure 1 As shown, a trained deep learning model can then be used to perform inference on input data. In the inference phase, data from real-world application scenarios is typically used as input. The trained deep learning model then obtains the inference result. The inference phase is the practical application of the trained deep learning model, enabling the rapid use of AI capabilities to solve specific technical problems.

[0121] The above only introduces the training of the most typical deep learning model as an example. The training of other types of models has slight differences, but the principle is similar. Most of them involve inference on the training data and adjusting the parameters in the model based on the inference results, with the goal of obtaining a parameter combination that makes the model performance stable.

[0122] 6. Feature Fusion

[0123] Feature fusion generates new features by using different features extracted from neural networks, making these new features more effective for classification, recognition, or detection. Feature fusion generally has two methods: concat and add. Concat is a sequential feature fusion method, directly concatenating two features. If the dimensions of two input features x and y are p and q, the dimension of the output feature z is p+q. Add, on the other hand, is a parallel fusion strategy that combines two feature vectors. For input features x and y, it produces a new feature z with the same number of channels. Taking image data as an example, add increases the amount of information in the features describing the image, but the dimensions of the image itself do not increase; only the amount of information in each dimension increases. Concat, however, merges the number of channels, meaning the features describing the image itself increase, but the amount of information in each feature dimension does not increase.

[0124] It should be understood that feature fusion can also be achieved in other ways, and for details, please refer to the relevant descriptions in the prior art. This application does not limit it in this regard.

[0125] 7. BEV Algorithm

[0126] The BEV algorithm aims to project image information collected by multiple sensors into a unified global view space (i.e., using a unified high-dimensional feature space to represent different low-dimensional view features), and then input it into the model for overall inference, ultimately obtaining a simplified two-dimensional view image. This transformation enables the vehicle to obtain an intuitive and spatially consistent environmental perception view, facilitating tasks such as 3D object detection, tracking, map building, and path planning.

[0127] It should be understood that for the specific content of the BEV algorithm and its related algorithms, please refer to the relevant descriptions in the prior art, and this application does not limit them.

[0128] 8. Prior information

[0129] Prior information is information about parameters or models that is known before statistical inference or decision-making. Prior information reflects prior beliefs or assumptions about parameters or models and can be obtained based on past experience, expert opinions, domain knowledge, etc.

[0130] For example, embodiments of this application can use maps as prior information to more comprehensively reflect road conditions and assist vehicles in obtaining more comprehensive static road element information.

[0131] 9. Lane line detection

[0132] Lane detection is a special type of object detection. General object detection and lane detection differ significantly in terms of object features. Specifically, in general object detection, the target object (e.g., cat, dog, face) occupies an area with a certain aspect ratio, making it easy to identify. However, lane lines in lane detection are long and narrow, making it difficult to accumulate enough features to describe the entire lane line, thus making them harder to identify. This places stringent demands on neural networks, making lane detection more challenging than general object detection. In the field of autonomous driving, lane detection is crucial for the accurate localization of autonomous vehicles and its impact on driving decisions. As one of the most important indicators of road surface information, lane lines effectively guide autonomous vehicles to drive within constrained road areas. Real-time detection of lane lines is a vital component of advanced driver assistance systems (ADAS) in autonomous vehicles, aiding in path planning, lane departure warnings, and providing a reference for precise navigation.

[0133] Figure 2 This is a schematic diagram of the system architecture 200 applicable to embodiments of this application.

[0134] like Figure 2 As shown, the data acquisition device 260 is used to collect training data and store the training data in the database 230. The training device 220 trains the target model / rule 201 based on the training data maintained in the database 230.

[0135] The following describes in more detail how the training device 220 obtains a target model / rule 201 based on training data, which can be used to implement the road prediction method provided in the embodiments of this application.

[0136] It should be noted that, in practical applications, the training data maintained in the database 230 may come from the data acquisition device 260 or other devices. The training device 220 may not necessarily train the target model / rule 201 entirely based on the training data maintained in the database 230; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0137] The target model / rule 201 trained by training device 220 can be applied to different systems or devices, for example, applied to Figure 2 The execution device 210 is shown. The execution device 210 can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) device / virtual reality (VR) device, in-vehicle terminal, etc.; the execution device 210 can also be a server or cloud platform, etc. The execution device 210 is equipped with an I / O interface 212 for data interaction with external devices. Users can input data into the I / O interface 212 through the client device 240. Furthermore, the input data can be user-inputted data, data uploaded by the user through a camera device, or data from a database; this application does not limit the scope of the data input.

[0138] The preprocessing module 213 is used to preprocess the input data received by the I / O interface 212. In this embodiment, the preprocessing module 213 can be used to obtain features of the image to be predicted. During the preprocessing of the input data by the execution device 210, or during the calculation module 211 of the execution device 210 performing calculations and other related processes, the execution device 210 can call data, code, etc. in the data storage system 250 for corresponding processing, or store the processed data, instructions, etc., into the data storage system 250. Finally, the I / O interface 212 returns the processing result to the client device 240, thereby providing it to the user.

[0139] It should be noted that the training device 220 can also perform different tasks for different objectives. Specifically, it can generate target models / rules 201 corresponding to different tasks based on different training data or based on the same training data and different labels. The corresponding target models / rules 201 can be used to achieve the above objectives or complete the above tasks, thereby providing users with the required multi-task results.

[0140] It should be understood that Figure 2 This is merely a schematic diagram of a system architecture. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 2 In this context, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed within the execution device 210.

[0141] Figure 3 This is a functional schematic diagram of the vehicle 300 applicable to the embodiments of this application.

[0142] Vehicle 300 may include a sensing system 310, a computing platform 320, and a display device 330. The sensing system 310 may include one or more sensors for sensing information about the environment surrounding the vehicle 300. For example, the sensing system 310 may include a positioning system, which may be a Global Positioning System (GPS), a BeiDou system, or another positioning system. Alternatively, the sensing system 310 may include one or more of the following: an inertial measurement unit (IMU), an accelerometer, a lidar, millimeter-wave radar, ultrasonic radar, and a camera device.

[0143] Some or all of the functions of vehicle 300 can be controlled by computing platform 320. Computing platform 320 may include one or more processors, such as processors 321 to 32n (n being a positive integer). A processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, graphics processing unit (GPU) (which can be understood as a type of microprocessor), or digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field-programmable gate array (FPGA). In reconfigurable hardware circuits, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement some or all of the functions of the aforementioned units. Furthermore, the processor can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing unit (DPU), etc. In addition, the computing platform 320 may also include a memory for storing instructions. Some or all of the processors 321 to 32n can call the instructions in the memory to implement the corresponding functions.

[0144] In vehicles 300, display devices 330 are mainly divided into two categories: the first is in-vehicle displays; the second is projection displays, such as head-up displays (HUDs). In-vehicle displays are physical displays and an important component of in-vehicle infotainment systems. Multiple displays can be installed in the cabin, such as digital instrument cluster displays, central control screens, displays in front of the front passenger (also known as the front-seat passenger), displays in front of the left and right rear passengers, and even the windows can be used as displays. Head-up displays, also known as head-up display systems, are mainly used to display driving information such as speed and navigation on a display device in front of the driver (e.g., the windshield). This reduces driver eye movement time, avoids pupil changes caused by eye movement, and improves driving safety and comfort. HUDs include, for example, combiner-HUD (C-HUD) systems, windshield-HUD (W-HUD) systems, and augmented reality HUD (AR-HUD) systems. It should be understood that HUDs can also evolve into other types of systems as technology progresses, and this application does not limit them.

[0145] The above description of the display device 330 uses an in-vehicle display screen and a projection display screen as examples, but the embodiments of this application are not limited thereto. For example, the display device 330 can also be a light display screen or a projection screen.

[0146] Vehicle 300 can be either an intelligent vehicle or a non-intelligent vehicle; this application embodiment does not limit this. The vehicle is equipped with sensors (including but not limited to: lidar, millimeter-wave radar, camera devices, ultrasonic sensors, global positioning systems, and inertial measurement units). These sensors are used to detect targets around the vehicle, such as trees in green belts, roadside landmarks, lane boundaries, or lane lines. The vehicle can be equipped with one or more sensors, and each type of sensor can be present in one or more instances. The sensors can be installed on the top of the vehicle, specifically in the middle of the top. This application embodiment does not limit the installation location or number of sensors in the vehicle. In this application embodiment, the vehicle can communicate with other objects based on vehicle-to-everything (V2X) wireless communication technology. For example, vehicle-to-vehicle (V2V) wireless communication technology can be used to achieve communication between vehicles. Communication between the vehicle and other objects can be based on high-fidelity wireless communication (e.g., wireless fidelity (Wi-Fi)), 5G mobile communication technology, or future communication networks. For example, 5G can be used to enable communication between vehicles and smart devices (such as smartphones or mobile devices that support location services).

[0147] Vehicle 300 may include ADAS, which uses various sensors on the vehicle to acquire information from the vehicle's surroundings and analyzes and processes the acquired information to achieve functions such as obstacle perception, target recognition, vehicle positioning, path planning, and driver monitoring / alerts, thereby improving the safety, automation, and comfort of driving the vehicle.

[0148] It should be understood that the system architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0149] Figure 4 This is a schematic diagram of the coverage area of ​​BEV features.

[0150] based on Figure 2 and Figure 3 The architecture shown, in related technologies, mostly employs the BEV paradigm to transform the road condition data features acquired by sensors into the BEV feature space when implementing road prediction, and then predicts the road structure based on these BEV features. However, the process of transforming road condition data features into BEV features suffers from feature truncation, such as... Figure 4As shown, only road condition data features within the BEV feature space will affect the prediction results. Therefore, when predicting static road elements based on explicit BEV features, only near-end elements within the BEV feature space can be identified, while far-end static road elements outside the BEV feature space cannot be effectively identified.

[0151] In view of this, embodiments of this application provide a method, apparatus, and vehicle for road prediction, which can effectively utilize all the information collected by sensors to achieve road prediction over a wider range.

[0152] The technical solution provided in this application will be described in detail below with reference to the accompanying drawings.

[0153] Figure 5 This is a schematic diagram of a road prediction method 500 applicable to embodiments of this application.

[0154] It should be understood that Figure 5 The steps or operations of the road prediction method are illustrated, but these steps or operations are merely to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of the application to... Figure 5 The specific numerical values ​​or specific scenarios illustrated. Those skilled in the art will understand based on... Figure 5 The examples given below can obviously be modified or varied in various ways, and such modifications and variations also fall within the scope of the embodiments of this application. The embodiments of this application can also perform other operations or... Figure 5 Variations of various operations within it.

[0155] It should be understood that the embodiments shown below do not particularly limit the specific structure of the execution entity of the method provided in the embodiments of this application. As long as a program containing the code of the method provided in the embodiments of this application can be run to perform road prediction according to the method provided in the embodiments of this application, it is acceptable. For example, the method provided in the embodiments of this application can be executed by a processing device, which can be a computing platform or a component of the computing platform (e.g., a processor, chip, or chip system). The computing platform can be a cloud device, a vehicle, or a terminal device (e.g., an in-vehicle terminal, an aircraft terminal, etc.). Of course, the method provided in the embodiments of this application can also be executed by a system composed of a cloud device and a vehicle.

[0156] Optionally, the method can be processed by the CPU in the computing platform, or by the CPU and GPU together, or it can be processed without a GPU, but by other processors suitable for neural network computing. This application does not impose any restrictions.

[0157] This method can be used in intelligent driving scenarios, such as adaptive cruise control, lane departure warning (LDW), and lane keeping assist (LKA), which involve road prediction. In intelligent driving scenarios, the road prediction method provided in this application can acquire input data through sensors on the vehicle and obtain road features from the input data, thereby realizing the aforementioned adaptive cruise control, LDW, or LKA.

[0158] like Figure 5 As shown, method 500 may include the following steps:

[0159] S501: Acquire road data, which is road condition data collected by at least one sensor.

[0160] It should be noted that the sensors include at least one of the following: a camera, a lidar, and a millimeter-wave radar. Correspondingly, the road data may also include at least one of the following: image data, video data, point cloud data, and a map containing the road.

[0161] For example, the road data can be image data and / or video data collected by an in-vehicle surround-view camera, point cloud data collected by LiDAR and / or millimeter-wave radar, and the map data can be a navigation map downloaded from the cloud or obtained from an external device, or a self-constructed map generated by a processing device based on historical data collection results. This application does not limit the source of the road data or the type of sensor.

[0162] Optionally, when the sensor acquires the road data, factors such as spatial interference and road bumps may cause the data collected by the sensor to be inaccurate. Therefore, the road data collected by the sensor can also be preprocessed. For example, the preprocessing may include noise reduction, derotation, alignment, downsampling, etc.

[0163] Optionally, road data collected by different types of sensors can be applied to different application scenarios.

[0164] For example, when the road data consists of image data and / or video data acquired by a camera, method 500 can be applied to a purely vision-based system; when the road data also includes point cloud data acquired by radar, method 500 can be applied to a purely perception-based system; and when the road data also includes map data, method 500 can be applied to a navigation system. In other words, when the road data is acquired by multiple types of sensors, method 500 can be applied to a multimodal system. The specific combination of sensors and the application scenarios applicable to the embodiments of this application are not limited herein.

[0165] In this case, appropriate sensors can be selected based on the specific application scenario of the processing equipment, thereby improving the flexibility of road prediction.

[0166] S502: Extract features from the road data to obtain the first feature.

[0167] Optionally, the process of feature extraction from the road data can be implemented by a feature encoder.

[0168] Specifically, the feature encoder can be a single encoder or an encoder group consisting of multiple encoders. When the feature encoder consists of multiple encoders, the parameters between different encoders can be independent, partially shared, or fully shared. Furthermore, the feature encoder can be based on a CNN architecture, a Transformer architecture, or other existing network architectures or combinations thereof. This application does not limit the number or specific structure of the encoders included in the feature encoder in practical applications.

[0169] S503: Input the first feature into the first prediction model to obtain the first road element.

[0170] Wherein, the first road element is the road element within the first range of the road.

[0171] Optionally, the first range is the range of roads covered by the road data.

[0172] Specifically, when the road data does not include a map containing the road, the first range is the road range covered by the road data collected by the sensor. When the road data includes a map of the road, the first range is the union of the road range #1 covered by the road data collected by the sensor and the road range #2 covered by the map (i.e., when the road range #2 covered by the map is greater than the road range #1 covered by the road data collected by the sensor, the first range is the road range #2 covered by the map).

[0173] Optionally, the road elements may include at least one of the following: lane lines, lane center lines, roads, and obstacles.

[0174] Furthermore, the first road element can be represented by a scalar segmentation diagram (e.g., lane line segmentation diagram, lane centerline segmentation diagram, road segmentation diagram, or obstacle segmentation diagram), or by a vector point set (e.g., an instance composed of a series of points representing a lane). This application does not limit the representation form of the first road element.

[0175] Optionally, the first prediction model is implemented based on a Transformer architecture or a CNN architecture. Furthermore, the first prediction model can be based on a single-task architecture to predict a single road element, or it can be based on a multi-task architecture to predict multiple road elements. This application does not limit the specific architecture of the first prediction model.

[0176] In this scenario, predictions can be made for a single road element, or multiple road elements can be predicted together, thus enhancing the flexibility of road prediction.

[0177] In one possible implementation, the first prediction model encodes the first feature within a first range to obtain a second feature; encodes the first feature within a second range to obtain a third feature, the second range being smaller than the first range; and decodes based on the second feature and the third feature to obtain the first road element.

[0178] In this context, employing encoding features corresponding to different ranges avoids the feature truncation problem caused by predicting road elements within a single range, improving the perception distance and prediction accuracy of road elements, and enabling prediction of road elements over a wider range. Furthermore, feature processing and fusion through multiple encoding methods provide feature representations of feature points from different spatial perspectives, thus more comprehensively representing the location information of each feature point. This further expands the information content of the input features at the decoding end, contributing to improved prediction accuracy and providing more development space for other downstream tasks based on road element prediction results. When the first range is the road area covered by the road data, prediction can be performed based on all information of the road condition data features acquired by the sensors, thereby obtaining road element prediction results over a wider range.

[0179] Optionally, the first prediction model may include a first encoder, which adds a positional encoding to each feature point in the first feature. The positional encoding describes the position of each feature point in the first feature within a first range in 3D space. Specifically, the first feature is input into the first encoder to obtain the second feature. Adding a positional encoding to each feature point in the first feature can also be understood as performing feature fusion on each feature point in the first feature and its corresponding positional encoding.

[0180] For example, feature fusion can be performed on each feature point in the first feature and its corresponding positional code using the concat or add method. For instance, the first feature and positional code can be concatenated; or, the values ​​of the first feature and positional code in each dimension can be merged. This application does not limit the fusion method of the first feature and positional code.

[0181] In this case, since the first encoder adds positional encoding to each feature point of the first feature, decoding based on the second feature can characterize all road elements within the road range covered by the road data, thereby enabling prediction of road elements at a greater distance.

[0182] Optionally, the first feature can be directly used as input to the first encoder, or a subset of the first features can be used as input to the first encoder. That is, the first feature can be selected before it is applied to the first encoder.

[0183] For example, the required features can be determined based on business needs. For instance, when road data is collected by sensors from multiple different perspectives, if it is necessary to extend the forward sensing distance, features representing the forward perspective (such as forward telephoto camera features or forward LiDAR features) can be selected from the first features. If it is necessary to extend the lateral sensing distance, features representing the forward perspective (such as lateral camera features) can be selected from the first features.

[0184] Optionally, the location encoding can be determined based on a first matrix, which is the projection transformation matrix corresponding to the sensor. That is, road data collected by different sensors have different perspectives. The projection transformation matrix of each sensor can be determined based on the corresponding parameters of the sensor (e.g., the intrinsic and extrinsic parameters of the camera, where the intrinsic parameters include focal length, optical center, or distortion, and the extrinsic parameters include rotation or translation, etc.). The road data from different perspectives can be transformed into the same 3D coordinate space through the projection transformation matrix corresponding to each sensor.

[0185] For example, when the sensor is radar, the road data is radar point cloud data. Therefore, the projection transformation matrix can rotate and translate features from different sensors so that the different feature points are located in the same 3D coordinate space. The specific method for determining the projection transformation matrix is ​​not limited in the embodiments of this application.

[0186] Optionally, each feature point in the first matrix and the first feature, and its corresponding position encoding, satisfy:

[0187] z ij ·[x ij ,y ij ,1] T=T i ·[x',y',z',1] T

[0188] Among them, T i Let x represent the projection transformation matrix corresponding to the i-th sensor, j represent the j-th feature point in the first feature, [x ij ,y ij ,z ij [x', y', z'] represents the position coordinates of the j-th feature point collected by the i-th sensor in the first feature, and [x', y', z'] represents the position code of the j-th feature point, where i and j are positive integers.

[0189] It should be noted that when the position coordinates of the j-th feature point acquired by the i-th sensor only include [x... ij ,y ij When (e.g., the i-th sensor is a camera that can only acquire planar data), road data can be processed through depth estimation to obtain z for the j-th feature point. ij The estimated value can also be determined by pre-setting a suitable z for the j-th feature point. ij Values. This application for z ij The process of determining it is not limited.

[0190] Optionally, the first prediction model may include a second encoder, which is used to project the first feature onto the bird's-eye view BEV space, and the second range is the range corresponding to the BEV space.

[0191] Specifically, the first feature is input into the second encoder to obtain the third feature.

[0192] The second encoder can be based on a CNN architecture, a Transformer architecture (e.g., BEVformer or BEVformer++), or other existing network architectures or combinations thereof. For the specific structure of the second encoder and the process of obtaining the third feature through the second encoder, please refer to relevant descriptions in the prior art; this application does not limit the scope of the embodiments.

[0193] In this case, the first feature is converted into the third feature in the BEV space by the second encoder, so that the first feature is presented more intuitively in the second range, and the prediction of near-end road elements is realized.

[0194] Optionally, the first prediction model may include a first decoder, which is used to decode the input features within the first range to obtain the first road features.

[0195] Specifically, the second feature and the third feature are input into the first decoder to obtain the first road element.

[0196] Optionally, the first decoder can be based on a Transformer decoder architecture, or it can be based on a combination of linear regression prediction and non-maximum suppression (NMS), or it can be based on other existing network architectures or combinations thereof. The specific structure of the first decoder and the process of obtaining the first path element through the first decoder are not limited in the embodiments of this application.

[0197] In this scenario, a single decoder is used to decode the encoding results of multiple encoders. The decoder can also obtain the potential associations of the same feature points in different representations based on a larger amount of feature information using different encoding representations. This reduces the time and resource overhead of training and inference using multiple decoders while further improving prediction accuracy.

[0198] Optionally, the first prediction model may include a first decoder and a second decoder.

[0199] The first decoder is used to decode the input features within the first range to obtain the first road element. The second decoder is used to decode the input features within the second range to obtain the second road element, wherein the second road element is the road element of the road within the second range.

[0200] Optionally, the third feature can be input into the second decoder to obtain the second road element; then the second feature and the second road element can be input into the first decoder to obtain the first road element.

[0201] In this scenario, using two decoders to decode the second and third features respectively yields prediction results for the first road element with a wider range and prediction results for the second road element with higher accuracy. Furthermore, inputting the second road element as prior information into the first decoder links its input and output. Using the prediction result within a smaller range as a reference for achieving a larger range of predictions effectively improves the computational convergence speed of the first decoder, further enhancing the prediction accuracy of the first road element.

[0202] For example, the second road element output by the second decoder can be processed into an m×n vector (m and n are positive integers, which can be determined according to the specific structure of the first decoder, for example, 48×256), as prior information for predicting the first road element, indicating the predicted second road element and its corresponding location information. Thus, the first decoder can determine some road elements in advance based on the correspondence between the second feature and the prior information.

[0203] Optionally, the second decoder can be based on a Transformer decoder architecture, or it can be based on a combination of linear regression prediction and NMS, or it can be based on other existing network architectures or combinations thereof. This application does not limit the specific structure of the second decoder or the process of obtaining the second path elements through the second decoder.

[0204] Optionally, in the above embodiments, the inputs of the first decoder and / or the second decoder may further include a query vector.

[0205] Specifically, the query vector is obtained through training from an initial query vector generated using a random initialization method. During training, the parameters of the initial query vector can be adjusted based on the gradient of the predicted loss values ​​of the first road element and / or the second road element, ultimately updating the query vector.

[0206] For example, the query vector can be an m×n vector (m and n are positive integers, which can be determined according to the specific structure of the first decoder, for example, it can be 48×256, that is, it has the same shape as the second road element as prior information mentioned above).

[0207] Furthermore, the representation of the second road element can be based on a scalar segmentation map (e.g., lane line segmentation map, lane centerline segmentation map, road segmentation map, or obstacle segmentation map), or it can be based on a vector point set (e.g., an instance representing a lane composed of a series of points). This application does not limit the representation format of the second road element.

[0208] In another possible implementation, the first prediction model encodes the first feature within the first range to obtain a second feature; encodes the first feature within the second range to obtain a third feature, the second range being smaller than the first range; and decodes based on the second feature and the third feature to obtain the first road element and the second road element.

[0209] Optionally, the first prediction model may include a first decoder and a second decoder.

[0210] The first decoder is used to decode the input features within the first range to obtain the first road element. The second decoder is used to decode the input features within the second range to obtain the second road element, which is the road element of the road within the second range. For details regarding the first decoder, second decoder, first road element, and second road element, please refer to the preceding description; further details are not provided here.

[0211] Specifically, the second feature can be input into the first decoder to obtain the first road element; the third feature can be input into the second decoder to obtain the second road element.

[0212] In this case, the first prediction model can output prediction results for road elements in different ranges, and the calculation process of different prediction results is decoupled, reducing the dependency between the first decoder and the second decoder and improving the structural flexibility of the first prediction model.

[0213] Optionally, the input to the first decoder and / or the second decoder may further include a query vector. The specific content of the query vector can be found above and will not be repeated here. Optionally, the first prediction model can also be applied to road element prediction within a wider range. The embodiments of this application do not limit the number of road element predictions within different ranges that the first prediction model can achieve. Compared to a road element prediction scheme that only includes two ranges, road element prediction within a wider range can expand the predictable range to some extent, but it also brings greater overhead.

[0214] For example, the first prediction model can also encode the first feature within a third range to obtain a fourth feature, wherein the third range is smaller than the second range.

[0215] Optionally, to correspond to a wider range of road element predictions, the first prediction model may also include more encoders and decoders. Therefore, the embodiments of this application do not limit the number of encoders and decoders included in the first prediction model.

[0216] For example, the first prediction model may further include a third encoder and a third decoder. The third encoder may be used to add positional encoding to each feature point in the first feature, or it may be used to project the first feature onto the bird's-eye view BEV space. For specific implementations of the third encoder, refer to relevant existing technologies; this application does not limit its scope. The third decoder is used to decode the input features within the third range to obtain the third road element, which is the road element within the third range. The third decoder may be based on a Transformer decoder architecture, or it may be based on a combination of linear regression prediction and NMS, or it may be based on other existing network architectures or combinations thereof. This application does not limit the specific structure of the third decoder or the process of obtaining the third road element through the third decoder.

[0217] Optionally, corresponding to the case where the first prediction model includes more encoders and decoders, the third feature and the third road element can also be input into the second decoder to obtain the second road element. That is, the road prediction results generated in a smaller range can be used as input features to achieve road prediction in a larger range, increasing the prior information for road element prediction in a larger range, thereby achieving more accurate road element prediction in a larger range.

[0218] For ease of description, the following methods 600-700 ( Figures 6-7 This description only illustrates the implementation of road element prediction in two ranges (near-end road element prediction and far-end road element prediction) and does not constitute a limitation on the number of different ranges of road element prediction that can be implemented in the embodiments of this application.

[0219] Figure 6 This is a schematic block diagram of a road prediction method 600 applicable to embodiments of this application.

[0220] The feature encoder 610 is used to extract features from road data to obtain the first feature.

[0221] The road data refers to real-time road condition data collected by at least one sensor.

[0222] Optionally, the road data may include at least one of the following: image data, video data, point cloud data, and a map containing the road.

[0223] For example, the road data can be image data and / or video data collected by an in-vehicle surround-view camera, point cloud data collected by LiDAR and / or millimeter-wave radar, and the map data can be a navigation map downloaded from the cloud or obtained from an external device, or a self-constructed map generated by a processing device based on historical data collection results. This application does not limit the source of the road data.

[0224] Optionally, when the sensor acquires the road data, factors such as spatial interference and road bumps may cause the data collected by the sensor to be inaccurate, thus making it unsuitable for direct application to the feature encoder 610. Therefore, the road data collected by the sensor can also be preprocessed. For example, the preprocessing may include noise reduction, derotation, alignment, downsampling, etc.

[0225] Optionally, road data collected by different types of sensors can be applied to different application scenarios.

[0226] For example, when the road data is image data and / or video data acquired by a camera, the method 600 can be applied to a purely vision-based system; when the road data also includes point cloud data acquired by radar, the method 600 can be applied to a purely perception-based system; when the road data also includes map data, the method 600 can be applied to a navigation system. In other words, when the road data is acquired by multiple types of sensors, the method 600 can be applied to a multimodal system. This application does not limit the types of sensors or their specific combinations.

[0227] It should be noted that the feature encoder 610 can be a single encoder or an encoder group consisting of multiple encoders. The feature encoder 610 can be based on a CNN architecture, a Transformer architecture, or other existing network architectures or combinations thereof. This application does not limit the number or specific structure of the encoders included in the feature encoder 610 in practical applications.

[0228] In addition, when the feature encoder 610 is composed of multiple encoders, the parameters between different encoders can be independent of each other, or they can be partially or completely shared.

[0229] BEV feature encoder 611 is used to project the output (first feature) of feature encoder 610 onto BEV space, the range of which is smaller than the range of roads covered by road data.

[0230] It should be noted that the BEV feature encoder 611 can be based on a CNN architecture, a Transformer architecture (e.g., BEVformer or BEVformer++), or other existing network architectures or combinations thereof. The specific structure of the feature encoder 610 and the specific method of implementing feature fusion are not limited in the embodiments of this application.

[0231] Optionally, the BEV feature encoder 611 can be used to perform feature fusion on the first feature, or it can be understood as being used to process features obtained through multiple sensors. Each new feature point after being processed by the BEV feature encoder 611 is determined by all or some of the feature points in the first feature.

[0232] The near-end road element decoder 620 is used to decode the output (fused features) of the BEV feature encoder 611 to obtain near-end road elements.

[0233] It should be noted that, since the range corresponding to the BEV space is smaller than the road range covered by the road data, the decoding based on the output of the BEV feature encoder 611 can only characterize the road elements (i.e., near-end road elements) within the range corresponding to the BEV space.

[0234] It should be understood that the input to the near-end road feature decoder 620 may also include a query vector. Specifically, the query vector is obtained through training from an initial query vector generated using a random initialization method. During training, the parameters of the initial query vector can be adjusted according to the gradient of the prediction loss value of the near-end road features, ultimately updating the query vector.

[0235] The near-end road feature decoder 620 can be based on a Transformer decoder architecture, or it can be based on a combination of linear regression prediction and non-maximum suppression (NMS), or it can be based on other existing network architectures or combinations thereof. This application does not limit the specific structure of the near-end road feature decoder 620 or the specific method of implementing near-end road feature prediction.

[0236] Furthermore, the near-end road features can be represented by scalar segmentation maps (e.g., lane line segmentation maps, lane centerline segmentation maps, road segmentation maps, or obstacle segmentation maps), or by vector point sets (e.g., instances composed of point columns representing lanes). This application does not limit the representation format of the output results.

[0237] The location encoding and feature fusion encoder 612 is used to add each feature point in the first feature to the location encoding, which is used to describe the position of each feature point in the first feature within the road area covered by the road data in 3D space.

[0238] Specifically, adding a positional code to each feature point in the first feature can also be understood as performing feature fusion on each feature point in the first feature and its corresponding positional code. For example, the first feature and positional code can be fused using the `concat` or `add` method, as described in the relevant description in the preceding terminology section. This application does not limit the fusion method of the first feature and positional code in its embodiments.

[0239] Optionally, the first feature can be directly used as input to the position encoding and feature fusion encoder 612, or a portion of the first feature can be used as input to the position encoding and feature fusion encoder 612. That is, the first feature can be selected before applying it to the position encoding and feature fusion encoder 612.

[0240] For example, the required features can be determined based on business needs. For instance, when road data is collected by sensors from multiple different perspectives, if it is necessary to extend the forward sensing distance, features representing the forward perspective (such as forward telephoto camera features or forward LiDAR features) can be selected from the first features. If it is necessary to extend the lateral sensing distance, features representing the forward perspective (such as lateral camera features) can be selected from the first features.

[0241] In one possible implementation, the position encoding and feature fusion encoder 612 can determine the position encoding of each feature point in the first feature based on a first matrix, where the first matrix is ​​the projection transformation matrix corresponding to the sensor. That is, road data collected by different sensors have different perspectives, and the projection transformation matrix of each sensor can be determined based on the corresponding parameters of the sensor (e.g., the intrinsic and extrinsic parameters of the camera, where the intrinsic parameters include focal length, optical center, or distortion, and the extrinsic parameters include rotation or translation, etc.). The road data from different perspectives are then transformed into the same 3D coordinate space through the projection transformation matrix corresponding to each sensor.

[0242] For example, when the sensor is radar, the road data is radar point cloud data, so the projection transformation matrix can rotate and translate the features from different sensors so that the different feature points are located in the same 3D coordinate space.

[0243] For example, each feature point in the first matrix and the first feature, and its corresponding position encoding, may also satisfy:

[0244] z ij ·[x ij ,y ij ,1] T =T i ·[x',y',z',1] T

[0245] Among them, T i Let x represent the projection transformation matrix corresponding to the i-th sensor, j represent the j-th feature point in the first feature, [x ij ,y ij ,z ij [x',y',z'] represents the position coordinates of the j-th feature point collected by the i-th sensor in the first feature, and [x',y',z'] represents the position code of the j-th feature point, where i and j are both positive integers.

[0246] It should be noted that when the position coordinates of the j-th feature point acquired by the i-th sensor only include [x... ij ,y ij When (e.g., the i-th sensor is a camera that can only acquire planar data), road data can be processed through depth estimation to obtain z for the j-th feature point. ij The estimated value can also be determined by pre-setting a suitable z for the j-th feature point. ij Values. This application for z ij The process of determining it is not limited.

[0247] Taking a camera as an example, assuming the camera's installation parameters are fixed, in actual production, it cannot be guaranteed that the camera installation position on all vehicles will be completely without deviation. At the same time, it cannot be guaranteed that the camera will not have positional deviations due to loosening during vehicle operation. Therefore, during the training of the position encoding and feature fusion encoder 612, it is necessary to supplement the small deviations of different camera positions to enhance the robustness of the model, so that the model can still output the correct position encoding result when there is a reasonable range of deviation between the actual installation position and the set installation position of the camera.

[0248] The remote road element decoder 621 is used to decode the output of the location encoding and feature fusion encoder 612 to obtain the remote road elements.

[0249] It should be noted that since the location encoding and feature fusion encoder 612 adds location encoding to each feature point, decoding based on the output of the location encoding and feature fusion encoder 612 can characterize all road elements (i.e., remote road elements) within the road range covered by the road data.

[0250] The remote road feature decoder 621 can be based on a Transformer decoder architecture, or it can be based on a combination of linear regression prediction and non-maximum suppression (NMS), or it can be based on other existing network architectures or combinations thereof. This application does not limit the specific structure of the remote road feature decoder 621 or the specific method for implementing remote road feature prediction.

[0251] Furthermore, the distant road elements can be represented by scalar segmentation maps (e.g., lane line segmentation maps, lane centerline segmentation maps, road segmentation maps, or obstacle segmentation maps), or by vector point sets (e.g., instances composed of point columns representing lanes). This application does not limit the representation format of the output results.

[0252] Optionally, the input to the far-end road feature decoder 621 may also include near-end road features. In other words, the output of the near-end road feature decoder 620 (the prediction result of the near-end road features) can be used as prior information to input the far-end road feature decoder 621, thereby improving the computational convergence speed of the far-end road feature decoder 621 and improving the prediction accuracy of the far-end road features.

[0253] For example, the output of the near-end road element decoder can be processed into an m×n vector (e.g., 48×256) as prior information. This prior information can be used to indicate the road elements within the predicted BEV space and their corresponding location information. The far-end road element decoder 621 can pre-determine some road elements based on the correspondence between the output of the location encoding and feature fusion encoder 612 and the prior information.

[0254] Optionally, the road features predicted by the near-end road feature decoder 620 and the far-end road feature decoder 621 include at least one of the following: lane lines, lane center lines, roads, and obstacles. That is, method 600 can predict a single road feature based on a single-task architecture, or it can predict multiple road features based on a multi-task architecture.

[0255] Figure 7 This is a schematic block diagram of a road prediction method 700 applicable to embodiments of this application.

[0256] Similar to Method 600 above, Method 700 also includes a feature encoder, a BEV feature encoder, a near-end road element decoder, a location encoder and feature fusion encoder, and a far-end road element decoder. For the specific content of the above modules, please refer to the description of the corresponding modules in Method 600. This application will not repeat it here.

[0257] It should be noted that, unlike method 600, Figure 7 In the method 700 shown, near-end road feature prediction and far-end road feature prediction are two independent branches. That is, the prediction results of near-end road features are no longer used as prior information for far-end road feature prediction, and the prediction results of near-end road features do not affect the prediction results of far-end road features.

[0258] Optionally, the input to the far-end road feature decoder 721 may also include a query vector. Specifically, the query vector is obtained through training from an initial query vector generated using a random initialization method. During training, the parameters of the initial query vector can be adjusted according to the gradient of the prediction loss value of the far-end road features, ultimately updating the query vector. For example, the query vector may be an m×n vector (e.g., 48×256, i.e., having the same shape as the near-end road feature prediction result used as prior information in method 600).

[0259] It should be understood that the following methods 800-900 ( Figures 8-9 This paper illustrates two methods for implementing remote road feature prediction with different numbers of encoders and / or decoders, and does not constitute a limitation on the number of encoders and / or decoders used to implement remote road feature prediction in the embodiments of this application.

[0260] Figure 8 This is a schematic block diagram of a road prediction method 800 applicable to embodiments of this application.

[0261] Similar to Method 700 described above, Method 700 also includes a feature encoder, a location encoder and feature fusion encoder, and a remote road element decoder. For the specific contents of the above modules, please refer to the descriptions of the corresponding modules in Method 600 and Method 700. This application will not elaborate on them here.

[0262] It should be noted that, unlike method 700, Figure 8 The method shown in 800 only predicts distant road features.

[0263] Figure 9 This is a schematic block diagram of a road prediction method 900 applicable to embodiments of this application.

[0264] Similar to Method 600 above, Method 900 also includes a feature encoder, a BEV feature encoder, a location encoder and feature fusion encoder, and a remote road element decoder. For the specific contents of the above modules, please refer to the description of the corresponding modules in Method 600. This application will not repeat them here.

[0265] It should be noted that, unlike method 600, Figure 9In the method 900 shown, the output of the BEV feature encoder 911 and the output of the position encoder and feature fusion encoder 912 are used together as input to the remote road element decoder 920. That is, only one decoder is used to decode the encoding results of multiple encoders, thereby increasing the number of features. This approach can improve prediction accuracy while reducing the time and resource overhead of using multiple decoders for training and inference.

[0266] Optionally, the input to the remote road feature decoder 920 may further include a query vector. Specifically, the query vector is obtained through training from an initial query vector generated using a random initialization method. During training, the parameters of the initial query vector can be adjusted according to the gradient of the prediction loss value of the remote road features, and finally the query vector is updated. The road prediction method provided by the embodiments of this application has been described above. The training process of the first prediction model provided by the embodiments of this application is described below. The training method of the first prediction model can be executed by a training device for the first prediction model. The training device for the first prediction model can be an image processing device (such as a cloud service device or a user device with sufficient computing power to execute the training method of the target neural network), or a system composed of a cloud service device and a user device. Exemplarily, the training method can be performed by... Figure 2 Training equipment 220 and / or Figure 3 The computing platform 320 in the middle is executed.

[0267] Optionally, the training method can be processed by the CPU, or by both the CPU and GPU, or it can use other processors suitable for neural network computing instead of a GPU. This application does not impose any restrictions.

[0268] Figure 10 This is a schematic diagram of a model training method 1000 applicable to the first prediction model provided in the embodiments of this application.

[0269] Method 1000 includes the following steps:

[0270] S1001: Obtain sample road data and first label data.

[0271] The sample road data includes road data of sample roads, and the first label data is the road elements of the sample roads within a first range.

[0272] The training device can collect sample road data through sensors (such as cameras, radar, etc.), obtain sample road data from a database, or receive sample road data sent by other devices. This application does not limit the method of obtaining sample road data.

[0273] When the first prediction model needs to be trained, the training device can acquire a batch of training samples, i.e., sample road data used for training. Among them, the first label data corresponding to the first sample data is known.

[0274] The training process for the first prediction model can be implemented in the following ways:

[0275] Method 1:

[0276] S1002: Based on the sample road data and the first label data, train the first initial model to obtain the first prediction model.

[0277] In one possible implementation, training the first prediction model on the first initial model based on the sample road data and the first label data includes: inputting the sample road data and the initial query vector into the first initial model to obtain a first prediction result for the road within a first range, wherein the initial query vector is a randomly initialized vector; and adjusting the parameters of the first initial model and the parameters of the initial query vector based on the difference between the first prediction result and the first label data to train the first prediction model and the query vector.

[0278] For details on obtaining the first prediction result, please refer to the aforementioned explanation. Figures 5 to 9 The relevant descriptions in the document are not repeated here.

[0279] Optionally, adjusting the parameters of the first initial model and the initial query vector based on the difference between the first prediction result and the first label data to train the first prediction model and the query vector may include: after obtaining the first prediction result, calculating the first prediction result and the first label data using a preset target loss function to obtain a first target loss, which indicates the difference between the first prediction result and the first label data. After obtaining the first target loss, the parameters of the first initial model can be updated based on the first target loss, and the updated first initial model can be trained using the next batch of training samples until the model training conditions are met (e.g., the first target loss converges, etc.), thereby obtaining the trained first prediction model.

[0280] Optionally, the training process also involves a query vector. An initial query vector is generated through random initialization. This initial query vector is also trained during the continuous updating of the first initial model parameters, resulting in a trained query vector. This trained query vector can be understood as the query vector used during inference. Figures 5 to 9 The query vector in the illustrated embodiment.

[0281] Method 2:

[0282] S1003: Get the second tag data.

[0283] The second label data refers to the road elements of the sample road within the second range.

[0284] S1004: Based on the sample road data, the first label data, and the second label data, train the first initial model to obtain the first prediction model.

[0285] In one possible implementation, step S1004 may include: inputting sample road data and an initial query vector into a first initial model to obtain a first prediction result for the road within a first range and a second prediction result for the road within a second range; adjusting the parameters of the first initial model and the parameters of the initial query vector based on the difference between the first prediction result and the first label data, and the difference between the second prediction result and the second label data, to train and obtain a first prediction model and a query vector.

[0286] The process for obtaining the first and second prediction results can be found in the preceding text. Figures 5 to 9 The relevant descriptions in the document are not repeated here.

[0287] Optionally, adjusting the parameters of the first initial model and the parameters of the initial query vector based on the difference between the first prediction result and the first label data, and the difference between the second prediction result and the second label data, to train the first prediction model and the query vector, may include: after obtaining the first prediction result, calculating the first prediction result and the first label data using a preset first target loss function to obtain a first target loss, which indicates the difference between the first prediction result and the first label data; calculating the second prediction result and the second label data using a preset second target loss function to obtain a second target loss, which indicates the difference between the second prediction result and the second label data. After obtaining the first target loss and the second target loss, the parameters of the first initial model can be updated based on the first target loss and the second target loss, and the updated first initial model can be trained using the next batch of training samples until the model training conditions are met (e.g., the first target loss and the second target loss converge, etc.), thereby obtaining the trained first prediction model.

[0288] Furthermore, the initial query vector involved in the training process is generated through random initialization. While continuously updating the initial model parameters, the initial query vector is also trained, resulting in a trained query vector. This trained query vector can be understood as the query vector used during inference. Figures 5 to 9 The query vector in the illustrated embodiment.

[0289] In one possible implementation, step S1004 may further include: inputting the sample road data and the initial query vector into the first initial model to obtain a second prediction result, wherein the initial query vector is a randomly initialized vector and the second prediction result is a prediction result of road elements of the sample road within the second range; obtaining a first prediction result based on the sample road data and the second prediction result, wherein the first prediction result is a prediction result of road elements of the sample road within the first range; and adjusting the parameters of the first initial model and the query vector based on the difference between the first prediction result and the first label data and the difference between the second prediction result and the second label data to train the first prediction model and the query vector.

[0290] Optionally, adjusting the parameters of the first initial model based on the difference between the first prediction result and the first label data, and the difference between the second prediction result and the second label data, to train the first prediction model, may include: after obtaining the second prediction result, calculating the second prediction result and the first label data using a preset second objective loss function to obtain a second objective loss, which indicates the difference between the second prediction result and the second label data; inputting the second prediction result as prior information and the first sample data into the first initial model to obtain the first prediction result; after obtaining the first prediction result, calculating the first prediction result and the first label data using a preset first objective loss function to obtain a first objective loss, which indicates the difference between the first prediction result and the first label data; after obtaining the first objective loss and the second objective loss, updating the parameters of the first initial model based on the first objective loss and the second objective loss, and training the first initial model with updated parameters using the next batch of training samples until the model training conditions are met (e.g., the first objective loss and the second objective loss converge), thereby obtaining the trained first prediction model.

[0291] The first prediction model trained in this embodiment has the ability to predict road features using input data. During the prediction process, it solves the feature truncation problem from the BEV perspective by predicting road condition data features acquired by sensors over a wider range, thus improving the perception distance of road features and enabling prediction of road features at greater distances. Furthermore, using the prediction results of near-end road features as input for far-end road feature prediction accelerates the convergence speed of the decoder, enhances the perception capability of the first prediction model, and achieves more accurate prediction of far-end road features.

[0292] To facilitate understanding of the above embodiments provided in this application, the following points are made.

[0293] (1) In the embodiments of this application, "instruction" may include direct instruction, indirect instruction, explicit instruction, and implicit instruction. When describing a certain instruction information for the purpose of indicating A, it can be understood that the instruction information carries A, directly indicates A, or indirectly indicates A.

[0294] In this application, the information indicated by the instruction information is called the information to be instructed. In specific implementations, there are many ways to indicate the information to be instructed, such as, but not limited to, directly indicating the information to be instructed, such as the information to be instructed itself or its index. It can also indirectly indicate the information to be instructed by indicating other information, where there is a relationship between the other information and the information to be instructed. It can also indicate only a part of the information to be instructed, while the other parts are known or pre-agreed upon. For example, the instruction of specific information can be achieved by using a pre-agreed (e.g., protocol-defined) arrangement of various pieces of information, thereby reducing instruction overhead to some extent. Furthermore, the information to be instructed can be sent as a whole or divided into multiple sub-information pieces, and the sending period and / or timing of these sub-information pieces can be the same or different.

[0295] (2) In this application, "send" and "receive" indicate the direction of signal transmission. For example, "send information to XX" can be understood as the destination of the information being XX, which may include direct transmission via the air interface or indirect transmission via the air interface by other units or modules. "Receive information from YY" can be understood as the source of the information being YY, which may include direct reception from YY via the air interface or indirect reception from YY via the air interface by other units or modules. "Send" can also be understood as the "output" of the chip interface, and "receive" can also be understood as the "input" of the chip interface. In other words, sending and receiving can occur between devices, such as between network devices and terminal devices, or within a device, such as between components, modules, chips, software modules, or hardware modules within the device via a bus, wiring, or interface.

[0296] (3) In the various embodiments of this application, unless otherwise specified or logically conflicting, the terms and / or descriptions of different embodiments are consistent and can be referenced by each other. The technical features of different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0297] (4) In this application, "first" and "second" are used for descriptive convenience only to distinguish objects and are not intended to limit the scope of the embodiments of this application. They are not used to describe the order or sequence of features. It should be understood that the objects described in this way can be interchanged where appropriate so as to describe solutions other than those in the embodiments of this application.

[0298] (5) In this application, “predefined” can be achieved by pre-storing the corresponding code, table or other means that can be used to indicate relevant information in the device. This application does not limit the specific implementation method.

[0299] (6) In this application, the words “exemplary,” “for example,” “exemplary,” “as another example,” etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design that is described as an “exemplary” in this application should not be construed as being more preferred or advantageous than other embodiments or designs.

[0300] (7) In this application, “comprising,” “including,” “having,” and variations thereof mean “including but not limited to,” unless otherwise specifically emphasized. “At least one” means one or more, and “more” means two or more.

[0301] (8) In this application, "and / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can mean: a, or, b, or, c, or, a and b, or, a and c, or, b and c, or, a, b, and c. Where a, b, and c can be single or multiple.

[0302] (9) In the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terms and / or descriptions of different embodiments are consistent and can be referenced by each other. The technical features of different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0303] (10) Some optional features in the various embodiments of this application may not depend on other features in some scenarios, or may be combined with other features in some scenarios, without limitation.

[0304] The above text combined Figures 1 to 10 The prediction method provided in the embodiments of this application is described in detail below. Figures 11-15 The embodiments of the apparatus of this application are described in detail below. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be referred to the foregoing method embodiments.

[0305] Figure 11This is a schematic block diagram of a prediction device 1100 provided in an embodiment of this application.

[0306] The device 1100 can be implemented by software, hardware, or a combination of both. The device 1100 provided in this application embodiment can implement the embodiments of this application. Figure 5 The method flow shown includes the following: the device 1100 includes an acquisition module 1110, a processing module 1120, and a prediction module 1130.

[0307] The acquisition module 1110 is used to acquire road data, which is road condition data collected by at least one sensor.

[0308] The processing module 1120 is used to extract features from the road data to obtain a first feature.

[0309] The prediction module 1130 is used to input the first feature into a first prediction model to obtain a first road element, wherein the first road element is a road element within a first range. Specifically, the first prediction model encodes the first feature within the first range to obtain a second feature; encodes the first feature within a second range to obtain a third feature, wherein the second range is smaller than the first range; and decodes based on the second feature and the third feature to obtain the first road element.

[0310] Optionally, the prediction module 1130 is further configured to input the third feature into a second decoder, the second decoder being configured to decode the input feature within the second range to obtain the second road element, the second road element being the road element of the road within the second range; and to input the second feature and the second road element into a first decoder, the first decoder being configured to decode the input feature within a first range to obtain the first road element.

[0311] Optionally, the prediction module 1130 is further configured to input the second feature and the third feature into a first decoder, the first decoder being configured to decode the input features within the first range to obtain the first road element.

[0312] Optionally, the prediction module 1130 is further configured to input the first feature into the first encoder to obtain the second feature, wherein the first encoder is configured to add position encoding to each feature point in the first feature, and the position encoding is configured to describe the position of each feature point in the first feature within the first range in 3D space.

[0313] Optionally, the prediction module 1130 is further configured to input the first feature into the second encoder to obtain the third feature, wherein the second encoder is configured to project the first feature onto the bird's-eye view BEV space, and the second range is the range corresponding to the BEV space.

[0314] Optionally, the prediction module 1130 is also configured to input the query vector into the first decoder and / or the second decoder.

[0315] Optionally, the first range is the range of roads covered by the road data.

[0316] Figure 12 This is a schematic block diagram of another use of the prediction device 1200 provided in the embodiments of this application.

[0317] The device 1200 can be implemented by software, hardware, or a combination of both. The device 1200 provided in this application embodiment can implement the embodiments of this application. Figure 10 The method flow shown includes the following: the device 1200 includes an acquisition module 1210 and a training module 1220.

[0318] The acquisition module 1210 is used to acquire sample road data and first label data. The sample road data includes road data of the sample road, and the first label data is the road elements of the sample road within the first range.

[0319] The training module 1220 is used to train the first prediction model on the first initial model based on the sample road data and the first label data.

[0320] Optionally, the acquisition module 1210 is further configured to acquire second label data, wherein the second label data is the road elements of the sample road within the second range. The training module 1220 is further configured to train the first initial model to obtain the first prediction model based on the sample road data, the first label data, and the second label data.

[0321] Optionally, the training module 1220 is further configured to input the sample road data and the initial query vector into the first initial model to obtain a first prediction result, wherein the initial query vector is a randomly initialized vector and the first prediction result is the prediction result of the road elements of the sample road within the first range; and adjust the parameters of the first initial model and the parameters of the initial query vector according to the difference between the first prediction result and the first label data to train the first prediction model and the query vector.

[0322] Optionally, the training module 1220 is further configured to input the sample road data and the initial query vector into the first initial model to obtain a second prediction result, wherein the initial query vector is a randomly initialized vector and the second prediction result is a prediction result of road elements of the sample road within the second range; obtain a first prediction result based on the sample road data and the second prediction result, wherein the first prediction result is a prediction result of road elements of the sample road within the first range; and adjust the parameters of the first initial model and the query vector based on the difference between the first prediction result and the first label data and the difference between the second prediction result and the second label data, so as to train the first prediction model and the query vector.

[0323] The device 1100 or device 1200 here may be embodied in the form of a functional module. The term "module" here may be implemented in software and / or hardware, without specific limitation.

[0324] For example, a "module" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The following describes the modules in device 1100 in detail, using device 1100 as an example.

[0325] For example, the implementation of the acquisition module 1110 in device 1100 will be described below. Similarly, the implementation of other modules in device 1100, such as processing module 1120 and prediction module 1130, can refer to the implementation of the acquisition module 1110.

[0326] As an example of a software functional unit, the acquisition module 1110 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the acquisition module 1010 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0327] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0328] As an example of a hardware functional unit, the acquisition module 1110 may include at least one computing device, such as a server. Alternatively, the acquisition module 1010 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0329] The multiple computing devices included in the acquisition module 1110 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition module 1110 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition module 1110 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0330] Therefore, the modules of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0331] It should be noted that the above embodiments of the device 1100, when executing the above methods, are only illustrative examples of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device 1100 can be divided into different functional modules to complete all or part of the functions described above. For example, the acquisition module 1110 can be used to execute any step in the above methods, the processing module 1120 can be used to execute any step in the above methods, and the prediction module 1130 can be used to execute any step in the above methods. The steps implemented by the acquisition module 1110, the processing module 1120, and the prediction module 1130 can be specified as needed. By implementing different steps in the above methods through the acquisition module 1110, the processing module 1120, and the prediction module 1130, all the functions of the device 1100 can be realized.

[0332] Furthermore, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments above, which will not be repeated here.

[0333] The method provided in this application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as processing units, memory, and memory control units; the functions and structure of this hardware will be described in detail later. The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. Optionally, the computer system can be a handheld device such as a smartphone, or a terminal device such as a personal computer; this application does not particularly limit this, as long as the method provided in this application can be used. The executing entity of the method provided in this application can be a computing device, or a functional module within the computing device capable of calling and executing programs.

[0334] The following is combined Figure 13 This application provides a detailed description of a computing device provided in an embodiment.

[0335] Figure 13 This is a schematic diagram of the architecture of a computing device 1300 provided in an embodiment of this application.

[0336] The computing device 1300 can be a server, a computer, or other device with computing capabilities. Figure 13The computing device 1300 shown includes at least one processor 1310 and a memory 1320.

[0337] It should be understood that this application does not limit the number of processors and memories in the computing device 1300.

[0338] The processor 1310 executes instructions in the memory 1320, causing the computing device 1300 to implement the method provided in this application. Alternatively, the processor 1310 executes instructions in the memory 1320, causing the computing device 1300 to implement the various functional modules provided in this application, thereby implementing the method provided in this application.

[0339] Optionally, the computing device 1300 also includes a communication interface 1330. The communication interface 1330 uses a transceiver module, such as, but not limited to, a network interface card or a transceiver, to enable communication between the computing device 1300 and other devices or communication networks.

[0340] Optionally, the computing device 1300 also includes a system bus 1340, wherein the processor 1310, memory 1320, and communication interface 1330 are respectively connected to the system bus 1340. The processor 1310 can access the memory 1320 through the system bus 1340; for example, the processor 1310 can perform data read / write or code execution in the memory 1320 through the system bus 1340. The system bus 1340 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 1340 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 13 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0341] In one possible implementation, the processor 1310 primarily functions to interpret the instructions (or code) of a computer program and process data within the computer software. The instructions of the computer program and the data within the computer software can be stored in memory 1320 or cache 1316.

[0342] Optionally, processor 1310 may be an integrated circuit chip with signal processing capabilities. By way of example and not limitation, processor 1310 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Among these, a general-purpose processor is a microprocessor, etc. For example, processor 1310 may be a central processing unit (CPU).

[0343] Optionally, each processor 1310 includes at least one processing unit 1312 and a memory control unit 1314.

[0344] Optionally, the processing unit 1312, also known as the core, is the most important component of the processor. The processing unit 1312 is manufactured from single-crystal silicon using a specific production process. All calculations, command reception, command storage, and data processing are performed by the core. Each processing unit independently executes program instructions, utilizing parallel computing capabilities to accelerate program execution. Various processing units have fixed logical structures; for example, a processing unit includes logical units such as a Level 1 cache, a Level 2 cache, an execution unit, an instruction-level unit, and a bus interface.

[0345] In one implementation example, the memory control unit 1314 controls the data interaction between the memory 1320 and the processing unit 1312. Specifically, the memory control unit 1314 receives memory access requests from the processing unit 1312 and controls access to memory based on the memory access requests. By way of example and not limitation, the memory control unit is a device such as a memory management unit (MMU).

[0346] In one implementation example, each memory control unit 1314 addresses the memory 1320 via the system bus. An arbitrator is configured in the system bus. Figure 13 (Not shown in the image), the arbitrator is responsible for handling and coordinating competing accesses of multiple processing units 1312.

[0347] In one implementation example, the processing unit 1312 and the memory control unit 1314 are connected via internal chip connection lines, such as address lines, thereby enabling communication between the processing unit 1312 and the memory control unit 1314.

[0348] Optionally, each processor 1310 also includes a cache 1316, which is a buffer for data exchange (called a cache). When the processing unit 1312 needs to read data, it first looks for the required data in the cache. If the data is found, it is executed directly; otherwise, it looks for the data in memory. Since the cache operates much faster than memory, the purpose of the cache is to help the processing unit 1312 run faster.

[0349] The memory 1320 provides runtime space for processes in the computing device 1300. For example, the memory 1320 stores the computer program (specifically, the program code) used to generate the process. After the computer program is run by the processor to generate a process, the processor allocates corresponding storage space for the process in the memory 1320. Furthermore, the aforementioned storage space further includes text segments, initialized data segments, bit initialized data segments, stack segments, heap segments, etc. The memory 1320 stores data generated during the process's execution, such as intermediate data or process data, in the storage space corresponding to the process.

[0350] Optionally, the memory is also called RAM, and its function is to temporarily store the data processed by the processor 1310, as well as the data exchanged with external storage devices such as hard disks. As long as the computer is running, the processor 1310 will load the data that needs to be processed into RAM for processing, and after the processing is completed, the processing unit 1312 will send the result out.

[0351] By way of example and not limitation, memory 1320 is volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory is read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM). It should be noted that the memory 1320 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0352] The above-described structure of the computing device 1300 is merely illustrative and is not intended to limit the application. The computing device 1300 in this application includes various hardware components found in existing computer systems. For example, the computing device 1300 may also include other storage devices besides the memory 1320, such as disk storage. Those skilled in the art should understand that the computing device 1300 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device 1300 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the computing device 1300 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 13 All the devices shown.

[0353] Figure 14 This is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of this application.

[0354] The computing device cluster includes at least one computing device. This computing device may be a server. In some embodiments, the computing device may also be a terminal device such as a desktop computer, laptop computer, or smartphone.

[0355] like Figure 14 As shown, the computing device cluster includes at least one computing device 1300. The memory 1320 of one or more computing devices 1300 in the computing device cluster may store the same instructions for performing the methods described above.

[0356] In some possible implementations, the memory 1320 of one or more computing devices 1300 in the computing device cluster may also each store a portion of the instructions for executing the above-described method. In other words, a combination of one or more computing devices 1300 can jointly execute the instructions of the above-described method.

[0357] It should be noted that the memory 1320 in different computing devices 1300 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned device. That is, the instructions stored in the memory 1320 of different computing devices 1300 can implement the functions of one or more modules within the aforementioned device.

[0358] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0359] Figure 15 This is a schematic diagram showing the connection between computing devices 1300A and 1300B via a network, as provided in the embodiments of this application.

[0360] like Figure 15 As shown, the two computing devices 1300A and 1300B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0361] It should be understood that Figure 15 The functions of computing device 1300A shown can also be performed by multiple computing devices 1300. Similarly, the functions of computing device 1300B can also be performed by multiple computing devices 1300.

[0362] This application also provides a vehicle that may include the aforementioned prediction device 1100.

[0363] This application also provides a computer program product containing instructions, which may be a software or program product containing instructions capable of running on a computing device or stored on any usable medium. When run on a computing device, it causes the computing device to perform the methods provided above, or causes the computing device to perform the functions of the apparatus provided above.

[0364] This application also provides a computer-readable storage medium, which can be any available medium capable of being stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that, when executed on a computing device, cause the computing device to perform the method provided above.

[0365] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0366] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0367] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0368] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0369] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0370] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0371] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0372] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of road prediction, characterized in that, The method comprises: acquiring road data, the road data being road condition data of a road collected by at least one sensor; performing feature extraction on the road data to obtain first features; inputting the first features into a first prediction model to obtain first road elements, the first road elements being road elements of the road within a first range; wherein the first prediction model encodes the first features within the first range to obtain second features; encoding the first features within a second range to obtain third features, the second range being smaller than the first range; decoding the second features and the third features to obtain the first road elements.

2. The method of claim 1, wherein, The decoding of the second features and the third features to obtain the first road elements comprises: inputting the third features into a second decoder, the second decoder being configured to decode input features within the second range to obtain second road elements, the second road elements being road elements of the road within the second range; inputting the second features and the second road elements into a first decoder, the first decoder being configured to decode input features within the first range to obtain the first road elements.

3. The method of claim 1, wherein, The decoding of the second features and the third features to obtain the first road elements comprises: inputting the second features and the third features into a first decoder, the first decoder being configured to decode input features within the first range to obtain the first road elements.

4. The method according to any one of claims 1 to 3, characterized in that, The encoding of the first features within the first range to obtain second features comprises: inputting the first features into a first encoder to obtain the second features, the first encoder being configured to add position encodings to each feature point in the first features, the position encodings being configured to describe positions of each feature point in the first features within the first range in a three-dimensional (3D) space.

5. The method according to any one of claims 1 to 4, characterized in that, The encoding of the first features within the second range to obtain third features comprises: inputting the first features into a second encoder to obtain the third features, the second encoder being configured to project the first features to a bird's eye view (BEV) space, the second range being a range corresponding to the BEV space.

6. The method according to any one of claims 2 to 5, characterized in that, The method further comprises inputting a query vector into the first decoder and / or the second decoder.

7. The method according to any one of claims 4 to 6, characterized in that, The adding of the position encodings to each feature point in the first features comprises: concatenating each feature point in the first features and the corresponding position encodings thereof; or superimposing each feature point in the first features and the corresponding position encodings thereof.

8. The method according to any one of claims 4 to 7, characterized in that, The position encodings are determined according to a first matrix, the first matrix being a projection transformation matrix corresponding to the sensor.

9. The method of claim 8, wherein, The first matrix, each feature point in the first features, and the position encodings satisfy: z ij ·[x ij ,y ij ,1] T =T i ·[x',y',z',1] T wherein T i represents the projection transformation matrix corresponding to the ith sensor, j represents the jth feature point in the first feature, [x ij ,y ij ,z ij ] represents the position coordinates of the jth feature point in the first feature collected by the ith sensor, [x', y', z'] represents the position encoding of the jth feature point, and i and j are positive integers.

10. The method according to any one of claims 1 to 9, characterized in that, The sensor comprises at least one of a camera, a laser radar, or a millimeter wave radar. The road data comprises at least one of image data, video data, point cloud data, or a map containing the road.

11. The method according to any one of claims 1 to 10, characterized in that, The road element comprises at least one of the following: a lane line, a lane center line, a road, or an obstacle.

12. The method according to any one of claims 1 to 11, characterized in that, The first prediction model is implemented based on a Transformer architecture or a convolutional neural network (CNN) architecture.

13. The method according to any one of claims 1 to 12, characterized in that, The first range is a road range covered by the road data.

14. The method according to any one of claims 1 to 13, characterized in that, The method further comprises: The first prediction model is trained according to sample road data and first label data, wherein the sample road data comprises road data of a sample road, and the first label data is a road element of the sample road within the first range.

15. The method according to any one of claims 1 to 14, characterized in that, The method further comprises: The first prediction model is trained according to sample road data, first label data, and second label data, wherein the sample road data comprises road data of a sample road, the first label data is a road element of the sample road within the first range, and the second label data is a road element of the sample road within the second range.

16. The method of claim 14, wherein, The first prediction model is trained in the following manner: The sample road data and an initial query vector are input into a first initial model to obtain a first prediction result, the initial query vector is a randomly initialized vector, and the first prediction result is a road element prediction result of the sample road within the first range; The parameters of the first initial model and the parameters of the initial query vector are adjusted according to the difference between the first prediction result and the first label data, so as to train the first prediction model and the query vector.

17. The method of claim 15, wherein, The first prediction model is trained in the following manner: The sample road data and an initial query vector are input into a first initial model to obtain a second prediction result, the initial query vector is a randomly initialized vector, and the second prediction result is a road element prediction result of the sample road within the second range; The first prediction result is obtained according to the sample road data and the second prediction result, and the first prediction result is a road element prediction result of the sample road within the first range; the parameters of the first initial model and the parameters of the query vector are adjusted according to the difference between the first prediction result and the first label data and the difference between the second prediction result and the second label data, so as to train the first prediction model and the query vector.

18. An apparatus for road prediction, the apparatus comprising: It comprises: An acquisition module configured to acquire road data, wherein the road data is road condition data collected by at least one sensor; A processing module configured to perform feature extraction on the road data to obtain first features; A prediction module configured to input the first features into a first prediction model to obtain first road elements, wherein the first road elements are road elements of the road within a first range; The first prediction model encodes the first features within the first range to obtain second features, encodes the first features within a second range to obtain third features, and decodes the second features and the third features based on the second features and the third features to obtain the first road elements, wherein the second range is smaller than the first range.

19. The apparatus of claim 18, wherein, The prediction module is configured to input the third feature into a second decoder, the second decoder being configured to decode the input feature in the second range to obtain a second road element, the second road element being a road element of the road in the second range. The second feature and the second road element are input into a first decoder, the first decoder being configured to decode the input feature in the first range to obtain the first road element.

20. The apparatus of claim 18, wherein, The prediction module is configured to input the second feature and the third feature into a first decoder, the first decoder being configured to decode the input feature in the first range to obtain the first road element.

21. The apparatus of any one of claims 18-20, wherein, The prediction module is configured to input the first feature into a first encoder to obtain the second feature, the first encoder being configured to add a position encoding to each feature point in the first feature, the position encoding being configured to describe a position of each feature point in the first feature in a three-dimensional (3D) space in the first range.

22. The apparatus of any one of claims 18-21, wherein, The prediction module is configured to input the first feature into a second encoder to obtain the third feature, the second encoder being configured to project the first feature to a bird's eye view (BEV) space, the second range being a range corresponding to the BEV space.

23. The apparatus of any one of claims 19-22, wherein, The prediction module is further configured to input a query vector into the first decoder and / or the second decoder.

24. The apparatus of any one of claims 18-23, wherein, The first range is a road range covered by the road data.

25. An apparatus for road prediction, the apparatus comprising: Comprises: a memory for storing a computer program; a processor for executing the computer program stored in the memory to cause the apparatus to perform the method of any one of claims 1 to 17.

26. A road prediction system characterized by, An apparatus for road prediction as claimed in any one of claims 18 to 25.

27. A vehicle characterized by An apparatus for road prediction as claimed in any one of claims 18 to 25, or a road prediction system as claimed in claim 26.

28. A computer-readable storage medium, characterized in that, A computer program stored thereon, which, when executed by a computer, causes the implementation of the method of any one of claims 1 to 17.

29. A chip or chip system, characterized by Comprises: circuitry for performing the method of any one of claims 1 to 17.

30. A computer program product, characterised in that, The computer program product comprises computer program code which, when executed on a computer, causes the implementation of the method of any one of claims 1 to 17. The computer program product comprises computer program code which, when executed on a computer, causes the implementation of the method of any one of claims 1 to 17.

Images