Data processing method for vehicle and training method of data processing model

Abstract

The present disclosure provides a data processing method for a vehicle, a training method and device of a data processing model, equipment, a medium and an autonomous vehicle, relates to the field of artificial intelligence, and in particular to the technical field of autonomous driving and computer vision. The specific implementation scheme of the data processing method for the vehicle is as follows: a perception network is used to encode scene data and motion data of the vehicle to obtain element features of environmental elements in a scene where the vehicle is located; the scene data is obtained based on original data collected by sensors on the vehicle; an aggregation network is used to aggregate the element features and map data corresponding to the location of the vehicle to obtain aggregated features; and a decoding network is used to decode the aggregated features to obtain a prediction result for the vehicle, wherein the prediction result includes prediction trajectory information of the vehicle, and at least one of the following information: structured information of the environmental elements, prediction position information of the environmental elements at a future time.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, specifically to the fields of autonomous driving and computer vision, and particularly to a data processing method for vehicles and a training method, apparatus, device, medium, and autonomous vehicle for a data processing model for vehicles. Background Technology

[0002] With economic development and improved living standards, safe and convenient travel has become one of people's primary needs, leading to the rapid development of autonomous driving technology. Currently, Level 2 autonomous driving technology is becoming increasingly mature, and Level 4 autonomous driving technology is expected to be put into commercial application.

[0003] However, the commercialization of Level 4 autonomous driving technology still faces bottlenecks, such as low traffic efficiency and low accuracy of planned routes. Summary of the Invention

[0004] This disclosure aims to provide a vehicle-specific data processing method and a training method, apparatus, device, and medium for a vehicle-specific data processing model that are conducive to improving the accuracy of path planning.

[0005] According to a first aspect of this disclosure, a data processing method for a vehicle is provided, comprising: encoding scene data and vehicle motion data using a perception network to obtain feature characteristics of environmental elements in the scene where the vehicle is located; obtaining scene data based on raw data collected by sensors on the vehicle; aggregating the feature characteristics and map data corresponding to the vehicle's location using an aggregation network to obtain aggregated features; and decoding the aggregated features using a decoding network to obtain a prediction result for the vehicle, wherein the prediction result includes predicted trajectory information of the vehicle and at least one of the following: structured information of environmental elements and predicted location information of environmental elements at future times.

[0006] According to a second aspect of this disclosure, a training method for a data processing model for a vehicle is provided, wherein the data processing model includes a perception network, an aggregation network, and a decoding network; the method includes: using the perception network to encode scene data and vehicle motion data included in the sample data to obtain feature characteristics of environmental elements in the scene where the vehicle is located; the scene data is obtained based on raw data collected by sensors on the vehicle; using the aggregation network to aggregate the feature characteristics and map data corresponding to the vehicle's location included in the sample data to obtain aggregated features; using the decoding network to decode the aggregated features to obtain a prediction result for the vehicle; the prediction result includes predicted trajectory information of the vehicle; and training the data processing model based on the predicted trajectory information and the actual trajectory information of the vehicle, wherein the prediction result also includes at least one of the following: structured information of environmental elements and predicted location information of environmental elements at future times.

[0007] According to a third aspect of this disclosure, a prediction device for a vehicle is provided, comprising: an encoding module for encoding scene data and vehicle motion data using a perception network to obtain feature characteristics of environmental elements in the scene where the vehicle is located; the scene data is obtained based on raw data collected by sensors on the vehicle; an aggregation module for aggregating the feature characteristics and map data corresponding to the vehicle's location using an aggregation network to obtain aggregated features; and a decoding module for decoding the aggregated features using a decoding network to obtain a prediction result for the vehicle, wherein the prediction result includes predicted trajectory information of the vehicle, and at least one of the following: structured information of environmental elements, and predicted location information of environmental elements at future times.

[0008] According to a fourth aspect of this disclosure, a training apparatus for a data processing model for a vehicle is provided, wherein the data processing model includes a perception network, an aggregation network, and a decoding network; the apparatus includes: an encoding module, used to encode scene data and vehicle motion data included in the sample data using the perception network to obtain feature characteristics of environmental elements in the scene where the vehicle is located; the scene data is obtained based on raw data collected by sensors on the vehicle; an aggregation module, used to aggregate the feature characteristics and map data corresponding to the vehicle's location included in the sample data using the aggregation network to obtain aggregated features; a decoding module, used to decode the aggregated features using the decoding network to obtain a prediction result for the vehicle; the prediction result includes predicted trajectory information of the vehicle; and a first training module, used to train the data processing model based on the predicted trajectory information and the actual trajectory information of the vehicle, wherein the prediction result also includes at least one of the following: structured information of environmental elements, and predicted location information of environmental elements at future times.

[0009] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the data processing method for a vehicle and / or the training method for a data processing model for a vehicle provided in this disclosure.

[0010] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to perform the data processing method for a vehicle and / or the training method for a data processing model for a vehicle provided in this disclosure.

[0011] According to a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program / instructions stored on at least one of a readable storage medium and an electronic device, wherein the computer program / instructions, when executed by a processor, implement the data processing method for a vehicle and / or the training method for a data processing model for a vehicle provided in this disclosure.

[0012] According to the eighth aspect of this disclosure, an autonomous vehicle is provided, the autonomous vehicle including the vehicle-specific data processing apparatus provided in the third aspect of this disclosure.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0015] Figure 1 This is a schematic diagram illustrating an application scenario of a data processing method for vehicles and a training method and apparatus for a data processing model for vehicles, according to embodiments of this disclosure.

[0016] Figure 2 This is a schematic flowchart of a data processing method for vehicles according to an embodiment of the present disclosure;

[0017] Figure 3 This is a schematic diagram illustrating the principle of obtaining scene data according to an embodiment of this disclosure;

[0018] Figure 4 This is a schematic diagram illustrating the principle of obtaining the feature characteristics of environmental elements using a sensing network;

[0019] Figure 5 This is a schematic diagram illustrating the principle of information aggregation using aggregation networks;

[0020] Figure 6 This is a schematic diagram illustrating the implementation principle of a data processing method for vehicles according to an embodiment of this disclosure.

[0021] Figure 7 This is a flowchart illustrating a method for training a data processing model for a vehicle according to an embodiment of the present disclosure.

[0022] Figure 8 This is a structural block diagram of a vehicle prediction device according to an embodiment of the present disclosure;

[0023] Figure 9 This is a structural block diagram of a training apparatus for a data processing model of a vehicle according to an embodiment of the present disclosure; and

[0024] Figure 10 This is a block diagram of an electronic device used to implement the vehicle data processing method and / or the vehicle data processing model training method according to the embodiments of this disclosure. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] Autonomous driving technology primarily relies on the collaborative processing of a perception module and a planning and control (PNC) module. The encoding module converts unstructured information acquired by sensors such as cameras and radar into structured information (structured representation). This structured information includes, but is not limited to, obstacle information and road feature information. Obstacle information can include information about other vehicles, pedestrians, non-motorized vehicles, and obstacles such as rocks. Road feature information can include lane markings, traffic lights, and other road surface information. This structured information needs to be matched with information contained in a high-precision map to accurately determine the position of the objects represented by the structured information on the high-precision map, thus fusing the information from the high-precision map with the perceived structured information. The PNC module performs prediction, planning, and control based on the fused structured information and the historical states of the objects represented by the structured information. Prediction mainly predicts the movement state of surrounding objects over a future period, planning mainly plans the vehicle's trajectory over a future period, and control converts the planned trajectory into specific control signals for the vehicle.

[0027] Typically, prediction and planning are strongly coupled. If we expand the timeline (t = 0s, 1s, 2s, 3s, ..., Ts), at each time point, surrounding objects and the main vehicle have multiple possible movements. For T time points, the number of possibilities expanded by time point can reach N. T There are N possibilities, where N is the average number of possibilities at each time point. The computational scale of this type is enormous, making prediction and planning virtually impossible. To address this problem, some mainstream methods decouple prediction and planning, i.e., prediction is performed first, and then planning is based on the prediction results. However, these mainstream methods ignore the fact that surrounding objects may change due to different decisions made by the driver, thus resulting in a loss of planning accuracy.

[0028] Overall, current autonomous driving technologies based on perception-prediction-planning mainly face the following types of bottlenecks and problems.

[0029] The first category is the problem of error accumulation. A serious error accumulation problem exists in the derivation process of prediction and planning. That is, as time progresses, the accuracy of the derivation results decreases rapidly, and the error expands exponentially. Furthermore, in existing technologies, the output information of perception must conform to a predefined structured paradigm. Once perception fails, there is no possibility of correction in the planning and control stages; new problems can only be added on top of the perception error.

[0030] The second category involves the problem of poor robustness arising from reliance on manually configured rules. To address the issue of the extremely large scale of computation, numerous prior rules are typically used to prune possibilities and accelerate processing. However, these prior rules are often based on expert experience and may lack rules for a large number of long-tailed or unexpected situations. Furthermore, conflicts and inconsistencies may exist between multiple prior rules set for various scenarios, making path optimization extremely difficult.

[0031] The third category includes methods that cannot perform end-to-end optimization of the prediction and planning processes, thus failing to simultaneously guarantee the safety, efficiency, and comfort of autonomous driving. Specifically, while coupled prediction and planning solutions can achieve overall optimization, the prohibitive implementation costs necessitate significant compromises in planning accuracy, thereby compromising safety, efficiency, and comfort. Rule-based methods, on the other hand, cannot perform end-to-end optimization and can only guarantee safety to a certain extent, failing to ensure comfort and travel efficiency.

[0032] The fourth category involves high optimization costs. Currently, the training of the encoding and predictive planning modules cannot rely on the final driving signals; it must depend on the accurate annotations of intermediate structured representations. This necessitates the annotation of a large amount of sensory data during training, resulting in high annotation costs and limitations on the available data. The current optimization process relies on high-precision maps, but collecting such maps is extremely expensive. In practice, issues such as temporary road repairs often lead to incompatibility between high-precision maps and actual data, rendering existing solutions unfeasible.

[0033] To address the aforementioned problems, this disclosure provides a data processing method for vehicles and a training method, apparatus, device, and medium for a data processing model for vehicles. The following will first consider... Figure 1 The application scenarios of the methods and apparatus provided in this disclosure are described.

[0034] Figure 1 This is a schematic diagram illustrating an application scenario of the data processing method for vehicles and the training method and apparatus for vehicle data processing models according to embodiments of this disclosure.

[0035] like Figure 1 As shown, the application scenario 100 of this embodiment may include an autonomous vehicle 110 and a server 120. The autonomous vehicle 110 may integrate an autonomous driving system, and the server 120 may be, for example, a backend management server that provides support for the operation of the autonomous driving system.

[0036] In one embodiment, the autonomous vehicle 110 may integrate various types of sensors that communicate with the autonomous driving system, such as visual cameras and radar-based ranging sensors. The visual cameras may include, for example, monocular cameras, binocular stereo cameras, panoramic cameras, and infrared cameras. The radar-based ranging sensors may include, for example, lidar, millimeter-wave radar, and ultrasonic radar. The autonomous driving system may process the data collected by the various types of sensors to determine the environmental information of the autonomous vehicle 110 and plan the path for the autonomous vehicle 110.

[0037] For example, the autonomous driving system can also download various types of maps from the server 120, specifically lane-level maps 130, and combine the lane-level maps 130 with data collected by sensors to perceive the surrounding environment and plan the path for the autonomous vehicle 110.

[0038] In one embodiment, server 120 may also provide a data processing model 140 for the autonomous driving system, which can process the data collected by sensors according to the data processing model 140 and combine it with lane-level data. Figure 1 30. Perform environmental perception and path planning. For example, data processing model 140 can take path planning (also known as trajectory prediction) as the main task and environmental perception (such as obstacle detection in the surrounding environment) as an auxiliary task, and simultaneously output the structured information of the surrounding environment obtained by perception and the trajectory information of the vehicle obtained by prediction.

[0039] In one embodiment, server 120 may, for example, combine labeled data, closed-loop data, and open-loop data to train data processing model 140.

[0040] In one embodiment, the autonomous driving system in the autonomous vehicle 110 may, for example, send the data collected by the sensors to the server 120, whereby the server 120 performs perception of the surrounding environment and path planning for the autonomous vehicle 110. Subsequently, the server 120 sends the predicted trajectory information to the autonomous driving system, which then determines the control signals for the autonomous vehicle 110 based on the trajectory information and controls the autonomous vehicle 110.

[0041] It should be noted that the vehicle data processing method provided in this disclosure can be executed by an autonomous vehicle or an autonomous driving system within an autonomous vehicle, or by the server 120. Correspondingly, the vehicle prediction device provided in this disclosure can be located in an autonomous vehicle or an autonomous driving system within an autonomous vehicle, or in the server 120. The training method for the vehicle data processing model provided in this disclosure can be executed by the server 120. Correspondingly, the training device for the vehicle data processing model provided in this disclosure can be located in the server 120.

[0042] It should be understood that Figure 1 The number and type of autonomous vehicles 110 and servers 120 shown are merely illustrative. Depending on implementation needs, there can be any number and type of autonomous vehicles 110 and servers 120.

[0043] The following will combine Figures 2-6 The data processing method for vehicles provided in this disclosure is described in detail.

[0044] Figure 2 This is a schematic flowchart of a data processing method for vehicles according to an embodiment of the present disclosure.

[0045] like Figure 2 As shown, the data processing method 200 for a vehicle in this embodiment may include operations S210 to S230.

[0046] In operation S210, a perception network is used to encode scene data and vehicle motion data to obtain the feature characteristics of environmental elements in the scene where the vehicle is located.

[0047] According to embodiments of this disclosure, scene data can be obtained based on raw data collected by sensors on the vehicle at historical moments. For example, the scene data can be obtained by transforming raw data collected by multiple sensors at historical moments into a unified coordinate system, and using the data in this unified coordinate system as the scene data. The raw data collected by the multiple sensors may include image data captured by multiple RGB cameras at multiple historical moments. Radar point cloud data Ultrasonic radar data And so on. Here, i represents multiple (e.g., N) rgb The i-th camera in a set of N RGB cameras, where t represents a specific moment among multiple historical moments, and N... lid N represents the number of point clouds that the lidar can collect. radThis represents the number of distances that the ultrasonic radar can detect. The unified coordinate system can be, for example, any two-dimensional or three-dimensional coordinate system. In one embodiment, to facilitate the fusion of data from multiple sensors, the unified coordinate system can be selected as a vehicle-based coordinate system.

[0048] According to embodiments of this disclosure, vehicle motion data may include, for example, data representing the positional changes of the vehicle between two adjacent historical moments, the vehicle's speed at each of multiple historical moments, and a transformation matrix between a coordinate system constructed based on each sensor and a coordinate system constructed based on the vehicle. The data on the vehicle's position change at time t relative to time (t-1) can be represented by rotation and displacement matrices. Among them, the coordinate system built based on the vehicle can include the bird's eye view (BEV) coordinate system.

[0049] According to embodiments of this disclosure, the perception network may employ a multi-sensor fusion framework, such as a BEV encoder within a multi-task multi-sensor fusion framework (BEVFusion) based on a unified Bird's-Eye View Representation. By using BEV as the unified representation for fusion, almost all perception data can be fused, and the process of transforming the original data to the BEV coordinate system can simultaneously preserve the geometric structure and semantic density of environmental elements.

[0050] According to embodiments of this disclosure, the scene in which the vehicle is located may include, for example, multiple environmental elements. The obtained element features may include the element features of each of the multiple environmental elements. When the scene data includes data from multiple historical moments, for each environmental element, the obtained element features are in the form of a feature sequence, which includes the element features of each environmental element at each historical moment.

[0051] In operation S220, an aggregation network is used to aggregate feature data and map data corresponding to the vehicle's location to obtain aggregated features.

[0052] Map data corresponding to the vehicle's location can be determined in the following ways: The vehicle's location is used to determine its corresponding map position in a lane-level map; data on lane markings, pedestrian crossings, traffic lights, etc., within a predetermined range of that map position are then used as map data. Specifically, traffic light data can be represented by points, and lane markings and pedestrian crossing data can be represented by line segments.

[0053] According to embodiments of this disclosure, the aggregation network may include, for example, a convolutional network or a network with a self-attention mechanism. In this embodiment, the embedding features of the map data and the embedding features of the prompt information can be determined separately to obtain map features and prompt features respectively. For example, data indicating each object (such as traffic lights, zebra crossings, lane lines, etc.) in the map data can be input into the embedding layer in sequence, and the embedding layer outputs an embedding feature sequence, where each embedding feature in the sequence represents an object. This embodiment can also simultaneously input feature features and map features into the aggregation network, and obtain aggregated features after processing by the aggregation network.

[0054] In one embodiment, the aggregation feature may include, for example, aggregated feature features and aggregated map features corresponding to feature features, map features and cue features, respectively.

[0055] In operation S230, a decoding network is used to decode the aggregated features to obtain the prediction results for the vehicle.

[0056] According to embodiments of this disclosure, the decoding network can be selected, for example, based on the type of prediction result obtained from the prediction task. For example, when there are multiple prediction tasks, the decoding network may include multiple decoding sub-networks corresponding to each of the multiple prediction tasks.

[0057] For example, the prediction result may include the vehicle's predicted trajectory information. The decoding network can then employ a decoder capable of generating 3D coordinates, such as a Transformer decoder. This embodiment can use a series of 3D coordinates output by the decoding network as the predicted trajectory information, and the trajectory fitted based on this series of 3D coordinates as the predicted trajectory. Specifically, when the aggregated features include aggregated feature characteristics, aggregated map features, and aggregated cue features, these aggregated features can be concatenated and used as input to the decoding network. After processing by the decoding network, the network outputs the predicted trajectory information.

[0058] For example, if the prediction result can include structured information about environmental elements, the decoding network can employ a network structure consisting of fully connected layers and normalization layers. The fully connected layers process the aggregated features to obtain the location information and bounding box information of the environmental elements. The normalization layers can be used, for example, to determine the category of the environmental element. Specifically, the aggregated features can be input into the fully connected layer, which outputs a 3+3+X dimensional vector. The first three dimensions represent the location of the environmental element in 3D space, and the middle three dimensions represent the length, width, and height of the bounding box. The last X dimensions, after processing by the normalization layer, yield a probability vector for multiple predetermined categories, including the probability that the environmental element belongs to each of the predetermined categories.

[0059] For example, if the prediction results can include the predicted location information of environmental elements at future times, then the decoding network can employ a fully connected layer. Specifically, aggregated features can be input into the fully connected layer, which then outputs a series of 3D vectors P. t，1，l P t，2，l , ..., P t,T′，l , representing the predicted positions of environmental element l at times (t+1), (t+2), ..., (t+T') after the current time t. Alternatively, a series of 3D vectors can also represent the predicted positions of environmental elements at multiple non-uniform times in the future.

[0060] It is understood that the above prediction results and the corresponding decoding network structure are merely examples to facilitate understanding of this disclosure, and this disclosure does not limit them. The aforementioned perceptual network, aggregation network, and decoding network can, for example, be connected sequentially to form an end-to-end data processing model.

[0061] In one embodiment, the prediction result may include the predicted trajectory information of the vehicle, and may also include at least one of the structured information of the aforementioned environmental elements and the predicted location information of the environmental elements at future times. This embodiment may treat the task of predicting the vehicle's trajectory information as the primary data processing task, and the task of obtaining the structured information of the environmental elements and the task of obtaining the predicted location information of the environmental elements at future times as auxiliary tasks.

[0062] This embodiment of the disclosure achieves parallel execution of the prediction and planning stages by obtaining predicted trajectory information, structured information of environmental elements, and predicted location information of environmental elements at future times through a decoding network. This avoids the error accumulation that exists in existing technologies that predict first and then plan. Furthermore, the features of the perceived scene data are implicitly transmitted to the decoding network used to predict vehicle trajectory information via the network, thus improving the accuracy of the information referenced during prediction and consequently improving the precision of the predicted vehicle trajectory information.

[0063] Furthermore, although the embodiments of this disclosure can employ an end-to-end model structure, the interpretability and reliability of the end-to-end model architecture can be improved due to the output of the prediction results of the aforementioned auxiliary tasks. Simultaneously, the failure to identify environmental elements due to the lack of a well-defined structured representation of certain environmental elements can be mitigated to some extent by employing gradient iteration within the end-to-end model architecture.

[0064] Figure 3 This is a schematic diagram illustrating the principle of obtaining scene data according to an embodiment of this disclosure.

[0065] like Figure 3 As shown, in one embodiment 300, when obtaining scene data based on the raw data 320 collected by the vehicle's sensors 310, the raw data 320 collected by the sensors 310 can first be converted to the BEV coordinate system to obtain pixel data 330 and depth data 340. For example, for radar point cloud data... Each point cloud (x, y, z) ∈ x lid，t The transformation matrix R between the coordinate system constructed from the sensors that collected the radar point cloud data in the vehicle's operating data described above and the BEV coordinate system can be used. cam，i，t Each point cloud (x, y, z) is mapped to a pixel in the BEV coordinate system using the following formula (1), thus obtaining pixel data 330. Accordingly, depth data 340 can be determined using the following formula (2).

[0066] u, v = R cam，i，t ·(x, y, z, 1) formula (1)

[0067]

[0068] Where x, y, z represent the coordinate positions of the point cloud in the coordinate system constructed by the sensor based on the radar point cloud data, u, v represent the coordinate positions of the mapped pixel points in the BEV coordinate system, and d represents the depth value.

[0069] Through mapping, for example, a two-dimensional image x can be obtained.cam，t (u, v) and the depth map x corresponding to the two-dimensional image d，t (u, v) = d. After obtaining pixel data 330 and depth data 340, this embodiment 300 can stitch the pixel data 330 and depth data 340 along the channel dimension to obtain perceptual data 350. For example, x can be... i，t (u, v) and x d，t (u, v) are concatenated along the channel dimension to obtain four-dimensional data as perceptual data 350. Where W and H represent the pixel width and pixel height of the two-dimensional image, respectively, i represents the i-th sensor among multiple sensors, and t represents a certain time. That is, the perceived data x can be obtained based on the raw data collected by the i-th sensor at time t. merg，i,t .

[0070] In one embodiment, the perceived data can be used as scene data, for example.

[0071] In one embodiment, after obtaining the perceived data 350, a residual network 360 can be used to process the perceived data 350, and the data processed by the residual network 360 can be used as scene data 370. Through the processing of the residual network, the shape and number of channels of the perceived data 350 can be changed, avoiding gradient explosion in the end-to-end network. It is understood that the residual network 360 can, for example, form an overall end-to-end network architecture with the perceived network, aggregation network and decoding network described above, and this disclosure does not limit it in this way.

[0072] For example, residual networks can adopt the ResNet-101 architecture to perceive data x. merg,i,t Scene data f obtained through residual network processing i，t For example, it can be represented by the following formula (3).

[0073]

[0074] It is understood that the above technical solution only takes the processing of raw data collected by a single sensor as an example. This embodiment can perform similar processing on the raw data collected by each of multiple sensors, and all the data obtained can constitute scene data in sequence form.

[0075] This embodiment of the disclosure improves the accuracy of the obtained scene data by converting the raw data collected by the sensor to the BEV coordinate system and retaining the depth information of the point cloud data. By using a residual network to process the stitched data, it is possible to use a deeper structure in the subsequent network, thereby improving the accuracy of the obtained prediction results.

[0076] Figure 4This is a schematic diagram illustrating the principle of obtaining the feature characteristics of environmental elements using a sensing network.

[0077] In one embodiment, the perception network may employ a deformable attention mechanism to encode scene and motion data, thereby obtaining feature characteristics of environmental elements. This approach allows the perception network to focus on key regions and capture more information features.

[0078] For example, a perceptual network can employ a deformable self-attention mechanism to iteratively encode the pre-trained query data, the scene data described above, and the motion data a predetermined number of times (K). The data obtained from these K iterations can then be used as the feature data of the environmental elements. For instance, randomly assigned initial data processed by an embedding layer can be used as the initial value of the query data. During the training of the perceptual network, the network parameters of the embedding layer are continuously adjusted until the training of the perceptual network is complete. In this embodiment, the data obtained after processing the initial data through the adjusted embedding layer can be used as the query data obtained from pre-training. For example, the length of the initial data can be determined based on the maximum number of environmental elements around the vehicle under normal circumstances. The larger the maximum number, the longer the length of the initial data.

[0079] In this embodiment, the query data obtained from pre-training can be used as the query feature (query), and key features (key) and value features (value) are obtained based on scene data and motion data. Then, a cross-self-attention mechanism is used to process the query feature, key feature, and value feature, and the resulting features are used as the features obtained in the first iterative encoding. Subsequently, this embodiment can use the features obtained in the first iterative encoding as the query, and combine them with the key feature and value feature for a second iterative encoding. This process continues until K iterative encodings are performed, resulting in the feature features of the environmental elements. It is understood that the principle of this K-iterative encoding is similar to the iterative encoding principle in the Transformer architecture, and will not be elaborated further here.

[0080] In one embodiment 400, as Figure 4As shown, the perceptual network can employ the deformable self-attention mechanism used in the network architecture for learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (BEVFormer) to perform iterative encoding. For example, the perceptual network can include K sequentially connected encoding subnetworks 410, each encoding subnetwork including a deformable attention mechanism layer DeformAttn 411 and a mapping-normalization layer 412 (abbreviated as FFN). The mapping-normalization layer 412 can be composed of two fully connected layers and a normalization layer sequentially connected.

[0081] Accordingly, during the K-fold encoding process, this embodiment can use query data 401 as the first query feature adopted in the first iteration encoding. The first query feature Motion data R car,t R cam,i,t 402 and scene data f i，t 403. The first input data is processed by the deformable self-attention layer 411 of the first-ranked encoding subnetwork 410 among the K encoding subnetworks 410, to achieve the first iteration encoding. This embodiment can then use the mapping-normalization layer of the first-ranked encoding subnetwork to process the features obtained from the first iteration encoding and the first query features. This process yields the second query feature for the second encoding subnetwork among the K encoding subnetworks 410. 404. Subsequently, this embodiment can use the second query feature... 404, Motion Data R car,t R cam,i,t 402 and scene data f i，t 403. Input the deformable self-attention layer included in the second encoding sub-network, and implement the second iterative encoding based on a similar principle. Then, use the mapping-normalization layer included in the second encoding sub-network to process the features obtained from the second iterative encoding and the second query features. This process yields the third query feature for the third coding subnetwork among the K coding subnetworks 410. Similarly, the Kth query feature obtained from the (K-1)th encoding subnetwork can be... 405. Motion data R car,t R cam,i,t 402 and scene data f i，t403. Input the deformable self-attention layer included in the Kth encoding sub-network, and implement the Kth iteration encoding based on a similar principle. Then, use the mapping-normalization layer included in the Kth encoding sub-network to process the features obtained from the Kth iteration encoding and the Kth query features. Processing is performed to obtain the (K+1)th query feature. 406. In this embodiment, the (K+1)th query feature can be... 406 is an environmental factor.

[0082] For the j-th encoding subnetwork among the K encoding subnetworks, the input data can be processed using formula (4) to obtain the (j+1)-th query feature.

[0083]

[0084] Where φ() is R based on the input motion data 402 cam,i,t The projection operation is performed on the j-th query feature. After processing by the DeformAttn layer, the obtained feature can be concatenated with the j-th query feature (i.e., the "+" operation in formula (4)). The mapping-normalization layer first processes the concatenated feature through two fully connected layers F, and then uses a normalization layer N to normalize the feature obtained from the two fully connected layers F, thereby obtaining the query feature.

[0085] In one embodiment, environmental elements may include a first type of elements. These first type of elements may include obstacles and / or road features detected by sensors around the vehicle. Accordingly, the perception network can function to perceive the feature features of each first type of element, and the pre-trained query data may include first query data q for each anchor point of the first type of elements. dyn，l，t The first type of element can be understood as a dynamic element. Obstacles can include other vehicles, pedestrians, non-motorized vehicles, etc. Road elements can include lane lines, traffic lights, etc. The number of data items in the first query is L. dyn The maximum number N of obstacles that can typically surround a vehicle. o The maximum number N of road elements surrounding the vehicle r The sum is used to represent L, that is: dyn =N o +N r In this embodiment, the feature characteristics obtained after processing by the perception network may include, for example, the feature characteristics of obstacles. o，l，t Feature characteristics e of 407 and road elements r,l,t408. Anchor points typically refer to the geometric center of the graphic.

[0086] In one embodiment, environmental elements may include a second type of elements, which may include each cell in a predetermined road surface grid. The predetermined road surface grid may be constructed based on a region within a predetermined range for the vehicle. Accordingly, the perception network can function as a means to perceive the feature characteristics of each second type of element, and the pre-trained query data may include second query data q for the anchor points of each element in the second type of element. stat，l，t In the second category of elements, the anchor point of each element can be understood as the center point of each cell. The number L of the second query data... stat The total number of cells included in a predetermined road surface grid can be L. stat =W B ×H B Among them, W B H B These refer to the number of cells in each row and the number of cells in each column of the predetermined road surface grid, respectively. It is understood that the predetermined range, the size of a single cell, etc., can be set according to actual needs, and this disclosure does not limit this. In this embodiment, the feature features obtained after processing by the perception network may, for example, include the feature features e of each cell. b,l,t 409.

[0087] When environmental factors include both type I and type II factors, this embodiment can uniformly represent the query data as q. l，t Where, l = [1, 2, ..., L] stat , ..., L stat +L dyn In this embodiment, the feature characteristics obtained after processing by the perception network may include, for example, the feature characteristics of obstacles. o，l，t Feature characteristics e of 407 and road elements r,l,t 408 and feature e of each cell b，l，t 409. This method allows for a comprehensive understanding of environmental elements. For example, if the query data includes L... stat +N o +N r Given a data point, the (K+1)th query feature obtained after the above K iterations of encoding will correspondingly include features related to L. stat +N o +N r The features corresponding to each data point can be used in this embodiment to identify the first L. stat Each feature serves as a feature of the cell, that is... Where i1 = [1, 2, ..., L]stat ], with the middle N o Each feature serves as an element feature of an obstacle, namely Where i2 = [1, 2, ..., N] o ], the last N r These features serve as the feature characteristics of road elements, namely... Where i3 = [1, 2, ..., N] r ].

[0088] In one embodiment, the various element features obtained above may include the features of each environmental element at K historical time points, that is, in the above element features, t = [0s, -1s, -2s, ..., -Ms]. Here, t = 0s represents the current time. That is, for each environmental element, a series of (M+1) element features can be obtained.

[0089] Figure 5 This is a schematic diagram illustrating the principle of information aggregation using aggregation networks.

[0090] In one embodiment, when aggregating data using an aggregation network, vehicle-specific prompt data can also be aggregated simultaneously. This prompt data may include, for example, route information, traffic information, and input commands received in response to passenger actions within the vehicle. Route information may be, for example, route information provided by a lane-level map or navigation map loaded into the vehicle's navigation system. Traffic information may include, for example, speed limits on the road where the vehicle is located. Input commands received in response to passenger actions may include instructions such as "speed up" or "smooth up." This embodiment, by simultaneously aggregating prompt data, provides richer information for the prediction process, thereby improving the accuracy of the prediction results.

[0091] For example, such as Figure 5 As shown, in this embodiment 500, the aggregation network 510 can simultaneously aggregate the feature characteristics of environmental elements obtained in the manner described above, the map data 501 corresponding to the vehicle's location, and the prompt data 502.

[0092] Among them, the feature elements may include, for example, the feature elements of obstacles described above. o，l，0 ..., e o，l，-K 523, Element characteristics of road elements e r,l,0 ..., e r,l,-K 524, and the cell's feature e b,l,0 ..., e b，l，-K 525.

[0093] The map features 521 of map data 501 can be determined using the method described above, that is, the vectorized representation e of map data 501 can be obtained.m，l Similarly, the processing method for map data can be similar to that used when determining the embedding features of map data. The positions of each point included in the route information in the prompt data 502 can be processed to obtain the embedding features corresponding to the route information. Speed ​​limit information can be discretized and then converted into embedding vectors to obtain the embedding features corresponding to the speed limit information. Instructions indicating "speed up" or "smooth up" can be segmented at the character or word level, and the resulting sequences can be embedded to obtain the embedding features corresponding to the instructions. In this embodiment, the feature obtained by concatenating various embedding features is used as the prompt feature 522 of the prompt data 502, that is, the vectorized representation e of the prompt data 502 is obtained. p，l .

[0094] This embodiment can simultaneously use map feature 521, cue feature 522, and feature features 523-525 of each environmental element as input to the aggregation network 510. The aggregation network 510 aggregates the input feature features, map features, and cue features to obtain aggregated features. In one embodiment, the aggregated features may include, for example, the aggregated map features corresponding to map feature 521. 531, the aggregated cue feature corresponding to cue feature 522 532, the aggregated feature corresponding to the feature feature 523 of the obstacle. 533, the aggregated feature corresponding to feature feature 524 of the road feature. 534, and the aggregated feature corresponding to feature feature 525 of cell 534. 535.

[0095] In one embodiment, the aggregation network 510 may, for example, employ a network structure consisting of K' aggregation layers built based on an attention mechanism. For instance, the aggregation network 510 may be a K'-level Transformer cascade network. In this embodiment, feature features, map features, and cue features can be input into the K'-level Transformer cascade network, and the K'-th level network in the K'-level Transformer cascade network can output aggregated features.

[0096] In one embodiment, the aggregation network can perform alternating temporal-spatial dimension aggregation on the input feature features, map features, and cue features based on an attention mechanism to obtain aggregated features. For example, the aggregation network can adopt the principle of joint modeling of temporal and spatial dimensions in Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting (AgentFormer) to aggregate the input features. AgentFormer allows the state of one object at a certain moment to directly affect the state of another object at a future moment. AgentFormer uses a temporal encoder to address the problem of lost temporal information, which appends timestamp features to each input feature. Object-aware attention generates two sets of key features and query features by performing different linear transformations on the input features; one set of key features and query features is used to calculate the attention between objects, while the other set is used to calculate the attention within objects. The temporal encoder uses an encoder similar to the position encoding in the original Transformer, calculating timestamp features based on the time step of the input features, and the timestamp uses the same sine design as the position encoding. By performing alternating aggregation of time and space dimensions, various types of features can be modeled in both time and space dimensions, thereby improving the expressive power of the aggregated features and enhancing the accuracy of subsequent prediction results.

[0097] In one embodiment, multiple historical moments (i.e., moments 0s, ..., -Ks) can be downsampled to obtain multiple sampling moments. During aggregation, the features of environmental elements, map features, and cue features at these multiple sampling moments are aggregated to obtain aggregated features. For example, downsampling can be performed using equal-interval sampling, or the sampling interval can gradually increase with the interval between historical moments and the current moment. Alternatively, a space-related temporal sampling method can be used; for example, if a vehicle has not moved within a certain historical time period, sampling can be performed only for that historical time period. By sampling historical moments, the expressive power of the obtained aggregated features can be maintained while reducing aggregation costs and improving data processing efficiency. It is understood that the above sampling methods are merely examples to facilitate understanding of this disclosure, and this disclosure does not limit them.

[0098] Figure 6 This is a schematic diagram illustrating the implementation principle of a data processing method for vehicles according to an embodiment of this disclosure.

[0099] As mentioned earlier, the data processing method can take the task of predicting the trajectory information of vehicles as the primary task, and take at least one of the tasks of obtaining the structured information of environmental elements and obtaining the predicted location information of environmental elements at future times as auxiliary tasks.

[0100] Based on this, such as Figure 6 As shown, the implementation principle 600 of the data processing method in this embodiment can be, for example, as follows: Scene data 611 and motion data 612 are used as inputs to a perception network 601, which encodes the feature characteristics of environmental elements in the scene where the vehicle is located. The feature characteristics, map data 613 (specifically, map features representing map data), and prompt data 614 (specifically, prompt features representing prompt data) are input to an aggregation network 602, which aggregates the input data to obtain aggregated features. The obtained aggregated features can, for example, include the aggregated map features corresponding to the map features described above. 615, aggregated prompt features corresponding to the prompt features 616, Aggregated feature characteristics corresponding to the feature characteristics of obstacles 617, Aggregated feature features corresponding to the feature features of road features 618, and the aggregated feature features corresponding to the feature features of the cell. 619. In this embodiment, aggregated features can be input into a decoding network 603, which outputs predicted trajectory information 621 for the vehicle corresponding to the primary task, and information corresponding to the auxiliary task. The information corresponding to the auxiliary task may include at least one of the following: structured information 622 corresponding to the task of obtaining structured information of environmental elements, and predicted location information 623 corresponding to the task of obtaining predicted location information of environmental elements at future times.

[0101] In one embodiment, environmental elements may include the first type of elements described above, namely, obstacle and road elements described above, and may also include the second type of elements described above, namely, each cell in the predetermined road surface grid. The structured information 622 described above may, for example, only be the structured information for each of the first type of elements, because each cell in the predetermined road surface grid is virtually constructed, while obstacles and road elements are real. Accordingly, the task of obtaining this structured information can be understood as an object detection task. Accordingly, the decoding network 603 may, for example, include a first decoding sub-network. The perceived element features may include a first feature sequence for the first type of elements, which includes the element features of each of the first type of elements. The aggregated features may include a first aggregated feature corresponding to the first feature sequence. In this embodiment, the first decoding sub-network can be used to decode the first aggregated feature to obtain the structured information for each of the first type of elements. For example, the first decoding sub-network may adopt the network structure described above, consisting of fully connected layers and normalization layers. The obtained structured information may include the position of each first-class feature in 3D space, the length, width, and height of the bounding box of each first-class feature, and the probability that each first-class feature belongs to each of the multiple predetermined categories.

[0102] For example, for obstacles and road features, there can be first-class features that can be distinguished into different subcategories. This embodiment can set up a decoding sub-network for each first-class feature in order to accurately classify the first-class features in each subcategory. For example, multiple predetermined categories for obstacles may include pedestrian categories, vehicle categories, non-motorized vehicle categories, etc. Multiple predetermined categories for road features may include lane line categories, traffic light categories, guide strip categories, etc.

[0103] In one embodiment, the task of obtaining the predicted location information of environmental elements at future times can be primarily used to obtain the predicted location information of obstacles, for example, because the location of road elements is usually fixed, and each cell in the predetermined road surface grid is virtually constructed. Accordingly, the decoding network may include a third decoding sub-network. This embodiment can use this third decoding sub-network to decode the target features in the first aggregated features, thereby obtaining the predicted location information of obstacles at future times. The target features are the features in the first aggregated features corresponding to the first feature sequence for obstacles. The third decoding sub-network may, for example, employ a fully connected layer. The specific implementation principle can be found in the principle of obtaining the predicted location information of environmental elements at future times described above, and will not be repeated here.

[0104] In one embodiment, the auxiliary task may further include, for example, the task of predicting the current drivable area of ​​the vehicle. The drivable area may be, for example, a drivable area within a predetermined range of the vehicle. Accordingly, the auxiliary task may be implemented based on features of cells in a predetermined road grid. Accordingly, the decoding network may include a second decoding sub-network for performing the task of predicting the current drivable area of ​​the vehicle, the perceived feature features including a second feature sequence for each cell, and the aggregated features including a second aggregated feature corresponding to the second feature sequence. This embodiment may employ the second decoding sub-network to decode the second aggregated features to obtain drivable information for each cell, which may include 0 indicating that each cell is drivable, or 1 indicating that each cell is not drivable. This embodiment may determine the drivable area 624 for the vehicle based on the drivable information of all cells in the predetermined road grid. For example, the area containing all cells with drivable information of 0 may be considered as the drivable area 624. For example, the second decoding sub-network may employ a deconvolution network structure. This embodiment may concatenate the drivable information of all cells according to the cell positions, for example, to obtain a drivable area map I for the vehicle. driverable .

[0105] In one embodiment, the auxiliary task may further include a driving experience prediction task, which can be, for example, the experience of a vehicle driving according to a determined predicted trajectory. By setting and training the driving experience prediction task, the rationality and accuracy of the predicted trajectory obtained by the end-to-end model can be improved. Correspondingly, the decoding network may include, for example, a fourth decoding sub-network, which can be used to decode all aggregated features obtained by the aggregation network to obtain the vehicle's driving experience information 625. By processing all aggregated features, the comprehensiveness of the information considered in the prediction process of the driving experience information can be ensured, thus improving the accuracy of the predicted driving experience information. The fourth decoding sub-network may, for example, employ a decoder from the field of natural language processing, such as a decoder from the Transformer architecture.

[0106] For example, the driving experience information 625 may include at least two of the following: information on whether a traffic violation occurred, information on whether a collision occurred, information on whether driving intervention is required, and driving comfort information. For example, taking the information on whether driving intervention is required as an example, the fourth decoding subnetwork may output a series of characters. This embodiment can determine whether driving intervention is required based on a specific character in this series of characters. The specific character may be, for example, the first character, and its position can be set according to the training objective and actual needs; this disclosure does not limit this. Alternatively, the fourth decoding subnetwork may be a multi-classification subnetwork, which can determine whether driving intervention is required based on the classification results. For example, if the classification result for whether driving intervention is required is 1, it indicates that driving intervention is required. Driving intervention refers to whether human intervention is needed during the operation of an autonomous vehicle.

[0107] To facilitate the implementation of the aforementioned vehicle data processing method, this disclosure also provides a training method for a vehicle data processing model, which will be described below in conjunction with... Figure 7 The training method is described in detail.

[0108] Figure 7 This is a flowchart illustrating a method for training a data processing model for a vehicle according to an embodiment of the present disclosure.

[0109] like Figure 7 As shown, the training method 700 for the vehicle data processing model in this embodiment may include operations S710 to S740. The data processing model may include, for example, the perception network, aggregation network, and decoding network described above.

[0110] When operating the S710, a perception network is used to encode the scene data and vehicle motion data included in the sample data to obtain the feature characteristics of environmental elements in the scene where the vehicle is located.

[0111] The sample data may include scene data and vehicle motion data from historical moments prior to a certain point in time. The scene data is obtained based on raw data collected by sensors on the vehicle, specifically the raw data used by the sensors at that historical moment. It is understood that the implementation principle of operation S710 is similar to that of operation S210 described above, and the perception network is similar to that described above, so it will not be repeated here.

[0112] When operating the S720, an aggregation network is used to aggregate the feature data and sample data, including the map data corresponding to the vehicle's location, to obtain aggregated features.

[0113] According to embodiments of this disclosure, the map data included in the sample data may be determined, for example, based on the vehicle's location at a historical time. The principle for determining the map data in this sample data is similar to the principle for determining the map data mentioned in the above-described operation S220 section, and operation S720 is similar to the implementation principle of operation S220 described above, and will not be repeated here.

[0114] In operation S730, a decoding network is used to decode the aggregated features to obtain a prediction result for the vehicle.

[0115] The prediction results include the vehicle's predicted trajectory information, which can be obtained by fitting the predicted position information of multiple times after a certain time.

[0116] In one embodiment, the prediction result may further include at least one of the following: structured information of environmental elements, and predicted location information of environmental elements at future times. Here, "future time" refers to a time after a certain time. This "certain time" can be the current time, or any time before the current time; this disclosure does not limit it.

[0117] In one embodiment, the implementation principle of operation S730 is similar to that of operation S230 described above, and will not be repeated here.

[0118] When operating the S740, the data processing model is trained based on the vehicle's predicted trajectory information and the vehicle's actual trajectory information.

[0119] According to embodiments of this disclosure, the actual trajectory information of the vehicle can be, for example, trajectory information obtained from a simulation environment, or information about the actual trajectory of the vehicle after a certain moment. For instance, this actual trajectory information can include the trajectory obtained in a scenario where the vehicle is under open-loop control. Accordingly, this embodiment can record the sensor data and motion data obtained in the scenario where the vehicle is under open-loop control, and incorporate map data to obtain sample data. It is understood that open-loop control refers to a system control method without feedback information. In human-machine systems designed with open-loop control, the design of operating instructions is crucial; errors can lead to irreparable losses.

[0120] In this embodiment, for example, the loss value using the L2 loss function can be determined based on the predicted trajectory information and the actual trajectory information, and this loss value can be used as the loss value for the data processing model to predict the trajectory. With the goal of minimizing this loss value, the backpropagation algorithm is used to adjust the network parameters in the data processing model, thereby training the data processing model. It is understood that the above loss function is merely an example to facilitate understanding of this disclosure, and this disclosure does not limit it.

[0121] For example, the loss value L for the trajectory predicted by the data processing model can be determined using the following formula (5). bc Among them, Y t Indicates the predicted trajectory. This represents the actual trajectory.

[0122]

[0123] In one embodiment, the prediction result may further include structured information of environmental elements. These environmental elements may include, for example, the first type of elements described above, namely, at least one of obstacle and road elements. When decoding the aggregated features, the first decoding sub-network described above can be used to decode the first aggregated features to obtain the structured information of each first type of element. In this embodiment, the sample data may be labeled data, specifically, the sample data may have first labeling information, which may indicate, for example, the structured information of the actual first type of element in the scene where the vehicle is located at a historical moment. It is understood that the structured information indicated by the labeling information may represent, for example, the 3D position of the actual first type of element, the length, width, and height of the bounding box of the actual first type of element, and the category of the actual first type of element. The predicted structured information includes the predicted 3D position of the first type of element, the length, width, and height of the bounding box of the predicted first type of element, and the probability value of the predicted first type of element belonging to each of multiple predetermined categories.

[0124] In this embodiment, for example, the data processing model can be trained based on the difference between the structured information of the real first type of element and the structured information of the predicted first type of element, so as to improve the accuracy of the data processing model in performing the auxiliary task of obtaining structured information, and thus improve the accuracy of the data processing model in predicting the vehicle's driving trajectory.

[0125] For example, this embodiment can first match the real first-class elements and the predicted first-class elements based on the structured information indicated by the first annotation information and the structured information in the prediction results to obtain at least one matching element pair. For example, during matching, the initial matching result can be determined based on the distance between the 3D positions. Then, the initial matching result is filtered based on the intersection-union ratio (IU) of the bounding boxes of the two elements in the initial matching result. For example, if the distance between the 3D position of the real first-class element and the 3D position of the predicted first-class element is less than a predetermined threshold, it can be determined that the real first-class element and the predicted first-class element match, and an initial matching result is obtained. If the IU between the bounding boxes of the real first-class elements and the predicted first-class elements in the initial matching result is less than a predetermined IU, then the matching result that includes both the real first-class element and the predicted first-class element is removed from the initial matching result. Finally, the two remaining matching first-class elements are considered as a matching element pair. After obtaining the matching feature pairs, this embodiment can determine the difference between the structured information of the predicted first-class feature and the structured information of the true first-class feature for each matching feature pair, and use this difference as the matching difference for each matching feature pair. For example, this matching difference can be obtained by summing the differences in 3D location, bounding box, and category. For example, the difference in 3D location can be determined based on the distance between 3D locations, the difference in bounding box can be determined based on the intersection-union ratio (IU), and the difference in category can be determined based on the probability value of the predicted first-class feature belonging to the category in the true structured information. In this embodiment, for example, the L1 loss function can be used to determine the loss value caused by the difference in 3D location, the IU loss function can be used to determine the loss value caused by the difference in bounding box, and the cross-entropy loss function can be used to determine the loss value caused by the difference in category. This embodiment can normalize and sum the three loss values, and use the summed loss value to represent the matching difference for each matching feature pair. Finally, this embodiment can use the sum of at least one matching difference for at least one matching feature pair as the loss value for the data processing model to perform the auxiliary task of obtaining structured information, and train the data processing model based on this loss value.

[0126] For example, if the first type of features includes obstacles and road features, then the loss value L of the data processing model performing the auxiliary task to obtain structured information about the obstacles is... object The loss value L of the auxiliary task of obtaining structured information of road elements by the data processing model can be expressed by the following formula (6). road It can be expressed using the following formula (7). Wherein, Structured information representing real obstacles, O l，t This represents the structured information of the predicted obstacles. Structured information representing real road features, r l，t This represents the structured information of the predicted road features. If there are multiple obstacle pairs resulting from matching real and predicted obstacles, the loss value L is obtained by summing the matching differences across these multiple obstacle pairs. object Similarly, if there are multiple road feature pairs (consisting of real and predicted road features) obtained from the matching, the matching differences among these multiple road feature pairs will be summed to obtain the loss value L. road .

[0127]

[0128] L road =∑ l Δ(r l gt r l，t ) Formula (7)

[0129] It is understandable that the labeled data can be obtained by manually labeling the data obtained from open-loop or closed-loop control of the vehicle.

[0130] In one embodiment, the prediction result may further include, for example, the predicted drivable area. Correspondingly, the environmental elements may also include the second type of elements described above. The second type of elements includes each cell in the predetermined road surface grid. Then, when decoding the aggregated features, the second decoding sub-network described above can be used to decode the second aggregated features described above to obtain the predicted drivable information for each cell. Based on the predicted drivable information of all cells in the predetermined road surface grid, the predicted drivable area for the vehicle can be determined. In this embodiment, the sample data may be, for example, labeled data; specifically, the sample data may have second labeling information indicating the drivable area within a predetermined range of the vehicle. It is understood that both the labeled drivable area and the predicted drivable area may be specific to a particular moment.

[0131] In this embodiment, for example, the data processing model can be trained based on the difference between the predicted drivable area and the drivable area indicated by the second annotation information, so as to improve the accuracy of the data processing model in performing the auxiliary task of predicting drivable areas, and thus improve the accuracy of the data processing model in predicting the vehicle's driving trajectory.

[0132] For example, the drivable area indicated by the second labeling information can be represented by Drivable Area Map I. gt This indicates that the predicted drivable area can be represented by Drivable Area Map I. driverable This embodiment indicates that the L2 loss function or similar methods can be used to determine the drivable area map I.gt Map I of Driving Area driverable The difference between them is used to obtain the loss L of the data processing model when performing the auxiliary task of predicting drivable areas. driverable For example, the loss L driverable The result can be obtained using the following formula (8). It is understood that the above L2 loss function is only an example to facilitate understanding of this disclosure, and this disclosure does not limit it.

[0133] L driverable =Δ(I gt -I driverable ) Formula (8)

[0134] In one embodiment, the prediction result may further include, for example, the predicted location information of environmental elements at future times. Specifically, it may include the predicted location information of obstacles in the first type of elements described above at future times. Then, when decoding the aggregated features, the third decoding sub-network described above can be used to decode the target features in the first aggregated features, thereby obtaining the predicted location information of obstacles at future times. In this embodiment, the sample data may be, for example, open-loop data or closed-loop data, and this disclosure does not limit it. Among them, the closed-loop data is the data generated by performing closed-loop control on the vehicle. Closed-loop control refers to a control relationship in which the output as the controlled entity returns to the input as the control entity in a certain way and exerts a control effect on the input entity.

[0135] Accordingly, this embodiment can also train the data processing model based on the difference between the predicted location information and the true location information of the obstacle, so as to improve the accuracy of the data processing model in performing the auxiliary task of predicting the location of the obstacle, and thus improve the accuracy of the data processing model in predicting the driving trajectory of the vehicle.

[0136] For example, in this embodiment, obstacles around the vehicle at a future time after a certain time, detected by the vehicle, can be considered as actual obstacles, and the positions of these obstacles can be considered as the ground truth positions of the actual obstacles. The predicted ground truth positions of the actual obstacles are then used as one of the training objectives of the model. Specifically, multiple predicted obstacles and multiple actual obstacles can be matched based on their predicted positions and the ground truth positions of the multiple actual obstacles detected by the vehicle, resulting in matched obstacle pairs. Subsequently, the data processing model is trained based on the difference between the predicted positions of the predicted obstacles and the ground truth positions of the actual obstacles in the matched obstacle pairs. For example, a maximum matching method can be used to match the actual obstacles and predicted obstacles at the future time t+Δt. The predicted positions of the multiple predicted obstacles at the predicted time t+Δt include N... o A predicted obstacle Predicted location The predicted locations of multiple actual obstacles at time t+Δt include N. o 'A predicted obstacle' Predicted location For example, the loss value L of the data processing model performing the auxiliary task of predicting obstacle locations can be obtained using the following formula (9) in this embodiment. future The difference between the predicted position of the obstacle and the actual ground truth position of the obstacle in each matched obstacle pair can be determined, for example, using Euclidean distance or L1 loss, and this disclosure does not limit this. This embodiment can use the sum of the differences obtained from multiple matched obstacle pairs as the loss value for time t+Δt, and the sum of the loss values ​​for multiple future times as the loss value L. future .

[0137]

[0138] In one embodiment, the prediction result may further include, for example, predicted vehicle driving experience information. Then, when decoding the aggregated features, the fourth decoding sub-network described above can be used to decode all features in the aggregated features to obtain the predicted vehicle driving experience information. The predicted driving experience information may include at least two of the following: prediction information on whether a traffic violation occurred, prediction information on whether a collision occurred, prediction information on whether driving intervention is needed, and prediction information on driving comfort. In this embodiment, the sample data can be closed-loop data, i.e., data obtained in a scenario where the vehicle is under closed-loop control. For example, sample data can be obtained by deploying the model in an autonomous vehicle or simulation environment and collecting raw data from sensors in the autonomous vehicle, driving intervention information, etc. It is understood that this predicted driving experience information may be for each of multiple future moments after a certain moment.

[0139] In this embodiment, for example, the data processing model can be trained based on the difference between the predicted information on whether driving intervention is needed and the recorded driving intervention information (i.e., the truth information on whether driving intervention is needed). This improves the accuracy of the data processing model in performing the auxiliary task of predicting whether intervention is needed, and thus helps to improve the accuracy of the data processing model in predicting the vehicle's driving trajectory.

[0140] For example, truth information on whether driving intervention is needed can be obtained using i t To represent, i t The value can be 0 or 1, representing no driving intervention required and driving intervention required, respectively. The prediction information for whether driving intervention is needed can be obtained using p. t To represent. p t The principle of i's value selection is related tot The principle for determining the value is similar. This embodiment can use the following formula (10) to determine the loss value L of the auxiliary task for determining whether the data processing model needs intervention in the prediction process. int Among them, p t with i t The differences between them can be represented, for example, by the loss value obtained by the L1 loss function, and this disclosure does not limit this.

[0141] L int =Δ(p t i t ) Formula (10)

[0142] For example, this embodiment can also employ reinforcement learning principles to train the data processing model, thereby avoiding overfitting caused by gradient descent. Specifically, this embodiment can determine the reward value for the data processing model based on at least two predicted driving experience information points. Then, combining this reward value with ground truth information regarding whether driving intervention is needed, a reward weight is determined. Finally, the difference between the predicted trajectory information and the actual trajectory information is weighted according to this reward weight to determine the prediction loss value of the data processing model.

[0143] Specifically, the weighted sum of at least two pieces of information included in the predicted driving experience can be used as the reward value r for the data processing model. t It is understandable that the prediction information regarding whether a traffic violation will occur, whether a collision will happen, whether driving intervention is needed, and driving comfort can all be represented numerically. For example, the prediction information regarding whether a traffic violation will occur (y)... traf The value y can be 0 (indicating no violation) or 1 (indicating violation), representing the prediction information of whether a collision has occurred. col，t The value can be 0 (indicating no collision) or 1 (indicating a collision), representing the driving comfort prediction information y. comf，t The larger the value of i, the lower the driving comfort. For example, in i t When the value is 0, this embodiment can combine the reward value with (1-i) t The product of (i) and (ii) is used as the reward weight. It's understandable that the reward weight can be negative, representing a penalty for the prediction result. t When the value is 1, this embodiment can directly set i to 1. t As a reward weight, the reward weight is increased when driving intervention occurs. In this way, through reinforcement learning, the prediction results of the data processing model can be made to converge with the results of driving intervention, which helps to improve the accuracy of the trajectory predicted by the data processing model.

[0144] For example, the reward value r t It can be calculated using the following formula (11). Where, rcol r traf r comf r int These represent the weighted values ​​for predictions of whether a collision will occur, whether a traffic violation will occur, the level of driving comfort, and whether driving intervention is required. For example, all these weighted values ​​are negative. It is understood that the values ​​of these weighted values ​​can be set according to actual needs.

[0145] r t =r col *y col，t +r traf *y traf，t +r comf *y comf，t +r int *i t Formula (11)

[0146] It is understandable that, considering that the truth information on whether driving intervention is needed can usually be recorded, this embodiment can use the recorded truth information to replace the prediction information on whether driving intervention is needed to calculate the reward value. In some embodiments, i in formula (11) can also be... t Replace with p t To calculate the reward value.

[0147] For example, in one embodiment, the prediction loss value L of the data processing model can be determined using the following formula (11). HIL Where γ is a hyperparameter set according to actual needs. t represents a certain moment mentioned above, and t′ represents a future moment. t′ This represents the reward value obtained based on the driving experience information at time t′ obtained from the prediction.

[0148]

[0149] It is understood that, in one embodiment, at least two of the multiple loss values ​​obtained by formulas (5) to (10) and (12) above can be used to train the data processing model, or multiple loss values ​​obtained by formulas (5) to (10) and (12) above can be used to train the data processing model simultaneously.

[0150] It is understood that, using the scheme provided in this disclosure, open-loop data can be used to train the data processing model. In this case, the loss value that can be used may include, for example, the loss value obtained by formula (5) described above. By manually labeling the open-loop data, the loss value that can be used may also include the loss value determined by at least one of formulas (6) to (8) described above. If closed-loop data is used to train the data processing model, the loss value that can be used may also include the loss value determined by at least one of formulas (9) to (10) and formula (12) described above. It is also understood that by using the loss values ​​determined by formulas (5), (9), (10) and (12) to train the data processing model, self-supervised training of the model can be achieved.

[0151] When considering multiple loss values ​​simultaneously, this embodiment can train the model with the goal of minimizing the overall loss value L in the early stages of training the data processing model. The overall loss L can be a weighted sum of the multiple loss values, and the weights can be set according to actual needs. In the simulation and real vehicle trial operation phases, a continuous learning approach can also be used to continuously train the data processing model. In continuous training, the overall loss value can be a weighted sum of the loss values ​​determined by the above formulas (9), (10), and (12).

[0152] In other words, using the solution provided in this disclosure, the data processing model can be trained simultaneously with open-loop data, labeled data, and closed-loop data. In the later stages of training, it can rely entirely on closed-loop data. Furthermore, the prediction of the trajectories of obstacle vehicles in the simulation environment can also utilize the data processing model of this embodiment, thus enabling the simulation environment to handle many long-tail cases. Compared to the real-vehicle environment, the simulation environment can simulate rare or even unpredictable long-tail cases in real-vehicle scenarios at a lower cost. These long-tail cases are crucial for the generalization and robustness of the data processing model in real-world environments.

[0153] It is understandable that, during the training process, the principle of using an aggregation network to obtain aggregated features can be similar to that described above. Figure 5 The principles described are similar. The principle for obtaining scene data from sample data is similar to the principle for obtaining scene data in the data processing method described above. The principle for encoding to obtain feature characteristics is similar to the principle described above for... Figure 4 The underlying principles are similar and will not be repeated here.

[0154] Based on the vehicle data processing method provided in this disclosure, this disclosure also provides a vehicle data processing apparatus. The following will be combined with... Figure 8 The device is described in detail.

[0155] Figure 8 This is a structural block diagram of a data processing apparatus for a vehicle according to an embodiment of the present disclosure.

[0156] like Figure 8 As shown, the data processing device 800 for a vehicle in this embodiment may include a first encoding module 810, a first aggregation module 820, and a first decoding module 830.

[0157] The first encoding module 810 is used to encode scene data and vehicle motion data using a perception network to obtain the feature characteristics of environmental elements in the scene where the vehicle is located; the scene data is obtained based on the raw data collected by sensors on the vehicle. In one embodiment, the first encoding module 810 can be used to perform the operation S210 described above, which will not be repeated here.

[0158] The first aggregation module 820 is used to aggregate feature elements and map data corresponding to the vehicle's location using an aggregation network to obtain aggregated features. In one embodiment, the first aggregation module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0159] The first decoding module 830 is used to decode the aggregated features using a decoding network to obtain a prediction result for the vehicle. The prediction result includes the vehicle's predicted trajectory information, and at least one of the following: structured information of environmental elements, and predicted location information of environmental elements at future times. In one embodiment, the first decoding module 830 can be used to perform the operation S230 described above, which will not be repeated here.

[0160] According to embodiments of this disclosure, the environmental elements include at least one of the following first-class elements: obstacle and road elements; the element features include a first feature sequence for each first-class element; the aggregated features include a first aggregated feature corresponding to the first feature sequence; the first decoding module 830 includes: a first decoding submodule, used to decode the first aggregated features using a first decoding subnetwork to obtain structured information of each first-class element, wherein the prediction result includes the structured information of each first-class element.

[0161] According to embodiments of this disclosure, environmental elements include a second type of elements, which includes each cell in a predetermined road surface grid; the predetermined road surface grid is constructed based on a region within a predetermined range for the vehicle; element features include a second feature sequence for each cell; aggregated features include a second aggregated feature corresponding to the second feature sequence; the first decoding module 830 includes: a second decoding submodule, used to decode the second aggregated feature using a second decoding subnetwork to obtain drivable information for each cell; and a region determination submodule, used to determine a drivable region for the vehicle based on the drivable information of all cells in the predetermined road surface grid, wherein the prediction result includes the drivable region.

[0162] According to embodiments of this disclosure, environmental elements include obstacles in a first category of elements; element features include a first feature sequence for each first category of elements; aggregated features include a first aggregated feature corresponding to the first feature sequence; the first decoding module 830 includes: a third decoding submodule, used to decode the target feature in the first aggregated feature using a third decoding subnetwork to obtain the predicted location information of the obstacle at a future time, wherein the target feature is the first aggregated feature corresponding to the first feature sequence for the obstacle; the prediction result includes the predicted location information of the obstacle at a future time.

[0163] According to an embodiment of this disclosure, the first decoding module includes: a fourth decoding submodule, used to decode all features in the aggregated features using a fourth decoding subnetwork to obtain vehicle driving experience information, wherein the driving experience information includes at least two of the following: information on whether a traffic violation occurred, information on whether a collision occurred, information on whether driving intervention is required, and driving comfort information; the prediction result includes the driving experience information.

[0164] According to an embodiment of this disclosure, the first aggregation module 820 includes: a first determining submodule, used to determine the embedding features of map data to obtain map features; a second determining submodule, used to determine the embedding features of prompt data to obtain prompt features; and an aggregation submodule, used to aggregate feature features, map features and prompt features using an aggregation network to obtain aggregated features.

[0165] According to embodiments of this disclosure, the aggregation submodule is used to: perform alternating aggregation of feature features, map features, and cue features based on an attention mechanism in the time-space dimension using an aggregation network to obtain aggregated features.

[0166] According to embodiments of this disclosure, the feature includes multiple features of environmental features at multiple historical moments; the aggregation submodule includes: a downsampling unit for downsampling multiple historical moments to obtain multiple sampling moments; and an aggregation unit for aggregating the features, map features, and cue features of environmental features at multiple sampling moments using an aggregation network to obtain aggregated features.

[0167] According to embodiments of this disclosure, the aforementioned data processing device 800 for a vehicle may further include: a data conversion module for converting raw data collected by sensors on the vehicle into a bird's-eye view coordinate system to obtain pixel data and depth data; a data stitching module for stitching pixel data and depth data in a channel dimension to obtain perception data; and a data processing module for processing the perception data using a residual network to obtain scene data.

[0168] According to an embodiment of this disclosure, the first encoding module 810 is used to perform iterative encoding of the query data, scene data and motion data obtained by pre-training by the perception network using a deformable self-attention mechanism for a predetermined number of times K, so as to obtain the feature characteristics of the environmental elements, wherein K is a natural number greater than 1.

[0169] According to embodiments of this disclosure, the perception network includes K sequentially connected encoding subnetworks, each encoding subnetwork including a deformable self-attention mechanism layer and a mapping-normalization layer. The first encoding module 810 includes an attention processing submodule, used to process the query data as the first query feature adopted in the first iteration encoding, and to process the i-th query feature, scene data, and motion data using the deformable self-attention mechanism layer included in the i-th encoding subnetwork to achieve the i-th iteration encoding; and a fusion submodule, used to process the features obtained from the i-th iteration encoding and the i-th query feature using the mapping-normalization layer included in the i-th encoding subnetwork to obtain the (i+1)-th query feature, wherein the feature of the environmental element includes the (K+1)-th query feature, and i is a natural number greater than or equal to 1 and less than or equal to K.

[0170] According to embodiments of this disclosure, the query data includes: first query data for anchor points of each element in a first class of elements included in the environmental elements; the first class of elements includes at least one of the following: obstacle and road elements; and second query data for the center point of each cell in a predetermined road surface grid included in the environmental elements, the predetermined road surface grid being constructed based on a predetermined area within a vehicle.

[0171] Based on the vehicle data processing device provided in this disclosure, this disclosure also provides an autonomous vehicle, which includes the vehicle data processing device. The on-board system of the autonomous vehicle can, for example, control the power system of the autonomous vehicle based on the predicted trajectory information obtained by the vehicle data processing device, so as to realize the autonomous driving of the autonomous vehicle.

[0172] Based on the training method for a vehicle data processing model provided in this disclosure, this disclosure also provides a training apparatus for a vehicle data processing model. The following will be combined with... Figure 9The device is described in detail.

[0173] Figure 9 This is a structural block diagram of a training apparatus for a data processing model of a vehicle according to an embodiment of the present disclosure.

[0174] like Figure 9 As shown, the training device 900 for the vehicle data processing model in this embodiment may include a second encoding module 910, a second aggregation module 920, a second decoding module 930, and a first training module 940. The data processing model includes a perception network, an aggregation network, and a decoding network.

[0175] The second encoding module 910 is used to encode the scene data and vehicle motion data included in the sample data using a perception network to obtain the feature characteristics of environmental elements in the scene where the vehicle is located; the scene data is obtained based on the raw data collected by sensors on the vehicle. In one embodiment, the second encoding module 910 can be used to perform the operation S710 described above, which will not be repeated here.

[0176] The second aggregation module 920 is used to aggregate the feature data and sample data, including the map data corresponding to the vehicle's location, using an aggregation network to obtain aggregated features. In one embodiment, the second aggregation module 920 can be used to perform the operation S720 described above, which will not be repeated here.

[0177] The second decoding module 930 is used to decode the aggregated features using a decoding network to obtain a prediction result for the vehicle; the prediction result includes the vehicle's predicted trajectory information. The prediction result also includes at least one of the following: structured information of environmental elements, and predicted location information of environmental elements at future times. In one embodiment, the second decoding module 930 can be used to perform the operation S730 described above, which will not be repeated here.

[0178] The first training module 940 is used to train the data processing model based on the vehicle's predicted trajectory information and the vehicle's actual trajectory information. In one embodiment, the first training module 940 can be used to perform the operation S740 described above, which will not be repeated here.

[0179] According to embodiments of this disclosure, the environmental elements include at least one of the following first-class elements: obstacle and road elements; the element features include a first feature sequence for each predicted first-class element; the aggregated features include a first aggregated feature corresponding to the first feature sequence; and the decoding network includes a first decoding sub-network. The second decoding module 930 described above includes: a first decoding sub-module, used to decode the first aggregated features using the first decoding sub-network to obtain structured information of each predicted first-class element, wherein the prediction result includes the structured information of each predicted first-class element.

[0180] According to embodiments of this disclosure, the sample data has first annotation information; the first annotation information indicates the structured information of a real first-type element in the scene where the vehicle is located; the training device 900 further includes: an element matching module, configured to match the real first-type element with the predicted first-type element based on the structured information indicated by the first annotation information and the structured information in the prediction result, to obtain at least one matching element pair; a matching difference determination module, configured to determine, for each of the at least one matching element pair, the difference between the structured information of the predicted first-type element and the structured information of the real first-type element, as the matching difference for each matching element pair; and a second training module, configured to train the data processing model based on at least one matching difference for the at least one matching element pair.

[0181] According to embodiments of this disclosure, environmental elements include a second type of elements, which includes each cell in a predetermined road surface grid; the predetermined road surface grid is constructed based on a region within a predetermined range for the vehicle; element features include a second feature sequence for each cell; aggregated features include a second aggregated feature corresponding to the second feature sequence; and the decoding network includes a second decoding sub-network. The second decoding module 930 includes: a second decoding sub-module, used to decode the second aggregated feature using the second decoding sub-network to obtain predicted drivable information for each cell; and a region determination sub-module, used to determine a predicted drivable region for the vehicle based on the predicted drivable information of all cells in the predetermined road surface grid, wherein the prediction result includes the predicted drivable region.

[0182] According to embodiments of this disclosure, the sample data has second annotation information; the second annotation information indicates a drivable area within a predetermined range of the vehicle. The training device 900 further includes a third training module for training a data processing model based on the difference between the predicted drivable area and the drivable area indicated by the second annotation information.

[0183] According to embodiments of this disclosure, environmental elements include obstacles in a first category of elements; element features include a first feature sequence for each first category of elements; aggregated features include a first aggregated feature corresponding to the first feature sequence. The decoding network includes a third decoding sub-network; the second decoding module 930 includes: a third decoding sub-module, used to decode target features in the first aggregated features using the third decoding sub-network to obtain predicted location information of the obstacles at future times, wherein the target features are the first aggregated features corresponding to the first feature sequence for the obstacles; the prediction result includes predicted location information of the obstacles at future times.

[0184] According to embodiments of this disclosure, the predicted location information includes the predicted locations of multiple predicted obstacles at future times. The training apparatus further includes: an obstacle matching module, configured to match the multiple predicted obstacles and multiple actual obstacles based on the predicted locations of the multiple predicted obstacles and the ground truth locations of multiple actual obstacles detected by the vehicle, to obtain matched obstacle pairs; and a fourth training module, configured to train a data processing model based on the differences between the predicted locations of the predicted obstacles and the ground truth locations of the actual obstacles in the matched obstacle pairs.

[0185] According to embodiments of this disclosure, the decoding network includes a fourth decoding sub-network. The second decoding module 930 further includes a fourth decoding sub-module, configured to decode all features in the aggregated features using the fourth decoding sub-network to obtain predicted vehicle driving experience information, wherein the driving experience information includes at least two of the following: prediction information of whether a traffic violation has occurred, prediction information of whether a collision has occurred, prediction information of whether driving intervention is required, and prediction information of driving comfort; the prediction result includes the predicted vehicle driving experience information.

[0186] According to embodiments of this disclosure, the predicted driving experience information includes prediction information on whether driving intervention is needed. The training device 900 further includes a fifth training module, used to train a data processing model based on the difference between the prediction information on whether driving intervention is needed and the ground truth information on whether driving intervention is needed.

[0187] According to embodiments of this disclosure, the training device 900 further includes: a reward value determination module, configured to determine a reward value for the data processing model based on a weighted sum of at least two pieces of information included in the predicted driving experience information; a weight determination module, configured to determine a reward weight based on the reward value and ground truth information indicating whether driving intervention is required; a loss value determination module, configured to determine a predicted loss value for the data processing model based on the reward weight and the difference between the predicted trajectory information and the actual trajectory information; and a sixth training module, configured to train the data processing model based on the predicted loss value.

[0188] According to embodiments of this disclosure, the sample data mentioned above includes data obtained in scenarios involving closed-loop control of the vehicle.

[0189] According to embodiments of this disclosure, the sample data mentioned above includes data obtained in scenarios involving open-loop control of the vehicle.

[0190] According to an embodiment of this disclosure, the second aggregation module 920 includes: a first determining submodule for determining the embedding features of map data to obtain map features; a second determining submodule for determining the embedding features of prompt data to obtain prompt features; and an aggregation submodule for aggregating feature features, map features, and prompt features using an aggregation network to obtain aggregated features.

[0191] According to embodiments of this disclosure, the aggregation submodule is used to: use an aggregation network to perform time-space dimension alternating aggregation of feature features, map features, and cue features based on an attention mechanism to obtain aggregated features.

[0192] According to embodiments of this disclosure, the feature includes multiple features of environmental features at multiple historical moments; the aggregation submodule includes: a downsampling unit for downsampling multiple historical moments to obtain multiple sampling moments; and an aggregation unit for aggregating the features, map features, and cue features of environmental features at multiple sampling moments using an aggregation network to obtain aggregated features.

[0193] According to embodiments of this disclosure, the training device further includes: a data conversion module for converting raw data collected by sensors on the vehicle into a bird's-eye view coordinate system to obtain pixel data and depth data; a data stitching module for stitching pixel data and depth data in the channel dimension to obtain perception data; and a data processing module for processing the perception data using a residual network to obtain scene data included in the sample data.

[0194] According to an embodiment of this disclosure, the second encoding module 910 is used to: use a deformable self-attention mechanism layer to iteratively encode the query data, scene data and motion data obtained from real-time training by the perception network for a predetermined number of times K, so as to obtain the feature characteristics of the environmental elements, wherein K is a natural number greater than 1.

[0195] According to embodiments of this disclosure, the perception network includes K sequentially connected encoding subnetworks, each encoding subnetwork including a deformable self-attention mechanism layer and a mapping-normalization layer; the aforementioned second encoding module 910 includes: an attention processing submodule, used to use query data as the first query feature adopted in the first iteration encoding, and to process the i-th query feature, scene data and motion data using the deformable self-attention mechanism layer included in the i-th encoding subnetwork to achieve the i-th iteration encoding; and a fusion submodule, used to process the features obtained from the i-th iteration encoding and the i-th query feature using the mapping-normalization layer included in the i-th encoding subnetwork to obtain the (i+1)-th query feature, wherein the feature of the environmental element includes the (K+1)-th query feature, and i is a natural number greater than or equal to 1 and less than or equal to K.

[0196] According to embodiments of this disclosure, the query data includes: first query data for anchor points of each element in a first class of elements included in the environmental elements; the first class of elements includes at least one of the following: obstacle and road elements; and second query data for the center point of each cell in a predetermined road surface grid included in the environmental elements, the predetermined road surface grid being constructed based on a predetermined area within a vehicle.

[0197] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information in this disclosed technical solution comply with relevant laws and regulations, necessary confidentiality measures have been taken, and it does not violate public order and good morals. In this disclosed technical solution, user authorization or consent has been obtained before acquiring or collecting user personal information.

[0198] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0199] Figure 10 A schematic block diagram of an example electronic device 1000 is shown, which can be used to implement the data processing method for a vehicle and / or the training method for a data processing model for a vehicle according to embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0200] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded from storage unit 1008 into random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0201] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0202] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as data processing methods for vehicles and / or training methods for vehicle data processing models. For example, in some embodiments, the data processing methods for vehicles and / or training methods for vehicle data processing models can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by computing unit 1001, one or more steps of the vehicle-specific data processing method and / or vehicle-specific data processing model training method described above can be performed. Alternatively, in other embodiments, computing unit 1001 can be configured to execute the vehicle-specific data processing method and / or vehicle-specific data processing model training method by any other suitable means (e.g., by means of firmware).

[0203] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0204] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0205] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0206] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0207] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0208] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0209] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0210] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method for vehicles, comprising: The scene data and the vehicle's motion data are encoded using a perception network to obtain the feature characteristics of environmental elements in the scene where the vehicle is located. The scene data is obtained based on the raw data collected by the sensors on the vehicle. An aggregation network is used to aggregate the feature elements and the map data corresponding to the location of the vehicle to obtain aggregated features; as well as The aggregated features are decoded using a decoding network to obtain a prediction result for the vehicle. The prediction result includes the predicted trajectory information of the vehicle, and at least one of the following: structured information of the environmental elements, and predicted location information of the environmental elements at future times after the current time. The environmental elements include at least one of the following first-class elements: obstacles and road elements, and The structured information of the environmental element includes the location of the environmental element in 3D space, the length, width, and height of the bounding box of the environmental element, and the probability of the environmental element belonging to each of a plurality of predetermined categories.

2. The method of claim 1, wherein, The element features include a first feature sequence for each first type of element; the aggregated features include a first aggregated feature corresponding to the first feature sequence; The step of using a decoding network to decode the aggregated features to obtain a prediction result for the vehicle includes: The first decoding sub-network is used to decode the first aggregated feature to obtain the structured information of each first type of element. The prediction results include structured information for each of the first-class elements.

3. The method of claim 1, wherein, The environmental elements include a second type of elements, which includes each cell in a predetermined road surface grid; the predetermined road surface grid is constructed based on a region within a predetermined range for the vehicle; the element features include a second feature sequence for each cell; the aggregated features include a second aggregated feature corresponding to the second feature sequence. The step of using a decoding network to decode the aggregated features to obtain a prediction result for the vehicle includes: The second decoding sub-network is used to decode the second aggregated features to obtain the drivability information of each cell; and Based on the drivable information of all cells in the predetermined road surface grid, the drivable area for the vehicle is determined. The prediction results include the drivable area.

4. The method of claim 1, wherein, The environmental elements include obstacles in the first category of elements; the element features include a first feature sequence for each first category of elements; the aggregated features include a first aggregated feature corresponding to the first feature sequence. The step of using a decoding network to decode the aggregated features to obtain a prediction result for the vehicle includes: The target features in the first aggregated features are decoded using a third decoding sub-network to obtain the predicted location information of the obstacle at future times. The target feature is a first aggregated feature corresponding to a first feature sequence for the obstacle; the prediction result includes the predicted location information of the obstacle at a future time.

5. The method of claim 1, wherein, The step of using a decoding network to decode the aggregated features to obtain a prediction result for the vehicle includes: The fourth decoding sub-network is used to decode all features in the aggregated features to obtain the vehicle's driving experience information. The driving experience information includes at least two of the following: information on whether a traffic violation occurred, information on whether a collision occurred, information on whether driving intervention is required, and driving comfort information; the prediction result includes the driving experience information.

6. The method of claim 1, wherein, The aggregation network is used to aggregate the feature elements and the map data corresponding to the vehicle's location to obtain aggregated features, including: Determine the embedding features of the map data to obtain map features; Determine the embedding features of the prompt data to obtain the prompt features; and The aggregated features are obtained by aggregating the feature features, map features, and prompt features using the aggregation network.

7. The method of claim 6, wherein, The aggregation of the feature features, map features, and prompt features using the aggregation network to obtain the aggregated features includes: The aggregation network is used to perform alternating aggregation of the element features, map features, and cue features based on an attention mechanism in the temporal and spatial dimensions to obtain the aggregated features.

8. The method of claim 6 or 7, wherein, The feature characteristics include multiple features of the environmental element at multiple historical moments; The aggregation of the feature features, map features, and prompt features using the aggregation network to obtain the aggregated features includes: Downsampling is performed on the multiple historical moments to obtain multiple sampling moments; as well as The aggregated features are obtained by aggregating the features of the environmental elements, the map features, and the cue features at multiple sampling times using the aggregation network.

9. The method according to claim 1, further comprising: The raw data collected by the sensors on the vehicle is converted to a bird's-eye view coordinate system to obtain pixel data and depth data; The pixel data and the depth data are stitched together along the channel dimension to obtain the perceptual data; as well as The scene data is obtained by processing the perceived data using a residual network.

10. The method of claim 1 or 9, wherein, The process of encoding scene data and vehicle motion data using a perception network to obtain the feature characteristics of environmental elements in the scene where the vehicle is located includes: The perception network employs a deformable self-attention mechanism to iteratively encode the pre-trained query data, the scene data, and the motion data a predetermined number of times (K) to obtain the feature characteristics of the environmental elements. Where K is a natural number greater than 1.

11. The method of claim 10, wherein, The perception network comprises K sequentially connected coding subnetworks, each coding subnetwork including a deformable self-attention mechanism layer and a mapping-normalization layer; The step of using a deformable self-attention mechanism layer to iteratively encode the pre-trained query data, the scene data, and the motion data a predetermined number of times K includes: using the query data as the first query feature used in the first iteration encoding; The i-th encoding subnetwork in the K encoding subnetworks is used to process the i-th query feature, the scene data, and the motion data to achieve the i-th iterative encoding; and The mapping-normalization layer included in the i-th encoding sub-network is used to process the features obtained from the i-th iteration encoding and the i-th query features to obtain the (i+1)-th query features. The feature of the environmental element includes the (K+1)th query feature, where i is a natural number greater than or equal to 1 and less than or equal to K.

12. The method of claim 10, wherein, The query data includes: First query data for the anchor point of each element in the first category of environmental elements; and The second query data pertains to the center point of each cell in a predetermined road surface grid, which is constructed based on a predetermined area within the vehicle.

13. A method of training a data processing model for a vehicle, wherein, The data processing model includes a perceptual network, an aggregation network, and a decoding network; the method includes: The sensor network is used to encode the scene data and vehicle motion data included in the sample data to obtain the feature characteristics of environmental elements in the scene where the vehicle is located. The scene data is obtained based on the raw data collected by the sensors on the vehicle. The aggregation network is used to combine the feature elements and the map data corresponding to the location of the vehicle, including the sample data, to obtain aggregated features; The aggregation features are decoded using the decoding network to obtain a prediction result for the vehicle; the prediction result includes the predicted trajectory information of the vehicle; and The data processing model is trained based on the predicted trajectory information and the actual trajectory information of the vehicle. The prediction result further includes at least one of the following: structured information of the environmental element, and predicted location information of the environmental element at future times after the current time. The environmental elements include at least one of the following first-class elements: obstacles and road elements, and The structured information of the environmental element includes the location of the environmental element in 3D space, the length, width, and height of the bounding box of the environmental element, and the probability of the environmental element belonging to each of a plurality of predetermined categories.

14. The method of claim 13, wherein, The element features include a first feature sequence for each first type of element to be predicted; the aggregated features include a first aggregated feature corresponding to the first feature sequence; the decoding network includes a first decoding sub-network; and the step of using the decoding network to decode the aggregated features to obtain a prediction result for the vehicle includes: The first decoding sub-network is used to decode the first aggregated features to obtain the structured information of each predicted first-class element. The prediction results include structured information for each of the first-class elements in the prediction.

15. The method of claim 14, wherein, The sample data has first annotation information; The first annotation information indicates the structured information of the actual first type of element in the scene where the vehicle is located; the method further includes: Based on the structured information indicated by the first annotation information and the structured information in the prediction result, the real first-class elements are matched with the predicted first-class elements to obtain at least one matching element pair. For each of the at least one pair of matching elements: determine the difference between the structured information of the predicted first-class elements and the structured information of the true first-class elements, as the matching difference for each pair of matching elements; and The data processing model is trained based on at least one matching difference for the at least one pair of matching elements.

16. The method of claim 13, wherein, The environmental elements include a second type of elements, which includes each cell in a predetermined road surface grid; the predetermined road surface grid is constructed based on a region within a predetermined range for the vehicle; the element features include a second feature sequence for each cell; the aggregated features include a second aggregated feature corresponding to the second feature sequence; the decoding network includes a second decoding sub-network; and the step of decoding the aggregated features using the decoding network to obtain a prediction result for the vehicle includes: The second decoding sub-network is used to decode the second aggregated features to obtain the predicted drivability information for each cell; and Based on the predicted drivable information of all cells in the predetermined road surface grid, the predicted drivable area for the vehicle is determined. The prediction results include the predicted drivable area.

17. The method of claim 16, wherein, The sample data has second annotation information; the second annotation information indicates the drivable area within a predetermined range of the vehicle; the method further includes: The data processing model is trained based on the difference between the predicted drivable area and the drivable area indicated by the second annotation information.

18. The method of claim 13, wherein, The environmental elements include obstacles in the first category of elements; the element features include a first feature sequence for each of the first category of elements; the aggregated features include a first aggregated feature corresponding to the first feature sequence; the decoding network includes a third decoding sub-network; the step of decoding the aggregated features using the decoding network to obtain a prediction result for the vehicle includes: The third decoding sub-network is used to decode the target features in the first aggregated features to obtain the predicted location information of the obstacle at future time. The target feature is a first aggregated feature corresponding to a first feature sequence for the obstacle; the prediction result includes the predicted location information of the obstacle at a future time.

19. The method of claim 18, wherein, The predicted location information includes the predicted locations of multiple obstacles at future times; the method further includes: Based on the predicted positions of the multiple predicted obstacles and the ground truth positions of the multiple actual obstacles detected by the vehicle, the multiple predicted obstacles and the multiple actual obstacles are matched to obtain matched obstacle pairs; and The data processing model is trained based on the difference between the predicted positions of obstacles in the matched obstacle pairs and the actual ground truth positions of the obstacles.

20. The method of claim 13, wherein, The decoding network includes a fourth decoding sub-network; the process of using the decoding network to decode the aggregated features to obtain a prediction result for the vehicle includes: The fourth decoding sub-network is used to decode all features in the aggregated features to obtain the predicted driving experience information of the vehicle. The driving experience information includes at least two of the following: prediction information on whether a traffic violation has occurred, prediction information on whether a collision has occurred, prediction information on whether driving intervention is required, and prediction information on driving comfort; the prediction result includes the predicted driving experience information of the vehicle.

21. The method of claim 20, wherein, The predicted driving experience information includes predictions of whether driving intervention is needed; the method further includes: The data processing model is trained based on the difference between the predicted information on whether driving intervention is needed and the true information on whether driving intervention is needed.

22. The method according to claim 20 or 21, further comprising: The reward value for the data processing model is determined by weighted sum of at least two pieces of information included in the predicted driving experience information. The reward weight is determined based on the reward value and the truth value information regarding whether driving intervention is required; The prediction loss value of the data processing model is determined based on the reward weight and the difference between the predicted trajectory information and the actual trajectory information. as well as The data processing model is trained based on the predicted loss value.

23. The method of any one of claims 18-21, wherein, The sample data includes data obtained in scenarios where the vehicle is under closed-loop control.

24. The method of claim 13, wherein, The sample data includes data obtained in scenarios where the vehicle is under open-loop control.

25. The method of claim 13, wherein, The aggregation network is used to combine the feature data and the map data corresponding to the vehicle's location, including the sample data, to obtain aggregated features, which include: Determine the embedding features of the map data to obtain map features; Determine the embedding features of the prompt data to obtain the prompt features; and The aggregated features are obtained by aggregating the feature features, map features, and prompt features using the aggregation network.

26. The method of claim 25, wherein, The aggregation of the feature features, map features, and prompt features using the aggregation network to obtain the aggregated features includes: The aggregation network is used to perform time-space dimension alternating aggregation of the element features, map features, and cue features based on an attention mechanism to obtain the aggregated features.

27. The method of claim 25 or 26, wherein, The feature characteristics include multiple features of the environmental element at multiple historical moments; The aggregation of the feature features, map features, and prompt features using the aggregation network to obtain the aggregated features includes: Downsampling is performed on the multiple historical moments to obtain multiple sampling moments; as well as The aggregated features are obtained by aggregating the features of the environmental elements, the map features, and the cue features at multiple sampling times using the aggregation network.

28. The method of claim 13, further comprising: The raw data collected by the sensors on the vehicle is converted to a bird's-eye view coordinate system to obtain pixel data and depth data; The pixel data and the depth data are stitched together along the channel dimension to obtain the perceptual data; as well as The perceived data is processed using a residual network to obtain the scene data included in the sample data.

29. The method of claim 13 or 28, wherein, The process of encoding the scene data and vehicle motion data included in the sample data using the perception network to obtain the feature characteristics of environmental elements in the scene where the vehicle is located includes: The perceptual network employs a deformable self-attention mechanism layer to iteratively encode the query data, scene data, and motion data obtained from real-time training a predetermined number of times (K) to obtain the feature characteristics of the environmental elements. Where K is a natural number greater than 1.

30. The method of claim 29, wherein, The perception network comprises K sequentially connected coding sub-networks, each coding sub-network including the deformable self-attention mechanism layer and the mapping-normalization layer; The step of using a deformable self-attention mechanism layer to iteratively encode the query data, the scene data, and the motion data obtained from real-time training a predetermined number of times K includes: using the query data as the first query feature used in the first iteration encoding; The i-th encoding subnetwork in the K encoding subnetworks is used to process the i-th query feature, the scene data, and the motion data to achieve the i-th iterative encoding; and The mapping-normalization layer included in the i-th encoding sub-network is used to process the features obtained from the i-th iteration encoding and the i-th query features to obtain the (i+1)-th query features. The feature of the environmental element includes the (K+1)th query feature, where i is a natural number greater than or equal to 1 and less than or equal to K.

31. The method of claim 29, wherein, The query data includes: First query data for the anchor point of each element in the first category of environmental elements; and The second query data pertains to the center point of each cell in a predetermined road surface grid, which is constructed based on a predetermined area within the vehicle.

32. A data processing apparatus for a vehicle, comprising: The encoding module is used to encode scene data and vehicle motion data using a perception network to obtain the feature characteristics of environmental elements in the scene where the vehicle is located. The scene data is obtained based on raw data collected by sensors on the vehicle; The aggregation module is used to aggregate the feature elements and the map data corresponding to the location of the vehicle using an aggregation network to obtain aggregated features; as well as The decoding module is used to decode the aggregated features using a decoding network to obtain a prediction result for the vehicle. The prediction result includes the predicted trajectory information of the vehicle, and at least one of the following: structured information of the environmental elements, and predicted location information of the environmental elements at future times after the current time. The environmental elements include at least one of the following first-class elements: obstacles and road elements, and The structured information of the environmental element includes the location of the environmental element in 3D space, the length, width, and height of the bounding box of the environmental element, and the probability of the environmental element belonging to each of a plurality of predetermined categories.

33. The apparatus of claim 32, wherein, The element features include a first feature sequence for each first type of element; the aggregate features include a first aggregate feature corresponding to the first feature sequence. The decoding module includes: The first decoding submodule is used to decode the first aggregated feature using a first decoding subnetwork to obtain the structured information of each first type of element. The prediction results include structured information for each of the first-class elements.

34. The apparatus of claim 32, wherein, The environmental elements include a second type of elements, which includes each cell in a predetermined road surface grid; the predetermined road surface grid is constructed based on a region within a predetermined range for the vehicle; the element features include a second feature sequence for each cell; The aggregated feature includes a second aggregated feature corresponding to the second feature sequence; the decoding module includes: The second decoding submodule is used to decode the second aggregated feature using a second decoding subnetwork to obtain the drivability information of each cell; and The region determination submodule is used to determine the drivable region for the vehicle based on the drivable information of all cells in the predetermined road surface grid. The prediction results include the drivable area.

35. The apparatus of claim 32, wherein, The environmental elements include obstacles in the first category of elements; the element features include a first feature sequence for each first category of elements; the aggregated features include a first aggregated feature corresponding to the first feature sequence. The decoding module includes: The third decoding submodule is used to decode the target features in the first aggregated features using a third decoding subnetwork to obtain the predicted location information of the obstacle at a future time. The target feature is a first aggregated feature corresponding to a first feature sequence for the obstacle; the prediction result includes the predicted location information of the obstacle at a future time.

36. The apparatus of claim 32, wherein, The decoding module includes: The fourth decoding submodule is used to decode all features in the aggregated features using the fourth decoding subnetwork to obtain the vehicle's driving experience information. The driving experience information includes at least two of the following: information on whether a traffic violation occurred, information on whether a collision occurred, information on whether driving intervention is required, and driving comfort information; the prediction result includes the driving experience information.

37. The apparatus of claim 32, wherein, The aggregation module includes: The first determining submodule is used to determine the embedding features of the map data to obtain map features; The second determining submodule is used to determine the embedding features of the prompt data, thereby obtaining the prompt features; and The aggregation submodule is used to aggregate the feature features, map features, and prompt features using the aggregation network to obtain the aggregated features.

38. The apparatus of claim 37, wherein, The aggregation submodule is used for: The aggregation network is used to perform alternating aggregation of the element features, map features, and cue features based on an attention mechanism in the temporal and spatial dimensions to obtain the aggregated features.

39. The apparatus of claim 37 or 38, wherein, The feature characteristics include multiple features of the environmental element at multiple historical moments; The aggregation submodule includes: A downsampling unit is used to downsample the multiple historical moments to obtain multiple sampling moments; as well as An aggregation unit is used to aggregate the features of the environmental elements, the map features, and the cue features at multiple sampling times using the aggregation network to obtain the aggregated features.

40. The apparatus of claim 32, further comprising: The data conversion module is used to convert the raw data collected by the sensors on the vehicle into a bird's-eye view coordinate system to obtain pixel data and depth data. The data stitching module is used to stitch the pixel data and the depth data in the channel dimension to obtain perceptual data; as well as The data processing module is used to process the perceived data using a residual network to obtain the scene data.

41. The apparatus of claim 32 or 40, wherein, The encoding module is used for: The perception network employs a deformable self-attention mechanism to iteratively encode the pre-trained query data, the scene data, and the motion data a predetermined number of times (K) to obtain the feature characteristics of the environmental elements. Where K is a natural number greater than 1.

42. The device of claim 41, wherein, The perceptual network comprises K sequentially connected encoding subnetworks, each encoding subnetwork including a deformable self-attention mechanism layer and a mapping-normalization layer; the encoding module includes: An attention processing submodule is used to take the query data as the first query feature for the first iteration of encoding, and to process the i-th query feature, the scene data, and the motion data using a deformable self-attention mechanism layer included in the i-th encoding subnetwork of the K encoding subnetworks, in order to achieve the i-th iteration of encoding; and The fusion submodule is used to process the features obtained from the i-th iteration encoding and the i-th query feature using the mapping-normalization layer included in the i-th encoding subnetwork, to obtain the (i+1)-th query feature. The feature of the environmental element includes the (K+1)th query feature, where i is a natural number greater than or equal to 1 and less than or equal to K.

43. The device of claim 41, wherein, The query data includes: First query data for the anchor point of each element in the first category of environmental elements; and The second query data pertains to the center point of each cell in a predetermined road surface grid, which is constructed based on a predetermined area within the vehicle.

44. A training apparatus for a data processing model for a vehicle, wherein, The data processing model includes a perception network, an aggregation network, and a decoding network; the device includes: The encoding module is used to encode the scene data and vehicle motion data included in the sample data using the perception network to obtain the feature characteristics of environmental elements in the scene where the vehicle is located; the scene data is obtained based on the raw data collected by the sensors on the vehicle. The aggregation module is used to combine the feature elements and the map data corresponding to the location of the vehicle, including the sample data, using the aggregation network to obtain aggregated features; A decoding module is used to decode the aggregated features using the decoding network to obtain a prediction result for the vehicle; the prediction result includes the predicted trajectory information of the vehicle; and The first training module is used to train the data processing model based on the predicted trajectory information and the actual trajectory information of the vehicle. The prediction result further includes at least one of the following: structured information of the environmental element, and predicted location information of the environmental element at future times after the current time. The environmental elements include at least one of the following first-class elements: obstacles and road elements, and The structured information of the environmental element includes the location of the environmental element in 3D space, the length, width, and height of the bounding box of the environmental element, and the probability of the environmental element belonging to each of a plurality of predetermined categories.

45. The device of claim 44, wherein, The feature includes a first feature sequence for each predicted first-class feature; the aggregated feature includes a first aggregated feature corresponding to the first feature sequence; the decoding network includes a first decoding sub-network; the decoding module includes: The first decoding submodule is used to decode the first aggregated features using the first decoding subnetwork to obtain the structured information of each predicted first-class feature. The prediction results include structured information for each of the first-class elements in the prediction.

46. The device of claim 45, wherein, The sample data has first annotation information; The first annotation information indicates the structured information of the actual first type of element in the scene where the vehicle is located; the device further includes: The feature matching module is used to match the real first-class features with the predicted first-class features based on the structured information indicated by the first annotation information and the structured information in the prediction result, so as to obtain at least one matching feature pair. A matching difference determination module is configured to, for each of the at least one pair of matching elements, determine the difference between the structured information of the predicted first-class elements and the structured information of the actual first-class elements, as the matching difference for each pair of matching elements; and The second training module is used to train the data processing model based on at least one matching difference for the at least one matching element pair.

47. The device of claim 44, wherein, The environmental elements include a second type of elements, which includes each cell in a predetermined road surface grid; the predetermined road surface grid is constructed based on a region within a predetermined range for the vehicle; the element features include a second feature sequence for each cell; The aggregated feature includes a second aggregated feature corresponding to the second feature sequence; the decoding network includes a second decoding sub-network; the decoding module includes: The second decoding submodule is used to decode the second aggregated features using the second decoding subnetwork to obtain the predicted drivability information for each cell; and The region determination submodule is used to determine the predicted drivable region for the vehicle based on the predicted drivable information of all cells in the predetermined road surface grid. The prediction results include the predicted drivable area.

48. The device of claim 47, wherein, The sample data has second annotation information; The second labeling information indicates the drivable area within a predetermined range of the vehicle; the device further includes: The third training module is used to train the data processing model based on the difference between the predicted drivable area and the drivable area indicated by the second annotation information.

49. The device of claim 44, wherein, The environmental elements include obstacles in the first category of elements; the element features include a first feature sequence for each first category of element; the aggregated features include a first aggregated feature corresponding to the first feature sequence; the decoding network includes a third decoding sub-network; the decoding module includes: The third decoding submodule is used to decode the target features in the first aggregated features using the third decoding subnetwork to obtain the predicted location information of the obstacle at a future time. The target feature is a first aggregated feature corresponding to a first feature sequence for the obstacle; the prediction result includes the predicted location information of the obstacle at a future time.

50. The device of claim 49, wherein, The predicted location information includes the predicted locations of multiple obstacles at future times; the device also includes: An obstacle matching module is used to match the predicted positions of the multiple predicted obstacles and the ground truth positions of the multiple actual obstacles detected by the vehicle, to obtain matched obstacle pairs; and The fourth training module is used to train the data processing model based on the difference between the predicted positions of the obstacles in the matched obstacle pairs and the actual ground truth positions of the obstacles.

51. The device of claim 44, wherein, The decoding network includes a fourth decoding sub-network; the decoding module further includes: The fourth decoding submodule is used to decode all features in the aggregated features using the fourth decoding subnetwork to obtain the predicted driving experience information of the vehicle. The driving experience information includes at least two of the following: prediction information on whether a traffic violation has occurred, prediction information on whether a collision has occurred, prediction information on whether driving intervention is required, and prediction information on driving comfort; the prediction result includes the predicted driving experience information of the vehicle.

52. The device of claim 51, wherein, The predicted driving experience information includes predictive information on whether driving intervention is needed; the device also includes: The fifth training module is used to train the data processing model based on the difference between the predicted information on whether driving intervention is needed and the true information on whether driving intervention is needed.

53. The apparatus according to claim 51 or 52, further comprising: The reward value determination module is used to determine the reward value for the data processing model based on the weighted sum of at least two pieces of information included in the predicted driving experience information. The weight determination module is used to determine the reward weight based on the reward value and the truth information of whether driving intervention is required; The loss value determination module is used to determine the predicted loss value of the data processing model based on the reward weight and the difference between the predicted trajectory information and the actual trajectory information. as well as The sixth training module is used to train the data processing model based on the predicted loss value.

54. The apparatus of any one of claims 49-52, wherein, The sample data includes data obtained in scenarios where the vehicle is under closed-loop control.

55. The device of claim 44, wherein, The sample data includes data obtained in scenarios where the vehicle is under open-loop control.

56. The device of claim 44, wherein, The aggregation module includes: The first determining submodule is used to determine the embedding features of the map data to obtain map features; The second determining submodule is used to determine the embedding features of the prompt data, thereby obtaining the prompt features; and The aggregation submodule is used to aggregate the feature features, map features, and prompt features using the aggregation network to obtain the aggregated features.

57. The device of claim 56, wherein, The aggregation submodule is used for: The aggregation network is used to perform time-space dimension alternating aggregation of the element features, map features, and cue features based on an attention mechanism to obtain the aggregated features.

58. The device of claim 56 or 57, wherein, The feature characteristics include multiple features of the environmental element at multiple historical moments; The aggregation submodule includes: A downsampling unit is used to downsample the multiple historical moments to obtain multiple sampling moments; as well as An aggregation unit is used to aggregate the features of the environmental elements, the map features, and the cue features at multiple sampling times using the aggregation network to obtain the aggregated features.

59. The apparatus of claim 44, further comprising: The data conversion module is used to convert the raw data collected by the sensors on the vehicle into a bird's-eye view coordinate system to obtain pixel data and depth data. The data stitching module is used to stitch the pixel data and the depth data in the channel dimension to obtain perceptual data; as well as The data processing module is used to process the perceived data using a residual network to obtain the scene data included in the sample data.

60. The device of claim 44 or 59, wherein, The encoding module is used for: The perception network employs a deformable self-attention mechanism layer to iteratively encode the query data, scene data, and motion data obtained during real-time training a predetermined number of times (K) to obtain the feature characteristics of the environmental elements. Where K is a natural number greater than 1.

61. The device of claim 60, wherein, The perception network comprises K sequentially connected encoding subnetworks, each encoding subnetwork including the deformable self-attention mechanism layer and a mapping-normalization layer; the encoding module includes: An attention processing submodule is used to take the query data as the first query feature for the first iteration of encoding, and to process the i-th query feature, the scene data, and the motion data using a deformable self-attention mechanism layer included in the i-th encoding subnetwork of the K encoding subnetworks, in order to achieve the i-th iteration of encoding; and The fusion submodule is used to process the features obtained from the i-th iteration encoding and the i-th query feature using the mapping-normalization layer included in the i-th encoding subnetwork, to obtain the (i+1)-th query feature. The feature of the environmental element includes the (K+1)th query feature, where i is a natural number greater than or equal to 1 and less than or equal to K.

62. The device of claim 60, wherein, The query data includes: First query data for the anchor point of each element in the first category of environmental elements; and The second query data pertains to the center point of each cell in a predetermined road surface grid, which is constructed based on a predetermined area within the vehicle.

63. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 31.

64. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 31.

65. A computer program product comprising a computer program / instructions stored on at least one of a readable storage medium and an electronic device, wherein the computer program / instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 31.

66. An autonomous vehicle, comprising the means according to any one of claims 32 to 43.

Images