Model training and reasoning method, control device and autonomous vehicle

Abstract

The invention provides a model training and reasoning method, a control device and an automatic driving vehicle, and relates to the field of artificial intelligence, in particular to the field of automatic driving. The model training method comprises the following steps: extracting feature information of first sensor data collected under first sensor configuration; determining spatio-temporal information of the first sensor data; training a task model in the automatic driving scene by taking the feature information and the spatio-temporal information of the first sensor data as training data, so that the task model learns a first association relationship between the feature information and the spatio-temporal information of the sensor data and a task prediction result, the first association relationship is used for enabling the task model to automatically adjust the task prediction result when the spatio-temporal information changes due to the change of the sensor configuration. The model migration cost is reduced, and the compatibility of the model to sensor configuration is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, particularly to the field of autonomous driving, and especially to a model training and inference method, control device, and autonomous vehicle compatible with sensor configuration. Background Technology

[0002] Autonomous vehicles collect environmental information through various sensors installed on the vehicle, such as cameras and radar, and perform intelligent calculations and reasoning through various intelligent task models, such as target detection models, to achieve automatic driving control.

[0003] The task model needs to be trained before it can be used for inference and application. During training, it is necessary to specify the sensor configuration, such as the sensor model / parameters, installation position and angle, and vehicle type. Sensor data is collected based on the specific sensor configuration and used as training data to train the task model.

[0004] The task model trained in this way is highly dependent on the sensor configuration during training, resulting in poor generalization ability. Once the sensor configuration during inference application changes compared to the sensor configuration during training, the detection performance of the task model will be affected. However, the sensor configuration during vehicle production often changes compared to the vehicle design, leading to changes in the detection performance of the vehicle's task model.

[0005] Some related technologies involve re-collecting training data and retraining the task model based on the actual sensor configuration, but the model transfer cost is relatively high.

[0006] There are also some related technologies that, during model inference, rely on virtual sensor technology to first correct the data collected by the sensors before inputting it into the task model for inference. However, this method requires that the sensor model / parameters, vehicle type, and installation location be basically consistent with those used during model training. It only allows for a small range of manufacturing tolerances or installation errors in the sensors; otherwise, this method will fail, and the model's compatibility with sensor configurations is very limited. Summary of the Invention

[0007] To reduce model transfer costs and improve model compatibility with sensor configurations, this disclosure proposes a model training and inference scheme.

[0008] This disclosure proposes a model training method compatible with sensor configurations, including: extracting feature information from first sensor data collected under a first sensor configuration;

[0009] Determine the spatiotemporal information of the first sensor data;

[0010] The feature information and spatiotemporal information of the first sensor data are used as training data to train the task model in the autonomous driving scenario. This enables the task model to learn the first correlation between the feature information and spatiotemporal information of the sensor data and the task prediction result. The first correlation is used to enable the task model to automatically adjust the task prediction result when the spatiotemporal information changes due to changes in sensor configuration.

[0011] In some embodiments, the spatiotemporal information includes spatial location coding, and the spatiotemporal information for determining the first sensor data includes:

[0012] By using the projection matrix from the sensor coordinate system to the unified coordinate system, the position information of the first sensor data in the sensor coordinate system is projected onto the unified coordinate system, thereby obtaining the position information of the first sensor data in the unified coordinate system.

[0013] The spatial position code of the first sensor data is obtained by encoding the position information of the first sensor data in a unified coordinate system.

[0014] In some embodiments, the first sensor data includes at least one of image data from a camera and point cloud data from a radar;

[0015] Projecting the position information of the first sensor data in the sensor coordinate system to a unified coordinate system includes at least one of the following:

[0016] For each pixel in the camera's image data, multiple position points are sampled along the direction of the camera's optical center ray, and the position coordinates of the multiple position points are projected from the camera coordinate system to a unified coordinate system.

[0017] For each voxel in the radar point cloud data, the position coordinates of each voxel are projected from the radar coordinate system to a unified coordinate system.

[0018] In some embodiments, encoding the position information of the first sensor data in a unified coordinate system to obtain the spatial position encoding of the first sensor data includes:

[0019] The position information of the first sensor data in a unified coordinate system is encoded using trigonometric functions to obtain the first code of the first sensor data;

[0020] The first code of the first sensor data is encoded using a first depth network to obtain a second code of the first sensor data, which serves as the spatial location code of the first sensor data.

[0021] In some embodiments, the first encoding of the first sensor data in a unified coordinate system is obtained by encoding the position information of the first sensor data using trigonometric functions, including:

[0022] Based on preset temperature parameters, the frequency of each encoding dimension is generated;

[0023] Determine the frequency response of the position information of the first sensor data in a unified coordinate system to each encoded dimension;

[0024] Based on the parity attribute of each encoding dimension, a sine function or a cosine function is selected to encode the corresponding response of each encoding dimension;

[0025] The codes of the corresponding responses of each coding dimension are concatenated to form the first code of the first sensor data.

[0026] In some embodiments, the first encoding of the first sensor data in a unified coordinate system is obtained by encoding the position information of the first sensor data using trigonometric functions, including:

[0027] Before determining the response, the position information of the first sensor data in a unified coordinate system is scaled using a scaling factor of a trigonometric function.

[0028] In some embodiments, the spatiotemporal information includes time encoding, and the spatiotemporal information for determining the first sensor data includes:

[0029] Calculate the relative time between each two adjacent frames of data based on the timestamp of each frame of data in the first sensor data.

[0030] The relative time is encoded using a second deep network to obtain the time encoding of the first sensor data.

[0031] In some embodiments, the method further includes: encoding attribute data of the first sensor that acquires the first sensor data;

[0032] Training the task model includes:

[0033] The task model is trained using the feature information and spatiotemporal information of the first sensor data, as well as the encoding of the attribute data of the first sensor, as training data. This allows the task model to learn a second correlation between the feature information and spatiotemporal information of the sensor data, the encoding of the attribute data of the sensor, and the task prediction result. This second correlation is used to enable the task model to automatically adjust the task prediction result when the spatiotemporal information or the attribute data of the sensor changes due to changes in the sensor configuration.

[0034] In some embodiments, the first sensor configuration includes at least one of the following: the installation configuration of the first sensor for acquiring first sensor data and attribute data.

[0035] In some embodiments, the task model includes an object detection model, a space occupancy detection model, or a trajectory prediction model.

[0036] This disclosure provides several embodiments of a model inference method compatible with sensor configurations, including:

[0037] Extract feature information from the second sensor data acquired under the second sensor configuration;

[0038] Determine the spatiotemporal information of the second sensor data;

[0039] Using the first correlation learned by the task model, the feature information and spatiotemporal information of the second sensor data are inferred to obtain the first task prediction result.

[0040] This disclosure proposes a model inference method compatible with sensor configurations, including: extracting feature information from second sensor data collected under a second sensor configuration;

[0041] Determine the spatiotemporal information of the second sensor data;

[0042] The attribute data of the second sensor that collects data from the second sensor is encoded;

[0043] By utilizing the second association learned by the task model, reasoning is performed on the feature information and spatiotemporal information of the second sensor data, as well as the encoding of the attribute data of the second sensor, to obtain the second task prediction result.

[0044] In some embodiments, the spatiotemporal information includes spatial location coding, and the spatiotemporal information for determining the second sensor data includes:

[0045] By using the projection matrix from the sensor coordinate system to the unified coordinate system, the position information of the second sensor data in the sensor coordinate system is projected onto the unified coordinate system, thereby obtaining the position information of the second sensor data in the unified coordinate system.

[0046] The spatial position code of the second sensor data is obtained by encoding the position information of the second sensor data in a unified coordinate system.

[0047] In some embodiments, the spatiotemporal information includes time encoding, and the spatiotemporal information for determining the second sensor data includes:

[0048] Calculate the relative time between each two adjacent frames of data based on the timestamp of each frame of data in the second sensor data.

[0049] The relative time is encoded using a second deep network to obtain the time encoding of the second sensor data.

[0050] This disclosure provides a control device comprising one or more modules that perform the methods of various embodiments.

[0051] Some embodiments of this disclosure provide a control device including: a memory; and a processor coupled to the memory, the processor being configured to execute the methods of various embodiments based on instructions stored in the memory.

[0052] This disclosure provides some embodiments of an autonomous vehicle, including a control unit configured to perform the methods of various embodiments.

[0053] Some embodiments of this disclosure provide a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the methods of various embodiments.

[0054] This disclosure provides, in some embodiments, a computer program product including computer instructions that, when executed by a processor, implement the methods of various embodiments.

[0055] By incorporating spatiotemporal information into the training data, the task model learns the correlation between the spatiotemporal information of sensor data and the task prediction results. When the spatiotemporal information changes due to changes in sensor configuration, the task model automatically adjusts the task prediction results, enabling it to automatically adapt to new sensor configurations and new prediction results. This improves the task model's compatibility with sensor configurations, eliminating the need for retraining or requiring only small-step fine-tuning based on the existing model, thus reducing the transfer cost of the task model. Furthermore, by also incorporating sensor attribute data into the training data, the task model learns the correlation between sensor attribute data and task prediction results. When sensor attribute data changes due to changes in sensor configuration, the task model automatically adjusts the task prediction results, adapting to complex sensor configuration change scenarios. Attached Figure Description

[0056] The accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. This disclosure can be more clearly understood from the following detailed description with reference to the accompanying drawings.

[0057] Obviously, the accompanying drawings described below are merely some embodiments of this disclosure. Those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0058] Figure 1 A schematic diagram of the electrical architecture of an autonomous vehicle according to some embodiments of the present disclosure is shown.

[0059] Figure 2 The diagram shows the external structure of an autonomous vehicle according to some embodiments of the present disclosure.

[0060] Figure 3 A schematic diagram illustrating a model training method with compatible sensor configurations according to some embodiments of this disclosure is shown.

[0061] Figure 4 A schematic diagram illustrating a model inference method with compatible sensor configurations according to some embodiments of this disclosure is shown.

[0062] Figure 5 A schematic diagram illustrating a four-dimensional world representation using multiple sensors according to some embodiments of this disclosure is shown.

[0063] Figure 6 A schematic diagram of the structure of a control device according to some embodiments of the present disclosure is shown.

[0064] Figure 7 A schematic diagram of the structure of a control device according to some embodiments of the present disclosure is shown. Detailed Implementation

[0065] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0066] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0067] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0068] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0069] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0070] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0071] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0072] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0073] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0074] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0075] Furthermore, in order to avoid obscuring this disclosure with unnecessary details, only processing steps and / or device structures closely related to the scheme at least according to this disclosure are shown in the accompanying drawings, while other details that are not closely related to this disclosure are omitted.

[0076] Figure 1 The diagram illustrates the electrical architecture of an autonomous vehicle according to some embodiments of this disclosure. Autonomous vehicles may be, for example, driverless cars, unmanned delivery vehicles, unmanned vending vehicles, etc.

[0077] like Figure 1 As shown, the autonomous vehicle 100 in this embodiment includes, for example, an autonomous driving module 110 and a chassis module 120. Depending on the needs, it may also include a remote monitoring and streaming module 130 and a cargo box module 140. For example, vehicles requiring remote monitoring are equipped with the remote monitoring and streaming module 130; vehicles without such requirements may omit it. Similarly, vehicles requiring cargo (such as trucks) are equipped with the cargo box module 140; vehicles without such requirements (such as passenger cars) may omit it.

[0078] The autonomous driving module 110 may include, as needed, one or more of the following: a central processing unit (Orin or Xavier module) 111, a traffic light recognition camera 112, a front camera 1131, a rear camera 1132, a left camera 1133, a right camera 1134, a LiDAR 114, a front blind spot radar 1151, a rear blind spot radar 1152, a left blind spot radar 1153, and a right blind spot radar 1154, a positioning module (such as BeiDou, GPS, etc.) 116, an inertial navigation unit 117, and a switch 118. Each camera can communicate with the autonomous driving module. To improve transmission speed and reduce wiring, GMSL (Gigabit Multimedia Serial Links) links can be used for communication. The central processing unit 111 can be implemented using a general-purpose central processing unit, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, or transistors, etc. The central processing unit 111 can be configured to perform autonomous driving control.

[0079] The chassis module 120 may include, as needed, one or more of the following: battery 121, power management device 122, chassis controller 123, motor driver 124, drive motor 125, and communication module 126. Battery 121 provides power to the entire autonomous vehicle system. Battery 121 includes a main battery 1211 and a standby battery 1212. When the autonomous vehicle is running, the main battery 1211 powers each module of the autonomous vehicle. When the autonomous vehicle is in standby mode, the standby battery 1212 powers the central processing unit 111 and the communication module 126. The power management device 122 converts the output of battery 121 into different voltage levels usable by each module and controls power-on and power-off. The chassis controller 123 receives motion commands from the autonomous driving module 110 and controls the autonomous vehicle's steering, forward, reverse, and braking. The communication module 126 communicates with a backend server, enabling remote control of the autonomous vehicle by backend operators. The communication module 126 includes a cellular wireless communication device 1261 and a radio frequency communication device 1262. Cellular wireless communication device 1261 communicates using cellular wireless communication technology, such as 2G (second generation), 3G (third generation), 4G (fourth generation), or 5G (fifth generation) cellular wireless communication technology. Radio frequency communication device 1262 communicates using radio frequency communication technology.

[0080] The remote monitoring streaming module 130 may include, as needed, one or more of the following: a front monitoring camera 1311, a rear monitoring camera 1312, a left monitoring camera 1313, a right monitoring camera 1314, and a streaming module 132. The streaming module 132 transmits the video data captured by the monitoring cameras 1311-1314 to the backend server for viewing by backend operators.

[0081] The cargo box module 140 may include a cargo box 141 as needed, which is a cargo-carrying device for the autonomous vehicle. The cargo box module 140 also includes a display and interaction module 142 for interaction between the autonomous vehicle and the user. Users can perform operations such as picking up items, storing goods, and purchasing goods through the display and interaction module 142. The type of cargo box 141 can be changed according to actual needs. For example, in a logistics scenario, the cargo box may include multiple sub-boxes of different sizes, which can be used to load goods for delivery. In a retail scenario, the cargo box can be set as a transparent box so that users can see the products for sale.

[0082] Figure 2 The diagram shows the external structure of an autonomous vehicle according to some embodiments of the present disclosure. Figure 2 The diagram shows the external structure of an autonomous vehicle capable of carrying cargo. Autonomous vehicles with different functions can have different external structures; for example, the external structure of a passenger-carrying autonomous vehicle can be referenced from that of a car. Figure 2 As shown, from the current perspective, the chassis 210, cargo box 141, display and interaction module 142, right-side camera 1134, lidar 114, rear blind spot radar 1152, left-side blind spot radar 1153, right-side blind spot radar 1154 and other equipment of the autonomous vehicle 200 can be seen.

[0083] The following is combined with Figure 3 and Figure 4 The model training and inference methods with compatible sensor configurations are described separately. The terms "first" and "second" in "first / second sensor configuration" and "first / second sensor data" are used to distinguish between the training and inference phases, indicating that different sensor configurations and sensor data can be used in the training and inference phases respectively.

[0084] Figure 3 A schematic diagram illustrating a model training method with compatible sensor configurations according to some embodiments of this disclosure is shown.

[0085] like Figure 3 As shown, depending on the training data, the model training method includes steps 310, 320, 340 (340-1), or steps 310, 320, 330, 340 (340-2).

[0086] Step 310: Extract feature information from the first sensor data collected under the first sensor configuration.

[0087] The first sensor configuration refers to a certain sensor configuration, such as, but not limited to, the installation configuration of the first sensor for collecting data, and at least one piece of information in the attribute data. The first sensor can be one or more sensors, including, but not limited to, various radars and cameras installed in autonomous vehicles. The sensor installation configuration includes, but is not limited to, installation position and angle, and the vehicle model; different models of autonomous vehicles have different structures and dimensions. The sensor attribute data includes, but is not limited to, the sensor model and physical parameters. The camera's physical parameters include, but are not limited to, focal length, field of view (FOV), image sensor size, pixel size, intrinsic parameters, and extrinsic parameters. The physical parameters of a lidar include, but are not limited to, scanning principle (solid-state, semi-solid-state, or mechanical), effective line count (the number of independent laser channels that the lidar simultaneously emits / receives in the vertical direction), point frequency (the number of effective point clouds output by the radar per unit time), intrinsic parameters, extrinsic parameters, horizontal and vertical angular resolution, ranging performance of different channels, and reflection intensity curves. The physical parameters of a millimeter-wave radar include, but are not limited to, point frequency, intrinsic parameters, extrinsic parameters, horizontal and vertical angular resolution, and velocity resolution.

[0088] The first sensor data refers to data from one or more sensors, including but not limited to image data from a camera and point cloud data from radar. The feature information of the first sensor data can be obtained by extracting features from it using a feature extraction model. This feature extraction model can be, for example, a model built on a deep network, such as, but not limited to, a CNN (Convolutional Neural Network) model or a Transformer model. Before feature extraction, the first sensor data can be preprocessed, for example, by removing distortion from the camera image data or performing motion compensation on the radar point cloud data.

[0089] Step 320: Determine the spatiotemporal information of the first sensor data.

[0090] The spatiotemporal information of sensor data represents its spatial location and temporal information. The spatiotemporal information of sensor data is obtained by concatenating these two components. This spatiotemporal information includes, for example, spatial location encoding and temporal encoding. Spatial location encoding converts the spatial location information of sensor data into a high-dimensional feature vector suitable for model input. Temporal encoding converts the temporal information of sensor data into a high-dimensional feature vector suitable for model input.

[0091] The determination of the spatial location code of the first sensor data includes the following steps (1) and (2).

[0092] (1) By using the projection matrix from the sensor coordinate system to the unified coordinate system, the position information of the first sensor data in the sensor coordinate system is projected onto the unified coordinate system to obtain the position information of the first sensor data in the unified coordinate system.

[0093] In this context, the sensor coordinate system is a local coordinate system with the sensor itself as the origin. The unified coordinate system is a defined common reference system that can be set, such as the vehicle coordinate system for an autonomous vehicle or the radar coordinate system. The projection matrix is ​​used to implement coordinate transformation from the sensor coordinate system to the unified coordinate system, and can be obtained through pre-calibration. For radar, the projection matrix can be, for example, a rigid body transformation matrix determined based on extrinsic parameters. For cameras, the projection matrix can be, for example, a composite matrix determined based on intrinsic and extrinsic parameters.

[0094] Spatial position coding based on a unified coordinate system can eliminate the spatial heterogeneity of multiple sensors, making the position information of different sensors comparable and fused under the same reference system.

[0095] (2) Encode the position information of the first sensor data in a unified coordinate system to obtain the spatial position code of the first sensor data, including the following steps (2-1) and (2-2).

[0096] (2-1) Use trigonometric functions to encode the position information of the first sensor data in a unified coordinate system to obtain the first code of the first sensor data.

[0097] An exemplary trigonometric function encoding method is described below.

[0098] The position vector of the first sensor data in the unified coordinate system is set as: p = (x, y, z)∈R³;

[0099] The encoding dimension is set to: d = 128 (per coordinate);

[0100] The temperature parameter is set to: T = 10000;

[0101] The scaling factor is set to: s = 2π.

[0102] First, optionally, the position information of the first sensor data in a unified coordinate system can be scaled using the scaling factor of the trigonometric function, p' = s · p = (2πx, 2πy, 2πz), p' = (x', y', z').

[0103] Next, based on preset temperature parameters, the frequency of each encoding dimension is generated.

[0104] , where the encoding dimension i∈{0, 1, ..., d-1}, and ^ represents the exponentiation operation.

[0105] Next, the frequency response of the first sensor data's position information in a unified coordinate system to each encoded dimension is determined. The frequency response is calculated for each coordinate separately:

[0106] X-coordinate encoding:

[0107] Y-coordinate encoding:

[0108] Z-coordinate encoding:

[0109] Next, based on the parity attribute of each encoding dimension, a sine function or a cosine function is selected to encode the corresponding response of each encoding dimension.

[0110] For even index i = 2k:

[0111]

[0112]

[0113]

[0114] For odd index i = 2k+1:

[0115]

[0116] Next, the codes of the corresponding responses from each coding dimension are concatenated to form the first code of the first sensor data:

[0117]

[0118] The final positional encoding PE(x,y,z) is a 384 (128×3) dimensional vector, i.e. .

[0119] By controlling the frequency distribution through temperature parameters, we can adapt to spatial relationships at different scales (such as close-up details and far-up globals); by alternating between sine and cosine functions in odd and even dimensions, we can avoid feature redundancy and improve the expressive power of the encoding.

[0120] (2-2) The first code of the first sensor data is encoded using the first depth network to obtain the second code of the first sensor data, which serves as the spatial location code of the first sensor data.

[0121] The first deep network is, for example, but not limited to, CNN, Transformer, PointNet, etc. The first encoding of the first sensor data is input into the first deep network for processing, and the second encoding of the first sensor data is output.

[0122] By using trigonometric function encoding, global spatial frequency features are captured, which is robust to translation and scaling; by using deep network encoding, high-frequency features are converted into fixed-dimensional vectors suitable for model input, introducing non-linear expressive power.

[0123] For example, for each pixel in the camera's image data, multiple location points are sampled along the direction of the camera's optical center ray. Since the depth of each pixel in the image is unknown, sampling multiple points simulates multiple possible locations of objects on this ray in space. The position coordinates of the multiple location points are projected from the camera coordinate system to the unified coordinate system using a pre-calibrated projection matrix from the camera to the unified coordinate system. The position coordinates of the multiple location points in the unified coordinate system are first encoded using trigonometric functions, and then encoded using a deep network to convert them into high-dimensional camera image spatial position codes.

[0124] For example, for each voxel in the radar point cloud data, the position coordinates of each voxel are projected from the radar coordinate system to the unified coordinate system through a pre-calibrated projection matrix of the radar to the unified coordinate system; the position coordinates of each voxel in the unified coordinate system are first encoded using trigonometric functions, and then encoded using a deep network, and converted into a high-dimensional radar point cloud spatial position code.

[0125] To address the unique characteristics of cameras (2D pixels with no depth) and radar (sparse 3D point clouds), differentiated spatial sampling and projection schemes are provided. For cameras, multiple depth points are sampled along the optical center ray to complete the 3D depth information of the 2D image, achieving pixel-level mapping to 3D physical space. For radar, voxelization processing is performed to convert the sparse point cloud into structured spatial units, improving the stability of position encoding.

[0126] The determination of the time encoding of the first sensor data includes: calculating the relative time between each two adjacent frames based on the timestamp of each frame in the first sensor data, and possibly normalizing the relative time; encoding the relative time using a second deep network to obtain the time encoding of the first sensor data. The second deep network is, for example, but not limited to, a recurrent neural network (RNN) or a Transformer. The relative time of each frame is input into the second deep network for processing, and the time encoding of the first sensor data is output.

[0127] By replacing absolute time with relative time (time difference between adjacent frames), the complexity and stability of temporal encoding are reduced, adapting to frame rate differences between different sensors. The second deep network encodes the relative time and outputs a fixed-dimensional vector, which is convenient for model processing.

[0128] Step 330: Encode the attribute data of the first sensor that collected the first sensor data.

[0129] As mentioned earlier, sensor attribute data includes, but is not limited to, sensor model, physical parameters, etc. The attribute data of the first sensor is processed using, for example, an embedding layer of a large language model to obtain embedded codes.

[0130] Step 340: Train the task model in the autonomous driving scenario.

[0131] The task models can be built based on the Transformer architecture, including but not limited to object detection models, space occupancy detection models, or trajectory prediction models. The feature information of the sensor data with added spatiotemporal information is called a four-dimensional world representation, and the training data in subsequent steps 340-1 and 340-2 are all four-dimensional world representations. The inputs to various task models are four-dimensional world representations; the object detection model outputs object detection boxes, which can be three-dimensional, including information such as center point position, size, orientation, velocity, and acceleration; the space occupancy detection model outputs the occupancy status of each spatial location within a preset spatial range, such as whether it is occupied and the type of content occupying it; the trajectory prediction model outputs the predicted motion trajectory of the vehicle itself or other moving obstacles in the surrounding area, used to control the movement of the vehicle.

[0132] Step 340-1: Use the feature information and spatiotemporal information of the first sensor data as training data to train the task model in the autonomous driving scenario, so that the task model learns the first correlation between the feature information and spatiotemporal information of the sensor data and the task prediction result. The first correlation is used to enable the task model to automatically adjust the task prediction result when the spatiotemporal information changes due to changes in sensor configuration.

[0133] During task model training, based on the feature and spatiotemporal information of the input sensor data, the parameters of the task model can be iteratively updated using the gap between the task prediction results and the task labels. This gradually reduces the gap between the task prediction results and the task labels, bringing them closer together. The parameters of the trained task model can represent the primary correlation between the learned feature and spatiotemporal information of the sensor data and the task prediction results. Based on this primary correlation, the task model automatically adjusts the task prediction results when the spatiotemporal information changes due to changes in sensor configuration.

[0134] During the training of the object detection model, the parameters of the object detection model can be iteratively updated using the difference information between the object detection box and the target box, so that the difference between the object detection box and the target box gradually decreases and tends to be consistent.

[0135] During the training of the space occupancy detection model, the parameters of the space occupancy detection model can be iteratively updated by using the difference between the space occupancy detection results and the space occupancy labels, so that the difference between the space occupancy detection results and the space occupancy labels gradually decreases and tends to be consistent.

[0136] During the training of the trajectory prediction model, the parameters of the trajectory prediction model can be iteratively updated by using the difference between the predicted motion trajectory and the motion trajectory label, so that the difference between the predicted motion trajectory and the motion trajectory label gradually decreases and tends to be consistent.

[0137] Step 340-2: Use the feature information and spatiotemporal information of the first sensor data, as well as the encoding of the attribute data of the first sensor, as training data to train the task model in the autonomous driving scenario. This enables the task model to learn the second correlation between the feature information and spatiotemporal information of the sensor data, the encoding of the attribute data of the sensor, and the task prediction result. The second correlation is used to enable the task model to automatically adjust the task prediction result when the spatiotemporal information or the attribute data of the sensor changes due to changes in the sensor configuration.

[0138] During task model training, based on the feature information, spatiotemporal information, and attribute data encoding of the input sensor data, the parameters of the task model can be iteratively updated using the gap between the task prediction results and the task labels. This gradually reduces the gap between the task prediction results and the task labels, bringing them closer together. The parameters of the trained task model can represent the second correlation between the learned feature information, spatiotemporal information, and attribute data encoding of the sensor data and the task prediction results. Based on this second correlation, the task model automatically adjusts the task prediction results when the spatiotemporal information or sensor attribute data changes due to changes in sensor configuration.

[0139] By incorporating spatiotemporal information into the training data, the task model learns the correlation between the spatiotemporal information of sensor data and the task prediction results. When the spatiotemporal information changes due to changes in sensor configuration, the task model automatically adjusts the task prediction results, enabling it to automatically adapt to new sensor configurations and new prediction results. This improves the compatibility of the task model with sensor configurations, eliminating the need for retraining or requiring only fine-tuning through small steps on the existing model, thus reducing the transfer cost of the task model.

[0140] Furthermore, by incorporating sensor attribute data into the training data, the task model learns the correlation between sensor attribute data and task prediction results. When sensor attribute data changes due to sensor configuration changes, the task model automatically adjusts the task prediction results to adapt to complex sensor configuration change scenarios.

[0141] Figure 4 A schematic diagram illustrating a model inference method with compatible sensor configurations according to some embodiments of this disclosure is shown.

[0142] like Figure 4 As shown, depending on the inference data, the model inference method includes steps 410, 420, 440 (440-1), or steps 410, 420, 430, 440 (440-2).

[0143] Step 410: Extract feature information from the second sensor data collected under the second sensor configuration.

[0144] The technical meaning of the second sensor configuration is similar to that of the first sensor configuration. Both refer to a certain sensor configuration, such as the sensor's installation configuration, attribute data, etc. However, the second sensor configuration can be different from the first sensor configuration. For example, one or more of the following configurations may change: sensor installation position, angle, vehicle model, sensor model, or physical parameters.

[0145] The technical meaning and acquisition method of the second sensor data and its characteristic information are similar to those of the first sensor data and its characteristic information, and will not be repeated here.

[0146] Step 420: Determine the spatiotemporal information of the second sensor data.

[0147] The spatiotemporal information of the second sensor data has a similar technical meaning and determination method to the spatiotemporal information of the first sensor data, and will not be elaborated here. The spatiotemporal information of sensor data includes, for example, spatial location encoding and time encoding. The spatial location encoding and time encoding of the sensor data are concatenated to form the spatiotemporal information of the sensor data.

[0148] The determination of the spatial location encoding of the second sensor data includes: projecting the position information of the second sensor data in the sensor coordinate system onto the unified coordinate system using a projection matrix from the sensor coordinate system to the unified coordinate system, thus obtaining the position information of the second sensor data in the unified coordinate system; and encoding the position information of the second sensor data in the unified coordinate system to obtain the spatial location encoding of the second sensor data. For example, the encoding method may be: using trigonometric functions to encode the position information of the second sensor data in the unified coordinate system to obtain a first encoding of the second sensor data; and using a first depth network to encode the first encoding of the second sensor data to obtain a second encoding of the second sensor data, which serves as the spatial location encoding of the second sensor data. For specific projection transformation and encoding processes, please refer to [link to documentation]. Figure 3 Examples are not described here.

[0149] The determination of the time encoding of the second sensor data includes: calculating the relative time between each two adjacent frames based on the timestamp of each frame in the second sensor data; and encoding the relative time using a second deep network to obtain the time encoding of the second sensor data. For the specific encoding process, please refer to [link to documentation]. Figure 3 Examples are not described here.

[0150] Step 430: Encode the attribute data of the second sensor that collects the data from the second sensor.

[0151] As mentioned earlier, the sensor's attribute data includes, but is not limited to, the sensor's model and physical parameters. The attribute data of the second sensor is processed using, for example, an embedding layer of a large language model to obtain embedded codes.

[0152] Step 440: The task model performs inference.

[0153] The task model can be built based on the Transformer architecture, and includes, but is not limited to, object detection models, space occupancy detection models, or trajectory prediction models. See [link to documentation] for the functions and inputs / outputs of various models. Figure 3 Examples are not described here.

[0154] Step 440-1: Using the first correlation learned by the task model, reason about the feature information and spatiotemporal information of the second sensor data to obtain the first task prediction result.

[0155] That is, the feature information and spatiotemporal information of the second sensor data are input into the task model trained in step 340-1 for processing, and the prediction result of the first task is output.

[0156] By applying the first association learned during the training phase to the inference phase, rapid deployment across sensor configurations is achieved. When spatiotemporal information changes due to changes in sensor configuration, the task model automatically adjusts the task prediction results, enabling it to automatically adapt to new sensor configurations and new prediction results. This improves the compatibility of the task model with sensor configurations, eliminating the need for retraining or requiring only fine-tuning through small steps on the basis of the original model, thus reducing the transfer cost of the task model.

[0157] Step 440-2: Using the second association learned by the task model, reason about the feature information and spatiotemporal information of the second sensor data and the encoding of the attribute data of the second sensor to obtain the prediction result of the second task.

[0158] That is, the feature information, spatiotemporal information, and attribute data of the second sensor are encoded and input into the task model trained in step 340-2 for processing, and the prediction result of the second task is output.

[0159] By applying the second association learned during the training phase to the inference phase, rapid deployment across sensor configurations is achieved. When spatiotemporal information or sensor attribute data changes due to changes in sensor configuration, the task model automatically adjusts the task prediction results, enabling it to automatically adapt to new sensor configurations and new prediction results. This improves the compatibility of the task model with sensor configurations, eliminating the need for retraining or requiring only fine-tuning through small steps on the basis of the original model, thus reducing the transfer cost of the task model.

[0160] Figure 5 The diagram illustrates a multi-sensor four-dimensional world representation of some embodiments of this disclosure. The feature information of sensor data with added spatiotemporal information is called the four-dimensional world representation. Image data from a camera undergoes feature extraction to obtain image features, which are then concatenated with the spatiotemporal encoding (spatial location encoding and data encoding) of the image data to obtain the four-dimensional world representation of the image data. Similarly, point cloud data from a radar undergoes feature extraction to obtain point cloud features, which are then concatenated with the spatiotemporal encoding (spatial location encoding and data encoding) of the point cloud data to obtain the four-dimensional world representation of the point cloud data. These four-dimensional world representations can be used as input data for a task model and applied to the model training or inference process.

[0161] Figure 6 A schematic diagram of the structure of a control device according to some embodiments of this disclosure is shown. For example... Figure 6 As shown, the control device 600 of this embodiment includes a memory 610 and a processor 620 coupled to the memory 610. The processor 620 is configured to execute the methods, model inference methods and / or model training methods of any of the foregoing embodiments based on instructions stored in the memory 610.

[0162] The control device 600 may also include an input / output interface 630, a network interface 640, a storage interface 650, etc. These interfaces 630, 640, 650, as well as the memory 610 and the processor 620, can be connected, for example, via a bus 660.

[0163] The memory 610 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, the operating system, application programs, boot loader, and other programs.

[0164] The processor 620 can be implemented using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, or transistors, etc. The processor 620 can be the central processing unit in an autonomous vehicle.

[0165] The input / output interface 630 provides a connection interface for input / output devices such as monitors, mice, keyboards, and touchscreens. The network interface 640 provides a connection interface for various networked devices. The storage interface 650 provides a connection interface for external storage devices such as SD cards and USB flash drives. The bus 660 can use any bus architecture from a variety of bus structures. For example, bus architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.

[0166] Figure 7 The diagram illustrates the structure of a control device according to some embodiments of this disclosure. The control device includes one or more modules that execute the methods, model inference methods, and / or model training methods described in any of the foregoing embodiments. Figure 7 As shown, the control device 700 in this embodiment includes a training module 710 and / or an inference module 720, but is not limited to this division method.

[0167] The training module 710 is configured to extract feature information from the first sensor data collected under the first sensor configuration; determine the spatiotemporal information of the first sensor data; and use the feature information and spatiotemporal information of the first sensor data as training data to train a task model in an autonomous driving scenario, so that the task model learns a first correlation between the feature information and spatiotemporal information of the sensor data and the task prediction result. The first correlation is used to enable the task model to automatically adjust the task prediction result when the spatiotemporal information changes due to the change of the sensor configuration.

[0168] The spatiotemporal information includes spatial position encoding. The training module 710 is configured to project the position information of the first sensor data in the sensor coordinate system to the unified coordinate system using a projection matrix from the sensor coordinate system to the unified coordinate system, thereby obtaining the position information of the first sensor data in the unified coordinate system. For example, for each pixel in the camera's image data, multiple position points are sampled along the direction of the camera's optical center ray, and the position coordinates of the multiple position points are projected from the camera coordinate system to the unified coordinate system. For each voxel in the radar's point cloud data, the position coordinates of each voxel are projected from the radar coordinate system to the unified coordinate system. The position information of the first sensor data in the unified coordinate system is encoded to obtain the spatial position encoding of the first sensor data. For example, trigonometric functions are used to encode the position information of the first sensor data in the unified coordinate system to obtain a first encoding of the first sensor data. A first depth network is used to encode the first encoding of the first sensor data to obtain a second encoding of the first sensor data, which serves as the spatial position encoding of the first sensor data.

[0169] The training module 710 is configured to scale the position information of the first sensor data in a unified coordinate system using a scaling factor of a trigonometric function; generate the frequency of each encoding dimension according to a preset temperature parameter; determine the response of the position information of the first sensor data in the unified coordinate system to the frequency of each encoding dimension; select a sine function or a cosine function to encode the corresponding response of each encoding dimension according to the parity attribute of each encoding dimension; and concatenate the codes of the corresponding responses of each encoding dimension as the first code of the first sensor data.

[0170] The spatiotemporal information includes time encoding. The training module 710 is configured to calculate the relative time of each two adjacent frames of data based on the timestamp of each frame of data in the first sensor data; and to encode the relative time using a second deep network to obtain the time encoding of the first sensor data.

[0171] The training module 710 is configured to encode the attribute data of the first sensor that collects the first sensor data; and to use the feature information and spatiotemporal information of the first sensor data and the encoding of the attribute data of the first sensor as training data to train the task model, so that the task model learns a second correlation between the feature information and spatiotemporal information of the sensor data, the encoding of the sensor attribute data and the task prediction result. The second correlation is used to enable the task model to automatically adjust the task prediction result when the spatiotemporal information or the sensor attribute data changes due to changes in sensor configuration.

[0172] The inference module 720 is configured to extract feature information from the second sensor data collected under the second sensor configuration; determine the spatiotemporal information of the second sensor data; and use the first correlation learned by the task model to infer the feature information and spatiotemporal information of the second sensor data to obtain the first task prediction result.

[0173] The inference module 720 is configured to extract feature information from the second sensor data collected under the second sensor configuration; determine the spatiotemporal information of the second sensor data; encode the attribute data of the second sensor that collected the second sensor data; and use the second association learned by the task model to infer the feature information, spatiotemporal information and the encoding of the attribute data of the second sensor data to obtain the second task prediction result.

[0174] The spatiotemporal information includes spatial location encoding. The inference module 720 is configured to project the position information of the second sensor data in the sensor coordinate system to the unified coordinate system through a projection matrix from the sensor coordinate system to the unified coordinate system, thereby obtaining the position information of the second sensor data in the unified coordinate system; and to encode the position information of the second sensor data in the unified coordinate system to obtain the spatial location encoding of the second sensor data.

[0175] The spatiotemporal information includes time encoding. The inference module 720 is configured to calculate the relative time of each two adjacent frames of data based on the timestamp of each frame of data in the second sensor data; and to encode the relative time using a second deep network to obtain the time encoding of the second sensor data.

[0176] This disclosure provides an autonomous driving vehicle, including control devices 600 and 700, configured to execute the methods, model inference methods, and / or model training methods described in any of the foregoing embodiments. The control devices 600 and 700 may, for example, be located in the autonomous driving module of the autonomous driving vehicle and are electrically connected to various sensors and a central processing unit.

[0177] This disclosure provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the methods, model inference methods, and / or model training methods described in any of the foregoing embodiments. The storage medium may be, for example, a non-transitory computer-readable storage medium.

[0178] This disclosure provides a computer program product, including computer instructions that, when executed by a processor, implement the methods, model inference methods, and / or model training methods described in any of the foregoing embodiments.

[0179] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more (non-transitory) computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, cloud storage, etc.) containing computer program code. A computer program product should be understood as a software product that primarily implements its solution through a computer program.

[0180] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0181] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0182] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

Claims

1. A model training method compatible with sensor configurations, comprising: Extract feature information from the first sensor data acquired under the first sensor configuration; Determine the spatiotemporal information of the first sensor data; The feature information and spatiotemporal information of the first sensor data are used as training data to train the task model in the autonomous driving scenario. This enables the task model to learn the first correlation between the feature information and spatiotemporal information of the sensor data and the task prediction result. The first correlation is used to enable the task model to automatically adjust the task prediction result when the spatiotemporal information changes due to changes in sensor configuration.

2. The method according to claim 1, wherein, The spatiotemporal information includes spatial location encoding, and the spatiotemporal information used to determine the first sensor data includes: By using the projection matrix from the sensor coordinate system to the unified coordinate system, the position information of the first sensor data in the sensor coordinate system is projected onto the unified coordinate system, thereby obtaining the position information of the first sensor data in the unified coordinate system. The spatial position code of the first sensor data is obtained by encoding the position information of the first sensor data in a unified coordinate system.

3. The method according to claim 2, wherein, The first sensor data includes at least one of image data from a camera and point cloud data from a radar. Projecting the position information of the first sensor data in the sensor coordinate system to a unified coordinate system includes at least one of the following: For each pixel in the camera's image data, multiple position points are sampled along the direction of the camera's optical center ray, and the position coordinates of the multiple position points are projected from the camera coordinate system to a unified coordinate system. For each voxel in the radar point cloud data, the position coordinates of each voxel are projected from the radar coordinate system to a unified coordinate system.

4. The method according to claim 2, wherein, Encoding the position information of the first sensor data in a unified coordinate system to obtain the spatial position code of the first sensor data includes: The position information of the first sensor data in a unified coordinate system is encoded using trigonometric functions to obtain the first code of the first sensor data; The first code of the first sensor data is encoded using a first depth network to obtain a second code of the first sensor data, which serves as the spatial location code of the first sensor data.

5. The method according to claim 4, wherein, Encoding the position information of the first sensor data in a unified coordinate system using trigonometric functions yields the first encoding of the first sensor data, including: Based on preset temperature parameters, the frequency of each encoding dimension is generated; Determine the frequency response of the position information of the first sensor data in a unified coordinate system to each encoded dimension; Based on the parity attribute of each encoding dimension, a sine function or a cosine function is selected to encode the corresponding response of each encoding dimension; The codes of the corresponding responses of each coding dimension are concatenated to form the first code of the first sensor data.

6. The method according to claim 5, wherein, Encoding the position information of the first sensor data in a unified coordinate system using trigonometric functions yields the first encoding of the first sensor data, including: Before determining the response, the position information of the first sensor data in a unified coordinate system is scaled using a scaling factor of a trigonometric function.

7. The method according to any one of claims 1-6, wherein, The spatiotemporal information includes time encoding, and the spatiotemporal information for determining the first sensor data includes: Calculate the relative time between each two adjacent frames of data based on the timestamp of each frame of data in the first sensor data. The relative time is encoded using a second deep network to obtain the time encoding of the first sensor data.

8. The method according to any one of claims 1-6, further comprising: The attribute data of the first sensor that collects data from the first sensor is encoded; Training the task model includes: The task model is trained using the feature information and spatiotemporal information of the first sensor data, as well as the encoding of the attribute data of the first sensor, as training data. This allows the task model to learn a second correlation between the feature information and spatiotemporal information of the sensor data, the encoding of the attribute data of the sensor, and the task prediction result. This second correlation is used to enable the task model to automatically adjust the task prediction result when the spatiotemporal information or the attribute data of the sensor changes due to changes in the sensor configuration.

9. The method according to any one of claims 1-6, wherein, The first sensor configuration includes at least one of the following: the installation configuration of the first sensor for collecting first sensor data, and attribute data.

10. The method according to any one of claims 1-6, wherein, The task model includes an object detection model, a space occupancy detection model, or a trajectory prediction model.

11. A model inference method compatible with sensor configuration, comprising: Extract feature information from the second sensor data acquired under the second sensor configuration; Determine the spatiotemporal information of the second sensor data; Using the first correlation learned by the task model, inference is performed on the feature information and spatiotemporal information of the second sensor data to obtain the first task prediction result. The task model is obtained by training using the method described in any one of claims 1-7 and 9-10.

12. A model inference method compatible with sensor configurations, comprising: Extract feature information from the second sensor data acquired under the second sensor configuration; Determine the spatiotemporal information of the second sensor data; The attribute data of the second sensor that collects data from the second sensor is encoded; By utilizing the second association learned from the task model, inference is performed on the feature information, spatiotemporal information, and attribute data encoding of the second sensor data to obtain the second task prediction result. The task model is obtained by training using the method described in claim 8.

13. The method according to claim 11 or 12, wherein, The spatiotemporal information includes spatial location encoding, and the spatiotemporal information for determining the second sensor data includes: By using the projection matrix from the sensor coordinate system to the unified coordinate system, the position information of the second sensor data in the sensor coordinate system is projected onto the unified coordinate system, thus obtaining the position information of the second sensor data in the unified coordinate system. The spatial position code of the second sensor data is obtained by encoding the position information of the second sensor data in a unified coordinate system.

14. The method according to claim 11 or 12, wherein, The spatiotemporal information includes time encoding, and the spatiotemporal information for determining the second sensor data includes: Calculate the relative time between each two adjacent frames of data based on the timestamp of each frame of data in the second sensor data. The relative time is encoded using a second deep network to obtain the time encoding of the second sensor data.

15. A control device, comprising: One or more modules that perform the method according to any one of claims 1-14.

16. A control device, comprising: Memory; And a processor coupled to the memory, the processor being configured to perform the method of any one of claims 1-14 based on instructions stored in the memory.

17. An autonomous vehicle, including a control unit configured to perform the method of any one of claims 1-14.

18. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method of any one of claims 1-14.

19. A computer program product comprising computer instructions that, when executed by a processor, implement the method of any one of claims 1-14.

Images