Control instruction generation method and device, automatic driving system and vehicle

Abstract

The embodiment of the invention relates to the technical field of vehicles, and discloses a control instruction generation method and device, an automatic driving system and a vehicle, which are used for improving the accuracy and scene adaptability of automatic driving decisions of the vehicle. The method comprises the following steps: acquiring multi-dimensional sensing data when a vehicle runs; the multi-dimensional perception data comprises the following data of at least two dimensions: image data, point cloud data, radar data, GPS / IMU data and map data; performing dimension reduction processing on the multi-dimensional sensing data, and projecting the multi-dimensional sensing data to a shared high-dimensional vector space; vector aggregation is carried out in the high-dimensional vector space, so that the distances of feature vectors describing the same physical object in data of different dimensions of the multi-dimensional perception data in the high-dimensional vector space are close; generating a target control instruction matched with the three-dimensional scene based on scene information of the three-dimensional scene restored from the high-dimensional vector space; the target control instruction is used for controlling the vehicle to realize a driving behavior conforming to a scene geometric structure and dynamic constraints in the three-dimensional scene.

Description

Technical Field

[0001] This invention relates to the field of vehicle technology, specifically to a method and apparatus for generating control commands, an automated driving system, and a vehicle. Background Technology With the continuous development of autonomous driving technology and the ongoing improvement in the accuracy of perception systems, the decision-making systems of autonomous vehicles still face severe challenges due to the high complexity and uncertainty of real-world traffic scenarios. Decision-making algorithms in related technologies often struggle to make completely accurate, human-like judgments when dealing with long-tail scenarios, dynamic interactive behaviors, and multi-objective game scenarios, easily leading to problems such as decision delays, unreasonable path planning, and even safety hazards. Particularly in scenarios such as complex urban intersections, unprotected left turns, and sudden obstacle avoidance, the inaccuracy of decision-making significantly increases the safety risks of the system, becoming one of the key bottlenecks restricting the large-scale application of autonomous driving technology.

[0002] Therefore, improving the accuracy and scenario adaptability of autonomous driving decision-making systems has become an urgent technical problem to be solved. Summary of the Invention

[0003] In view of the shortcomings of the above-mentioned related technologies, the purpose of this application is to provide a method, device, autonomous driving system and vehicle for generating control commands, so as to improve the accuracy and scenario adaptability of vehicle autonomous driving decisions.

[0004] In a first aspect, embodiments of this application provide a method for generating control commands, comprising: acquiring multi-dimensional perception data during vehicle operation; the multi-dimensional perception data includes data in at least two of the following dimensions: image data, point cloud data, radar data, GPS / IMU data, and map data; performing dimensionality reduction processing on the multi-dimensional perception data and projecting it onto a shared high-dimensional vector space; performing vector aggregation in the high-dimensional vector space so that the distance between feature vectors describing the same physical object in different dimensions of the multi-dimensional perception data in the high-dimensional vector space is less than a preset threshold; generating target control commands adapted to the three-dimensional scene based on scene information of the three-dimensional scene reconstructed from the high-dimensional vector space; the scene information of the three-dimensional scene is used to characterize the driving environment currently in which the vehicle is located; the target control commands are used to control the vehicle to achieve driving behavior that conforms to the scene geometry and dynamic constraints in the environment corresponding to the three-dimensional scene description.

[0005] The method provided in this application acquires multi-dimensional perception data (including at least two dimensions from images, point clouds, radar, GPS / IMU, and map data) during vehicle operation, providing a rich data foundation for subsequent perception fusion. Then, the heterogeneous data is dimensionality-reduced and projected onto a shared high-dimensional vector space, achieving mathematical alignment of different modal data within a unified feature space. Based on this, vector aggregation is performed in the high-dimensional vector space, bringing different dimensional feature vectors describing the same physical object closer together (with a distance less than a preset threshold), effectively solving the problem of spatial and semantic matching and association of the same target in multi-source perception data. Finally, based on the three-dimensional scene information (representing the vehicle's driving environment) reconstructed from this high-dimensional vector space, precisely adapted target control commands are generated. This solution, through a full-process data processing and fusion mechanism, ensures that the final output driving behavior conforms to geometric structures and motion constraints in complex dynamic scenarios, significantly improving the accuracy and scene adaptability of autonomous driving decisions.

[0006] One possible implementation involves dimensionality reduction of the multidimensional sensing data, including: slicing the data of each dimension in the multidimensional sensing data to generate a corresponding token sequence; and performing dimensionality reduction on the token sequence corresponding to each dimension in the multidimensional sensing data.

[0007] In this implementation, the token sequence is obtained by slicing the perception data of each dimension and then reducing the dimensionality of each dimension. While retaining key features, the amount of data is compressed, providing high-quality input for the efficient aggregation of multi-source features in high-dimensional space and the accurate restoration of the three-dimensional scene. This ensures that the final generated driving instructions are both real-time and scene-adaptable.

[0008] In one possible implementation, the data in each dimension of the multidimensional sensing data is sliced ​​to generate a corresponding token sequence, including: for image data, dividing it into multiple image blocks and performing linear mapping to generate an image token sequence; and / or, for point cloud data, performing voxel encoding to generate a point cloud token sequence; and / or, for radar data, performing target detection and point cloudification to generate a radar token sequence; and / or, for GPS / IMU data, performing data slicing based on preset slicing parameters to generate a pose token sequence; and / or, for map data, performing raster segmentation or vector slicing to generate a map token sequence.

[0009] In this implementation, differentiated slicing strategies are designed for the characteristics of different modal data: image data is segmented into linear maps, compressing spatial redundancy while preserving rich texture semantics; point cloud data is voxelized and encoded, structuring the disordered 3D point cloud into a regular token sequence; radar data undergoes target detection and point cloudification, transforming sparse raw echoes into token representations with target semantics; GPS / IMU data is sliced ​​according to preset parameters, discretizing the continuous pose flow into temporal tokens; and map data is rasterized or vector-sliced, transforming high-precision map prior information into structured tokens. This heterogeneous data unified tokenization processing method not only preserves the original features of each modal data (image texture, point cloud spatial structure, radar target characteristics, temporal relationships of poses, and geometric constraints of maps), but also transforms them into a unified format that is computationally achievable and aggregable in a high-dimensional vector space. This provides a high-quality input foundation for subsequent alignment and fusion of multi-source features and accurate reconstruction of 3D scenes, significantly improving the autonomous driving system's perception and understanding of complex environments.

[0010] In one possible implementation, if the amount of point cloud data exceeds a preset data amount threshold or the resolution exceeds a preset resolution threshold, the method further includes: reducing the sampling frequency of the point cloud data; and / or, applying an attention sparsity mechanism to the point cloud token sequence generated after the point cloud data conversion, in order to reduce the computational cost of self-attention computation.

[0011] When the point cloud data volume is too large or the resolution is too high, this solution reduces the sampling frequency to compress the input data volume. At the same time, it adopts an attention sparsity mechanism for the point cloud token sequence, selectively ignoring irrelevant or redundant feature interactions during the self-attention calculation process. This significantly reduces the computational complexity while ensuring that key geometric information is not lost, effectively solving the real-time bottleneck problem caused by radar and ensuring that control commands can still be generated efficiently in complex scenarios.

[0012] In one possible implementation, the method further includes: when processing the multidimensional sensing data into slices, attaching a spatiotemporal label to the token sequence generated for each slice; the spatiotemporal label is used to restore the token sequence to the original spatiotemporal corresponding geometric coordinates when restoring the scene information of the three-dimensional scene.

[0013] When slicing perceptual data from various dimensions to generate token sequences, a spatiotemporal label is attached to each token, essentially labeling each data fragment with its spatiotemporal coordinate identity information in the real world. This allows for precise reverse mapping of abstract token features back to their original geometric positions in 3D space after subsequent vector aggregation in a high-dimensional vector space, based on these labels. This ensures that the reconstructed 3D scene is not only semantically consistent but also geometrically precisely aligned, providing a reliable scene foundation for generating driving commands that conform to actual spatiotemporal constraints.

[0014] In one possible implementation, based on scene information of a 3D scene reconstructed from a high-dimensional vector space, target control instructions adapted to the 3D scene are generated. This includes: inputting multimodal fusion features obtained through vector aggregation into a large model; the large model then reconstructs the scene from the multimodal fusion features to obtain scene information of the 3D scene; wherein, during the scene reconstruction process, the large model assigns weights to features corresponding to different modal data; features corresponding to image data are assigned higher weights than other features; and the large model generates target control instructions adapted to the 3D scene based on the scene information of the 3D scene.

[0015] This scheme assigns differentiated weights to features of different modalities when using a large model for scene reconstruction, with a particular emphasis on setting the weights of image data features higher than those of other modalities. This design fully considers the rich information advantages of image data in semantic understanding, texture details, and scene classification, enabling the large model to prioritize visual information for lane line recognition, traffic sign interpretation, and dynamic target classification when constructing 3D scenes. Simultaneously, it integrates geometric and motion features from point cloud and radar data as supplementary verification. Through this hierarchical multimodal fusion mechanism, the reconstructed 3D scene possesses both precise spatial geometric constraints and rich semantic context, providing a high-quality decision-making basis for the large model to generate control commands that conform to human driving habits and are adapted to the complex current environment.

[0016] In one possible implementation, the method further includes: acquiring training samples, which include multidimensional sensory data and their corresponding real control commands; inputting the multidimensional sensory data into an initial large model, which outputs predicted control commands and executes multiple sub-tasks; the multiple sub-tasks include at least a perception detection task, a map segmentation task, and an occupancy prediction task; determining the sub-task loss value corresponding to each sub-task; determining the control command loss value between the predicted control commands and the real control commands; constructing a comprehensive loss function based on the sub-task loss values ​​and the control command loss values; updating the parameters of the initial large model based on the comprehensive loss function until the training termination condition is met, thus obtaining a trained large model.

[0017] This approach constructs a joint training framework encompassing multiple sub-tasks, including perception detection, map segmentation, and occupancy prediction. This allows the large model to simultaneously learn and generate control commands while simultaneously developing a deep understanding of the environment. The loss function corresponding to each sub-task imposes multi-dimensional constraints on the model parameters, effectively preventing the model from prematurely converging to a single task and neglecting feature learning in intermediate layers. This multi-task joint optimization approach enables the features extracted by the model to simultaneously possess object recognition, geometric modeling, and motion prediction capabilities, thereby significantly improving the accuracy and generalization ability of downstream control command generation.

[0018] In one possible implementation, a comprehensive loss function is constructed based on the subtask loss value and the control instruction loss value, including: dynamically adjusting the weights of each subtask according to the gradient change or performance index of each subtask so that the weighted loss values ​​of each subtask converge; performing a weighted summation of the subtask loss values ​​based on the dynamically adjusted weights; and constructing a comprehensive loss function based on the weighted summation of the subtask loss values ​​and the control instruction loss value.

[0019] This approach dynamically adjusts the weights of each subtask based on its gradient changes or performance metrics when constructing the comprehensive loss function, ensuring that the loss values ​​for different tasks remain balanced during training. This dynamic weighting mechanism prevents a single subtask from dominating the model's update direction due to excessively large gradients, ensuring that the model converges synchronously across multiple perception tasks. By bringing the weighted losses of each subtask closer together, the multimodal fusion features learned by the model can meet the accuracy requirements of each perception task, providing a more balanced and robust feature representation for the final control command generation.

[0020] One possible implementation method further includes: for the original data lost during the dimensionality reduction process, using a character mapping method to associate it with the high-dimensional vector space; the character mapping is implemented based on a predefined character dictionary, which is used to call the corresponding original data according to the usage scenario in subsequent tasks.

[0021] This solution addresses the unavoidable loss of original data information during dimensionality reduction by introducing a character mapping mechanism based on a predefined character dictionary. This mechanism associates the lost key original data with characters in a high-dimensional vector space. This compensation design essentially creates an external knowledge index for the model, allowing for on-demand retrieval of the corresponding original data in subsequent tasks, based on specific usage scenarios (such as requiring precise geometry of the original point cloud or detailed texture of an image). This mechanism effectively preserves the traceability of the original data without increasing the computational burden on the main model architecture, providing flexible data retrieval capabilities for decision-making scenarios requiring high-fidelity information.

[0022] Secondly, embodiments of this application provide a control command generation apparatus, comprising: an acquisition module for acquiring multi-dimensional perception data during vehicle operation; the multi-dimensional perception data includes data in at least two of the following dimensions: image data, point cloud data, radar data, GPS / IMU data, and map data; a projection module for performing dimensionality reduction processing on the multi-dimensional perception data and projecting it onto a shared high-dimensional vector space; an aggregation module for performing vector aggregation in the high-dimensional vector space, such that the distance between feature vectors describing the same physical object in different dimensions of the multi-dimensional perception data in the high-dimensional vector space is less than a preset threshold; and a generation module for generating target control commands adapted to the three-dimensional scene based on scene information of the three-dimensional scene reconstructed from the high-dimensional vector space; the scene information of the three-dimensional scene is used to characterize the driving environment currently in which the vehicle is located; and the target control commands are used to control the vehicle to achieve driving behavior that conforms to the scene geometry and dynamic constraints in the environment corresponding to the three-dimensional scene description.

[0023] Thirdly, embodiments of this application provide an autonomous driving system, including: a control command generation device and a data acquisition device provided in the second aspect; the data acquisition device is used to acquire multi-dimensional perception data during vehicle operation and send the multi-dimensional perception data to the control command generation device, so that the control command generation device can implement the autonomous driving function using the control command generation method provided in the first aspect and its possible implementations.

[0024] Fourthly, embodiments of this application provide a vehicle including the control command generation device provided in the second aspect; or, including the autonomous driving system provided in the third aspect.

[0025] Fifthly, this application provides an electronic device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement the method described in the first aspect and any possible implementation thereof.

[0026] In a sixth aspect, this application provides a computer-readable storage medium that, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the methods described in the first aspect and any of their possible implementations.

[0027] In a seventh aspect, this application provides a computer program product including computer instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0028] It should be noted that the technical effects of any of the implementation methods in aspects two through seven can be found in the technical effects of the corresponding implementation methods in aspect one, and will not be repeated here. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application will be described below.

[0030] Figure 1 A schematic diagram of the structure of a vehicle provided in an embodiment of this application; Figure 2 A flowchart illustrating a method for generating control instructions provided in an embodiment of this application; Figure 3 A flowchart illustrating a model training method provided in this application embodiment; Figure 4 A flowchart illustrating a method for generating control instructions provided in an embodiment of this application; Figure 5 A flowchart illustrating another method for generating control instructions provided in an embodiment of this application; Figure 6 A schematic diagram of a control command generation device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an instruction generation device provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] It should be noted that in the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0033] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0034] In the embodiments of this application, "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0035] For ease of description, the application scenarios involved in this application will be introduced first.

[0036] Please see Figure 1 , Figure 1 This is a schematic diagram of a vehicle structure provided in an embodiment of this application. The vehicle can be, but is not limited to, a pure electric vehicle (PEV / BEV), a hybrid electric vehicle (HEV), a range-extended electric vehicle (REEV), a plug-in hybrid electric vehicle (PHEV), or a new energy vehicle.

[0037] like Figure 1 As shown, the vehicle 100 provided in this application embodiment may include: an image sensor 11, a radar 12, a global positioning system (GPS) and an inertial measurement unit (IMU) (hereinafter referred to as GPS / IMU system 13), and a controller 14.

[0038] The image sensor 11, also known as a camera, is used to acquire environmental image data around the vehicle. Specifically, a high-resolution complementary metal-oxide-semiconductor (CMOS) image sensor can be used, configured as a multi-camera group (such as a forward-looking telephoto lens, a forward-looking wide-angle lens, a surround-view fisheye lens, and a panoramic camera) to cover a 360° field of view around the vehicle. It supports a global shutter to reduce rolling shutter effect during high-speed motion, and features high dynamic range and light-emitting diode (LED) flicker suppression to ensure clear image data is acquired even in high-contrast lighting or nighttime environments. This application does not limit the specific implementation of the image sensor.

[0039] Radar 12 may include lidar and millimeter-wave radar. The lidar may be mechanical or solid-state lidar, capable of outputting three-dimensional point cloud data (including information such as distance and reflection intensity). In practical applications, it can be configured as a forward-facing main lidar and a blind-spot lidar as needed, and this application embodiment does not impose any limitations on this. The millimeter-wave radar may be a 4D imaging millimeter-wave radar or a traditional high-precision millimeter-wave radar, capable of simultaneously outputting target range, velocity, horizontal azimuth angle, and elevation angle (4D radar) information.

[0040] The GPS / IMU system 13 can integrate a high-precision global navigation satellite system (GNSS) receiver with a fiber optic / microelectromechanical system (MEMS) inertial measurement unit (IMU). It supports real-time kinematic (RTK) differential positioning, providing centimeter-level positioning accuracy; the IMU outputs three-axis acceleration, three-axis angular velocity, and attitude angle data. This application does not limit the specific implementation of the GPS / IMU system.

[0041] The controller 14 can be a multi-core heterogeneous high-performance on-board computing unit, integrating a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU). It is responsible for executing algorithmic tasks such as data slicing, dimensionality reduction, high-dimensional vector space aggregation, large model scene reconstruction, and target control command generation. It also sends control commands to the vehicle's drive-by-wire actuators via controller area network flexible data rate (CAN FD) or the on-board Ethernet interface.

[0042] It should be noted that the system architecture and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. Figure 1 The vehicle components and structures shown are merely illustrative. In practical applications, vehicles may include more or fewer components. For example, a vehicle may also include a map storage unit, a time synchronization unit, drive-by-wire, steering-by-wire, and brake-by-wire. Those skilled in the art will recognize that, with the evolution of vehicle architecture and the emergence of new business scenarios, the technical solutions provided in this application are equally applicable to similar technical problems.

[0043] The method for generating control instructions provided in this application embodiment will now be described with reference to the accompanying drawings. The method for generating control instructions provided in this application embodiment can be applied to an instruction generation device. The instruction generation device can be the controller of the aforementioned vehicle, or it can be an electronic device such as a personal computer (PC), laptop computer, mobile device, tablet computer, or laptop computer that is communicatively connected to the vehicle. This application embodiment does not limit the specific form of the electronic device. Alternatively, the instruction generation device can also be a server, or a server cluster composed of multiple servers. In some implementations, the server cluster can be a distributed cluster server. This application embodiment does not impose any limitations in this regard.

[0044] Please see Figure 2 The method for generating control commands provided in this application includes the following steps: S101. Acquire multi-dimensional perception data during vehicle operation.

[0045] The instruction generation device first needs to acquire multi-dimensional perception data collected by the vehicle during operation in real time. Multi-dimensional perception data refers to a collection of heterogeneous information from multiple sources, collected by various types of onboard sensors, describing the vehicle's current operating status and surrounding environment. Specifically, this data includes at least two or more of the following five dimensions: image data, point cloud data, radar data, GPS / IMU data, and map data.

[0046] The image data, acquired by the vehicle's optical cameras, contains rich texture, color, and semantic information, helping to identify targets such as lane lines, traffic signs, pedestrians, and vehicles. Point cloud data, acquired by LiDAR, presents the geometric contours and precise locations of objects in three-dimensional spatial points, with each point containing information such as three-dimensional coordinates and reflection intensity. Radar data, from millimeter-wave or 4D millimeter-wave radar, outputs information such as the distance, radial velocity, horizontal azimuth, and pitch angle of the detected targets in the form of sparse point clouds or target lists, offering the advantage of all-weather operation. GPS / IMU data, provided by a combination of the Global Positioning System and Inertial Measurement Unit, outputs information such as the vehicle's absolute latitude and longitude coordinates, elevation, speed, attitude angles, and three-axis acceleration, serving as the core source for vehicle positioning and motion state perception. Map data, including high-precision maps or standard navigation maps, provides prior information on static environmental elements such as lane lines, curbs, traffic signs, and traffic lights in vector or raster form, providing important prior constraints for the perception system.

[0047] In some embodiments, if the data itself has time skew or inconsistent coordinates, subsequent vector aggregation will lose its physical meaning. Therefore, the instruction generation device needs to ensure that the data from different sensors maintain strict alignment in both time and space dimensions. Time alignment requires that the data frames acquired by all sensors correspond to the same moment, and spatial alignment requires that the data from different sensors can be transformed into the same reference coordinate system. In other words, the core task of the instruction generation device is to accurately synchronize and trigger data acquisition from multiple sensors, laying the foundation for subsequent processing.

[0048] As a specific implementation, the command generation device connects to various sensors via a high-speed in-vehicle communication network. The device receives precise pulse-second signals from GPS / IMU or broadcasts using the IEEE 802.1AS precise time protocol, and sends synchronization trigger signals to all sensors, including cameras, LiDAR, and millimeter-wave radar, ensuring they are exposed or scanned simultaneously. Images captured by cameras are processed by an image signal processor and output as YUV or RGB format image data; LiDAR scans can output 3D point cloud data packets containing hundreds of thousands or even millions of points; millimeter-wave radar can output target lists or 4D point cloud data; and GPS / IMU can output continuous pose data streams. Map data is typically pre-loaded into the device's memory, and the command generation device retrieves local map tiles within a certain range around the vehicle in real time based on the current GPS location. All collected data is timestamped for subsequent processing.

[0049] S102. Reduce the dimensionality of the multidimensional sensing data and project it onto a shared high-dimensional vector space.

[0050] After successfully acquiring multidimensional sensing data, the instruction generation device needs to perform dimensionality reduction processing on the original heterogeneous data and project the dimensionality-reduced result into a shared high-dimensional vector space. Dimensionality reduction refers to compressing the high-dimensional, redundant information in the original data into a low-dimensional feature vector representation through feature extraction and encoding. For example, an original image with millions of pixels contains a large amount of pixel-level redundant information; dimensionality reduction aims to extract compact semantic features from it. Similarly, a LiDAR point cloud containing hundreds of thousands of points needs to be compressed into key features that describe its geometric structure.

[0051] A shared high-dimensional vector space refers to a predefined, unified mathematical space with fixed dimensions (e.g., 512, 768, or 1024 dimensions). After dimensionality reduction, all data from different modalities are further mapped into this space, transforming image, point cloud, radar, positioning, and map data from their originally incompatible mathematical forms into vector forms that can be computed and compared within the same space.

[0052] Image data is essentially a two-dimensional grid of pixels; point cloud data is an unordered set of three-dimensional points; radar data is a sparse list of targets; GPS / IMU data is a continuous time-series signal; and map data is vector graphics or raster images. These data have completely different mathematical structures in their original space, making direct fusion impossible. Therefore, a dedicated encoder must be designed for each modality to convert them into feature vector sequences, and then a unified projection head must be used to map these feature vectors from different encoders into the same vector space.

[0053] As a feasible implementation method, S102 can be specifically implemented as follows: S1021. Slice the data of each dimension in the multidimensional sensing data to generate the corresponding token sequence.

[0054] First, the data in each dimension of the multidimensional sensing data is sliced ​​to generate a corresponding token sequence. Slicing refers to dividing the original data into a series of basic units, each called a token, based on the characteristics of different modalities and using appropriate partitioning methods. These tokens are equivalent to structured representation fragments of the original data, laying the foundation for subsequent feature extraction. Through slicing, the originally diverse original data is uniformly transformed into a serialized token form, with each token carrying local feature information of the corresponding modality.

[0055] One implementation approach involves dividing image data into multiple image blocks and performing linear mapping to generate an image token sequence. Specifically, the original image is uniformly divided into multiple fixed-size rectangular image blocks; for example, a standard-sized image is divided into several 16-pixel by 16-pixel blocks. This partitioning discretizes the continuous image space into a series of ordered image block units, each containing local texture, color, and contour information of the corresponding region in the original image. After partitioning, each image block undergoes linear mapping, i.e., a learnable fully connected network or convolutional layer is used to convert the pixel matrix within each block into a one-dimensional feature vector. These feature vectors are combined according to the arrangement of the image blocks in the original image to form the image data token sequence. Each token corresponds to a local region in the original image, carrying the visual feature information of that region. The advantage of this approach is that it compresses the high-dimensional pixel space into a compact token sequence, significantly reducing the data dimensionality while preserving the semantic information of the image, providing a structured input form for subsequent dimensionality reduction processing.

[0056] As one implementation method, voxel encoding is performed on point cloud data to generate a point cloud token sequence. Point cloud data is essentially an unordered set of three-dimensional points, each containing information such as spatial coordinates and reflection intensity. Directly processing this unordered structure has high computational complexity. Therefore, the instruction generation device first divides the three-dimensional space around the vehicle into regularly arranged three-dimensional grid units, each unit called a voxel. The size of the voxel can be preset according to the perception accuracy requirements, for example, set as a small cube of 0.1 meters on each side. After completing the spatial division, the device processes each non-empty voxel unit, performing feature aggregation on all the point cloud data contained in the voxel, for example, by taking the average value, the maximum value, or using a network such as PointNet to extract the local geometric features within the voxel. Each non-empty voxel generates a corresponding feature vector after feature extraction. These feature vectors are organized according to the positional relationship of the voxels in three-dimensional space to form the token sequence of the point cloud data. This voxel encoding method transforms unordered point cloud data into a regular, ordered token sequence. Each token corresponds to a local region in three-dimensional space, carrying the geometric structure information of that region, providing a regular data format for subsequent processing.

[0057] As one implementation method, radar data undergoes target detection and point cloud processing to generate radar token sequences. Radar data typically exists in two forms: a sparse point cloud where each point contains information such as distance, velocity, and horizontal angle; and a target list after radar signal processing, where each target corresponds to a detected object and its attributes. The device first performs target detection processing on the raw radar echo, extracting valid targets from the noise using algorithms such as constant false alarm rate (CFAR) detection. For each detected target, the device further extracts its core attributes, including radial distance, radial velocity, and horizontal azimuth angle. For 4D millimeter-wave radar, elevation angle information is also extracted, forming multi-dimensional features for each target. Subsequently, the device performs point cloud processing on these target features, converting each target into a virtual point containing spatial coordinates and motion attributes, thus forming a structured radar token sequence. Each token corresponds to a real target detected by the radar, carrying the target's distance, velocity, and angle information. This processing method transforms the raw radar echo signal into a point cloud-like token sequence, enabling better alignment and fusion of radar data with other spatial sensing data in subsequent processing.

[0058] As one implementation method, GPS / IMU data is processed by slicing based on preset slicing parameters to generate a pose token sequence. GPS / IMU data provides real-time information such as vehicle position, speed, attitude, and acceleration in the form of high-frequency continuous output. This data is infinitely continuous on the time axis and cannot be directly processed as a discrete token sequence. The device first discretizes the continuous data stream according to preset slicing parameters. The slicing parameters can be a fixed time interval, such as slicing a data segment every 100 milliseconds; or a fixed travel distance interval, such as slicing a data segment every 1 meter traveled; or they can be dynamically set according to the data volume. This application embodiment does not impose any restrictions on this. Within each slice window, the instruction generation device performs feature encoding on the continuous pose data sequence within that time period. For example, it extracts motion features within the window through a temporal network, or directly selects pose data at key time points within the window as representatives. Each slice window generates a corresponding feature vector after processing. These feature vectors are combined in chronological order to form the pose data token sequence. Each token corresponds to the vehicle's motion state within a specific time period or distance segment, carrying the changes in position, attitude, and motion information within that time period, providing a foundation for subsequent spatiotemporal alignment of the vehicle's motion state with environmental perception information.

[0059] As a specific implementation, this application relaxes the temporal alignment requirements of the slice sequences. The model no longer mandates that all slices strictly correspond to precise timestamps, but instead allows slice sequences to be dynamically spliced, overlaid, or locally aligned according to task requirements in subsequent processing. Correspondingly, the slice size is determined primarily based on data balance and computational efficiency, rather than using a fixed time interval as a benchmark. This design effectively reduces the reliance on strict time synchronization of multi-source sensors, improves the model's adaptability to different acquisition frequencies and transmission delays, and simplifies the preprocessing workflow before multimodal data fusion.

[0060] One implementation approach involves rasterizing or tiling map data to generate a sequence of map tokens. Map data provides static environmental elements such as road structure, traffic signs, and lane topology as prior information. These elements are typically stored in vector format, containing precise geometric coordinates and semantic labels. When using a rasterization strategy, the device first divides the local map area around the vehicle into a regular two-dimensional raster grid, with each grid cell corresponding to a fixed-size area on the ground. Then, vector-formatted map elements are projected onto these grids, assigning each grid a corresponding semantic label, such as whether the grid belongs to a lane line, curb, drivable area, or obstacle. Each grid cell is encoded to generate a corresponding feature vector, and these feature vectors are arranged spatially according to the grid to form a sequence of map tokens. When using a vector tiling strategy, the device directly divides vector map elements according to spatial regions, for example, dividing the map into fixed-size vector tiles, each tile containing several complete map elements. Then, a graph neural network is used to encode the set of vector elements within each tile to generate corresponding feature vectors, forming a sequence of map tokens in vector form. Regardless of the strategy employed, the resulting map token sequence transforms static prior map information into a structured representation compatible with real-time perception data formats, providing a foundation for the subsequent fusion of prior knowledge and real-time perception.

[0061] In this implementation, differentiated slicing strategies are designed for the characteristics of different modal data: image data is segmented into linear maps, compressing spatial redundancy while preserving rich texture semantics; point cloud data is voxelized and encoded, structuring the disordered 3D point cloud into a regular token sequence; radar data undergoes target detection and point cloudification, transforming sparse raw echoes into token representations with target semantics; GPS / IMU data is sliced ​​according to preset parameters, discretizing the continuous pose flow into temporal tokens; and map data is rasterized or vector-sliced, transforming high-precision map prior information into structured tokens. This heterogeneous data unified tokenization processing method not only preserves the original features of each modal data (image texture, point cloud spatial structure, radar target characteristics, temporal relationships of poses, and geometric constraints of maps), but also transforms them into a unified format that is computationally achievable and aggregable in a high-dimensional vector space. This provides a high-quality input foundation for subsequent alignment and fusion of multi-source features and accurate reconstruction of 3D scenes, significantly improving the autonomous driving system's perception and understanding of complex environments.

[0062] As a feasible implementation method, the control command generation method provided in this application embodiment further includes: when processing the multidimensional perception data into slices, attaching a spatiotemporal label to the token sequence generated for each slice; the spatiotemporal label is used to restore the token sequence to the original spatiotemporal corresponding geometric coordinates when restoring the scene information of the three-dimensional scene.

[0063] The instruction generation device slices the perceived data from various dimensions and generates a token sequence, while also attaching a corresponding spatiotemporal tag to each generated token. A spatiotemporal tag is an identifier that records the temporal and spatial location information of the original data corresponding to the token in the real world. For image tokens generated from image data, the spatiotemporal tag records the pixel coordinates of the image patch in the original image, as well as the global timestamp and corresponding camera extrinsic parameters when the image frame was acquired; for voxel tokens generated from point cloud data, the spatiotemporal tag records the geometric coordinate range of the voxel in three-dimensional space and the point cloud scanning time; for radar tokens generated from radar data, the spatiotemporal tag records the spatial location of the target in the radar coordinate system and the detection time; for pose tokens generated from GPS / IMU data, the spatiotemporal tag records the acquisition time and geographic coordinates corresponding to the pose; for map tokens generated from map data, the spatiotemporal tag records the precise location of the map element in the global coordinate system. This spatiotemporal tag is equivalent to each token's "identity certificate" and "coordinate index" in the real world, firmly binding the abstract mathematical vector to the specific spatiotemporal location in the real world.

[0064] The purpose of actively preserving spatiotemporal information during the slicing stage is to establish traceability from the feature space to the physical space. When the token sequences enter subsequent dimensionality reduction, projection, and vector aggregation processes, they undergo various mathematical transformations and interactions in the high-dimensional vector space. The original coordinate information may be diluted or lost during this process. The spatiotemporal tag, as a bypass information, does not participate in feature calculations in the high-dimensional vector space but remains associated with the token sequence as metadata. The advantage of this design is that the device can enjoy the convenience of flexible feature fusion in the high-dimensional vector space without losing precise control over the original spatial location due to feature transformations.

[0065] In its implementation, after completing the slicing process for each modality, the instruction generation device immediately attaches a structured spatiotemporal label field to each generated token. For image tokens, the label includes the pixel coordinates of the top-left corner of the image patch in the original image, the image width and height, the acquisition timestamp, and the camera's pose matrix at that moment. For point cloud tokens, the label includes the center point coordinates and size of the voxel, and the point cloud acquisition timestamp. For radar tokens, the label includes the target's distance and angle in radar polar coordinates, the converted Cartesian coordinates, and the detection timestamp. For pose tokens, the label includes the latitude, longitude, elevation, heading angle, and velocity at that moment. For map tokens, the label includes the global coordinate range of the raster or vector element. These spatiotemporal labels accompany the token sequence throughout the entire processing flow. Until the final 3D scene reconstruction is required, the device can use these labels to remap the feature vectors after vector aggregation back to the original real spatiotemporal coordinates. For example, when a device needs to reconstruct the 3D spatial location corresponding to an image token, it can read the spatiotemporal tag attached to the token, combine it with the camera imaging model and the vehicle's pose at the time, and calculate the geometric coordinates of the image block in the real world, thereby achieving accurate backtracking from abstract features to physical space. This spatiotemporal tagging mechanism effectively ensures the geometric accuracy of 3D scene reconstruction, enabling the final generated control commands to strictly conform to the spatial constraints of the real world.

[0066] As can be seen, in this implementation, when slicing the perceived data of each dimension to generate a token sequence, a spatiotemporal label is attached to each token, which is equivalent to labeling each data fragment with its spatiotemporal coordinate identity information in the real world. This enables the subsequent vector aggregation in the high-dimensional vector space to accurately map the abstract token features back to their original geometric positions in three-dimensional space based on these labels, thereby ensuring that the reconstructed three-dimensional scene is not only semantically consistent but also has precisely aligned geometric coordinates, providing a reliable scene foundation for generating driving instructions that conform to actual spatiotemporal constraints.

[0067] S1022. Perform dimensionality reduction processing on the token sequence corresponding to each dimension in the multidimensional sensing data.

[0068] Dimensionality reduction refers to compressing each token sequence into a lower-dimensional feature vector representation through a feature extraction network. This removes redundant information from the original data while retaining key features that characterize the core semantics and geometric attributes of the modality. This reduces computational complexity and prepares the data for subsequent projection into a shared space. After dimensionality reduction, each dimension of data is transformed into a set of compact feature vectors. Although these vectors have reduced dimensionality, they still retain the unique information of their respective modalities, such as the texture semantics of images, the spatial structure of point clouds, the target characteristics of radar, the temporal relationships of poses, and the geometric constraints of maps.

[0069] For example, as a feasible implementation, for image data, the device inputs the original image with a size of 640×480×3 into a lightweight visual Transformer encoder, divides the image into 16×16 image blocks, and then linearly maps each image block into a 128-dimensional vector. After 6 layers of Transformer self-attention calculation, the final output is a dimensionality-reduced image feature map sequence with a size of 128×40×30.

[0070] As a feasible implementation method, for point cloud data, the instruction generation device inputs the original point cloud containing N points, each with three-dimensional coordinates and reflection intensity information, into a voxel encoding network. First, it performs voxelization with a size of 0.1 meters square. Within each non-empty voxel, PointNet is used to extract local geometric features, generating a voxel feature map with 64 channels. Then, it is processed by a 3-layer sparse convolutional network for dimensionality reduction, and finally outputs a dimensionality-reduced point cloud feature map sequence with a size of 64×32×32.

[0071] As a feasible approach, for radar data, the device extracts the core attributes of each radar target, including center position, radial velocity and radar cross section, to form an M×5 target feature matrix. Then, these attribute features are input into a two-layer multilayer perceptron network to output a 64-dimensional reduced radar feature vector sequence.

[0072] As a feasible approach, for GPS / IMU data, the device inputs a continuous 10-frame pose data stream into a long short-term memory network, sets the hidden layer dimension to 128, and outputs a 128-dimensional reduced pose feature vector sequence after temporal feature extraction.

[0073] As a feasible approach, for map data, the device retrieves a local rasterized map based on the current GPS location, inputs the raster map into a ResNet-18 convolutional neural network encoder, and finally outputs a 256-dimensional reduced map feature map sequence.

[0074] As a feasible implementation, after dimensionality reduction of each modality, the device flattens the feature vectors output by all modalities and inputs them into a shared projection head. This projection head consists of three fully connected network layers with dimensions of 512, 768, and 1024 respectively, uniformly mapping the features of all modalities to a shared high-dimensional vector space of 1024. In this shared space, a contrastive learning mechanism is used to shorten the distance between cross-modal feature vectors belonging to the same physical object.

[0075] As can be seen from S1021-S1022, in this implementation method, the token sequence is obtained by slicing the perception data of each dimension and reducing the dimensionality of each dimension. While retaining the key features, the amount of data is compressed, which provides high-quality input for the efficient aggregation of multi-source features in high-dimensional space and the accurate restoration of the three-dimensional scene. This ensures that the final generated driving command has both real-time performance and scene adaptability.

[0076] S103. Perform vector aggregation in the high-dimensional vector space so that the distance between the feature vectors describing the same physical object in different dimensions of the multidimensional perception data in the high-dimensional vector space is less than a preset threshold.

[0077] The preset threshold is pre-set and can be determined according to needs in practical applications; this application does not impose any restrictions on it. In a specific embodiment, the preset threshold can be set according to the safety level of the autonomous driving system. For example, in L3 autonomous driving mode, to ensure high perception accuracy, the distance threshold can be set to 0.3; in L2 assisted driving mode, to consider recall rate, the threshold can be set to 0.5. Those skilled in the art can flexibly adjust this threshold according to actual business needs (such as perception accuracy requirements, computing resource limitations), and its specific value does not constitute a limitation of this application.

[0078] The instruction generation device needs to perform vector aggregation operations within a shared high-dimensional vector space. Vector aggregation refers to the mathematical interaction and fusion of all feature vectors that have undergone dimensionality reduction and been projected into this space. This achieves alignment of multimodal data at the feature level. Specifically, it aims to ensure that feature vectors from different dimensions of data, but describing the same physical object in the real world, are less than a preset threshold apart in this high-dimensional vector space. For example, when there is a real car in front of a vehicle, the image data contains visual feature vectors corresponding to the car, the point cloud data contains point cloud feature vectors corresponding to the car's geometric contour, and the radar data may also contain radar target feature vectors corresponding to the car. After projection, these feature vectors from different sensors may initially be scattered in different positions in the high-dimensional space. Vector aggregation aims to bring them closer together through a mathematical mechanism, ultimately aggregating them to jointly represent the unified physical entity of "the car in front."

[0079] The instruction generation device needs to establish relationships between all feature vectors in this space. These relationships are calculated based on the similarity between vectors; higher similarity indicates a greater likelihood that the two vectors point to the same physical object. The device iteratively optimizes this process, attracting highly similar vectors closer together until their distance is reduced to within a preset threshold; simultaneously, low-similarity vectors repel each other or remain far apart. Through this aggregation process, the originally isolated and scattered modal features ultimately form feature clusters based on physical objects. Each cluster contains all feature vectors from different sensors describing the same entity. This provides a highly integrated and mutually corroborating feature foundation for a comprehensive understanding of the 3D scene.

[0080] The instruction generation device combines the feature vectors of all modalities output from the previous stage into a complete feature sequence, and then inputs this sequence into a neural network structure based on an attention mechanism, such as the encoder part of the Transformer model. Under the action of the attention mechanism, each feature vector calculates a similarity score with all other feature vectors and updates its own numerical representation based on these scores. Vector pairs with high scores pass information to each other during the update process and shorten their distance. This process is repeated in a multi-layer neural network. Through multiple iterations of optimization, the relative positions between feature vectors are continuously adjusted, eventually bringing the distance between feature vectors belonging to the same physical object within a preset threshold range. When the aggregation is complete, the device outputs a set of highly fused multimodal features. At this point, each real-world object is described in a high-dimensional vector space by a set of closely spaced feature vectors.

[0081] S104. Based on the scene information of the three-dimensional scene reconstructed from the high-dimensional vector space, generate target control commands that are adapted to the three-dimensional scene.

[0082] Among them, the scene information of the three-dimensional scene is used to characterize the driving environment in which the vehicle is currently located; the target control command is used to control the vehicle to achieve driving behavior that conforms to the scene geometry and dynamic constraints in the environment corresponding to the three-dimensional scene description.

[0083] The command generation device needs to generate target control commands that are precisely adapted to the current driving environment based on the 3D scene information reconstructed from the high-dimensional vector space. The 3D scene information reconstructed from the high-dimensional vector space refers to decoding the fused features obtained after vector aggregation, transforming them from abstract vector form back into specific information that clearly represents the vehicle's current driving environment. This scene information should include the geometric structure and dynamic elements of the vehicle's surrounding environment, such as the position and direction of lane lines, road boundaries, and the position and movement of other vehicles and pedestrians. The target control command is the control signal ultimately sent to the vehicle's actuators. Its function is to control the vehicle to achieve driving behavior that conforms to the geometric structure and dynamic constraints of the environment described by the 3D scene. Conforming to the geometric structure and dynamic constraints means that the vehicle's trajectory cannot exceed the physical boundaries of the road, cannot collide with obstacles, and must also comply with kinematic limitations such as acceleration, deceleration, and steering, as well as traffic rules.

[0084] The instruction generation device first needs to reconstruct the abstract high-dimensional vector into a concrete scene description; this process is equivalent to completing the transformation from features to cognition. Only with a clear understanding of the current environment can the device make reasonable driving decisions. After obtaining the 3D scene information, the device needs to use the scene as a constraint to plan driving behaviors suitable for the current situation and transform these behaviors into executable low-level control signals. The entire logical chain involves extracting environmental cognition from fused features, deriving driving decisions from environmental cognition, and finally transforming the decisions into physical control.

[0085] The instruction generation device inputs the multimodal fusion features obtained after vector aggregation into a scene reconstruction module. This module converts the feature vectors into a structured or rasterized 3D scene representation through a decoding network, such as a semantic occupancy grid centered on the vehicle and viewed from a bird's-eye view, or an instantiated scene list containing all dynamic object detection boxes and trajectories. The reconstructed scene information fully describes the driving environment in which the vehicle is located at the current moment, including the vehicle's position, drivable area boundaries, lane topology, and the real-time status of all surrounding dynamic objects. After scene reconstruction is completed, the device's behavior planning module receives this scene information as input and plans a driving trajectory suitable for the current scene based on preset driving task objectives, such as safely reaching the destination, obeying traffic rules, and ensuring passenger comfort. Geometrically, this trajectory must be located within the drivable area and avoid all obstacles; dynamically, it must meet the physical constraints of vehicle acceleration, deceleration, and steering. Finally, the device's control conversion module converts the planned trajectory into specific low-level control commands, including throttle opening or brake pressure to control the vehicle's longitudinal movement, and steering wheel angle to control the vehicle's lateral movement. These control commands are sent to the vehicle's drive-by-wire actuators via the vehicle network, driving the vehicle to travel along the planned trajectory in the current three-dimensional scene.

[0086] As a feasible implementation method, S104 can be specifically implemented as follows: S1041. The multimodal fusion features obtained through vector aggregation are input into the large model, which then performs scene reconstruction on the multimodal fusion features to obtain the scene information of the three-dimensional scene. During the scene reconstruction process, the large model assigns weights to the features corresponding to different modal data. The features corresponding to the image data are assigned higher weights than the features assigned to other features.

[0087] Large-scale models refer to deep neural networks that have been pre-trained on large-scale data and possess powerful representational and generalization capabilities. They are typically built on the Transformer architecture and can handle multimodal inputs and generate structured outputs. After the multimodal fused features are input into the large-scale model, it begins to perform scene reconstruction tasks, that is, decoding the abstract, aggregated feature vectors into specific scene descriptions that humans and machines can understand. This includes static elements such as the position and direction of lane lines, the boundaries of curbs, and the semantics of traffic signs, as well as dynamic elements such as the position, speed, and predicted trajectories of surrounding vehicles.

[0088] It's important to note that during scene reconstruction, the large model assigns different weights to features corresponding to different modalities. This weighting mechanism reflects the varying importance of each modality in scene understanding. Specifically, features corresponding to image data are given higher weights than other features. This means that when decoding 3D scenes, the large model relies more heavily on the rich texture and semantic information provided by image data. Image data boasts the highest spatial resolution and the richest semantic content, clearly presenting details such as lane lines, traffic signs, pedestrian postures, and vehicle types—information crucial for accurately understanding driving scenarios. In contrast, while point cloud data offers high geometric accuracy, it lacks semantic information; radar data provides accurate speed measurement, but its point cloud structure is sparse; GPS / IMU provides vehicle status rather than environmental information; and map data offers prior static information. Therefore, assigning higher weights to image features during scene reconstruction allows the large model to prioritize visual information for lane line recognition, traffic sign interpretation, and dynamic target classification, while simultaneously integrating geometric and motion features from point cloud and radar data for supplementary verification and accuracy enhancement. This multimodal fusion mechanism, which clearly distinguishes between primary and secondary elements, ensures that the reconstructed 3D scene possesses both precise spatial geometric constraints and rich semantic context.

[0089] S1042. Based on the scene information of the 3D scene from the large model, generate target control commands that are adapted to the 3D scene.

[0090] After successfully obtaining a structured 3D scene description, the large model continues to use this scene information as input, combined with the vehicle's current state and preset driving task objectives, to generate driving decisions and control commands. The large model integrates scene information with learned prior driving knowledge to plan a driving trajectory that conforms to the scene's geometry and dynamic constraints, and then translates this trajectory into specific low-level control commands. These control commands include throttle opening or brake pressure to control the vehicle's longitudinal movement, and steering wheel angle to control the vehicle's lateral movement. Finally, these commands are sent via the vehicle network to the drive-by-wire actuators, driving the vehicle along the planned trajectory within the current 3D scene. This end-to-end generation method ensures that the entire process from scene understanding to decision-making and control is completed within the unified framework of the large model, avoiding information loss and error accumulation in traditional modular architectures.

[0091] As another feasible approach, large models do not use the same set of weights during inference, but rather adjust the weights of different modal features in real time based on the complexity of the current task. The implementation involves three stages: The first stage, rapid scene understanding: The model uses only image features, assigning an image weight of 1.0 and other modal weights of 0, to generate a coarse-grained scene description within 50ms, including the vehicle's lane, whether there are obstacles ahead, and the approximate drivable area. This stage is used to quickly filter out situations that require special attention.

[0092] The second stage, precise spatial positioning: When an obstacle is detected within 50 meters ahead in the first stage, the model activates point cloud and radar features, setting the image weight to 0.6, the point cloud weight to 0.3, and the radar weight to 0.1. The point cloud features are used to accurately measure the distance and size of the obstacle, and the radar features are used to obtain the relative velocity of the obstacle, completing precise positioning within 20ms.

[0093] The third stage, fine-grained detail understanding: When the vehicle is less than 30 meters from an obstacle and requires precise decision-making, the model further activates map and GPS / IMU features, setting image weights to 0.5, point cloud weights to 0.2, radar weights to 0.1, map weights to 0.1, and GPS / IMU weights to 0.1. Map features are used to confirm lane topology and traffic rules, while GPS / IMU features are used to accurately estimate the vehicle's motion state, completing the final trajectory planning and command generation within 30ms.

[0094] This dynamic weight adjustment mechanism enables the model to adaptively allocate computing resources based on the complexity of the scenario while ensuring real-time performance. More modalities are used to ensure accuracy during critical tasks, while only a single modality is used to ensure efficiency during simple tasks.

[0095] As shown in S1041-S1042, this scheme assigns differentiated weights to features of different modalities when using a large model for scene reconstruction, and specifically sets the feature weights corresponding to image data higher than those of other modalities. This design fully considers the rich information advantages of image data in semantic understanding, texture details, and scene classification, enabling the large model to prioritize visual information for lane line recognition, traffic sign interpretation, and dynamic target classification when constructing 3D scenes, while incorporating geometric and motion features from point cloud, radar, and other data as supplementary verification. Through this multimodal fusion mechanism with clear primary and secondary elements, the reconstructed 3D scene possesses both precise spatial geometric constraints and rich semantic context, providing a high-quality decision-making basis for the large model to generate control commands that conform to human driving habits and are adapted to the current complex environment.

[0096] As can be seen from S101-S104, the method provided in this application provides a rich data foundation for subsequent perception fusion by acquiring multi-dimensional perception data (including at least two dimensions from images, point clouds, radar, GPS / IMU, and map data) during vehicle operation. Then, the heterogeneous data is dimensionality-reduced and projected onto a shared high-dimensional vector space, achieving mathematical alignment of different modal data within a unified feature space. Based on this, vector aggregation is performed in the high-dimensional vector space, bringing different dimensional feature vectors describing the same physical object closer together (distance less than a preset threshold), effectively solving the problem of spatial and semantic matching and association of the same target in multi-source perception data. Finally, based on the three-dimensional scene information (representing the vehicle's driving environment) reconstructed from this high-dimensional vector space, precisely adapted target control commands are generated. This solution, through a full-process data processing and fusion mechanism, ensures that the final output driving behavior conforms to geometric structures and motion constraints in complex dynamic scenarios, significantly improving the accuracy and scene adaptability of autonomous driving decisions.

[0097] In some embodiments, when the amount of point cloud data exceeds a preset data amount threshold or the resolution exceeds a preset resolution threshold, the method further includes: reducing the sampling frequency of the point cloud data; and / or, applying an attention sparsity mechanism to the point cloud token sequence generated after the point cloud data conversion, so as to reduce the computational cost of self-attention computation.

[0098] Point cloud data is collected by LiDAR. With the increase in the number of LiDAR beams and the higher scanning frequency, the amount of data in a single frame of point cloud can reach hundreds of thousands or even millions of points. If such massive amounts of point cloud data are sliced, encoded, and attention-based in every instance, it will put enormous pressure on the device's computing resources and processing time, potentially failing to meet the stringent real-time requirements of autonomous driving systems. Therefore, the device needs to dynamically adjust its processing strategy based on actual conditions, minimizing the computational load while ensuring perception accuracy.

[0099] Specifically, the instruction generation device first monitors and assesses the volume and resolution of the point cloud data in real time. Preset data volume thresholds can be set based on the device's computing power limit and real-time requirements; for example, an optimization mechanism is triggered when the number of points in a single frame exceeds 100,000. Preset resolution thresholds can be set according to the needs of the sensing task; for example, an optimization mechanism is triggered when the point cloud density exceeds 100 points per cubic meter. When the point cloud data volume or resolution exceeds these preset thresholds, the device determines that it is currently under high load and needs to initiate computing resource optimization measures.

[0100] As one implementation method, the instruction generation device can reduce the sampling frequency of point cloud data, thereby reducing the amount of data that needs to be processed in the time dimension. For example, if the LiDAR outputs point cloud data at a frequency of 20 Hz, the device can reduce the sampling frequency to 10 Hz, that is, process it once every frame, or dynamically adjust the sampling frequency according to the current driving speed of the vehicle. When the vehicle is traveling at low speed or stationary, the sampling frequency is reduced, and when the vehicle is traveling at high speed, the normal sampling frequency is restored, thereby directly reducing the amount of data that needs to be processed per unit time.

[0101] As an alternative implementation, the instruction generation device can employ an attention sparsity mechanism on the point cloud token sequence generated after point cloud data conversion. This strategy reduces the computational cost of self-attention calculation in the spatial dimension. After voxelization encoding, each non-empty voxel corresponds to a token, and the number of these tokens can still be large. The standard self-attention mechanism requires calculating the attention weight between each token and all other tokens, with computational complexity proportional to the square of the number of tokens. When the number of tokens is too large, this fully connected self-attention computation becomes unbearable. The core idea of ​​the attention sparsity mechanism is to break this fully connected computational pattern, allowing each token to interact with only a subset of related tokens. Specific implementations can be based on spatial proximity, allowing attention calculations only between tokens of adjacent voxels; or based on content similarity, grouping tokens using clustering or hashing methods and performing attention calculations only within the group; or using random masks to randomly ignore a portion of attention connections. Through this sparsity processing, the computational cost of self-attention can be reduced from quadratic to linear or linear-logarithmic levels, significantly alleviating the computational burden.

[0102] In practical applications, devices can employ one optimization method alone or combine two methods depending on the specific situation. For example, when the amount of point cloud data is extremely large, the device can first appropriately reduce the sampling frequency, and then apply a sparse attention mechanism to the remaining point cloud token sequences. Through dual optimization in both the temporal and spatial dimensions, it ensures that the point cloud data processing remains within the device's real-time processing capabilities. This dynamic resource optimization mechanism allows the device to flexibly balance perception accuracy and computational efficiency in different scenarios, ensuring the integrity of perception in critical scenarios while avoiding unnecessary waste of computing power.

[0103] In this implementation, when the amount of point cloud data is too large or the resolution is too high, this solution reduces the sampling frequency to compress the amount of input data. At the same time, it combines the attention sparsity mechanism for the point cloud token sequence to selectively ignore irrelevant or redundant feature interactions during the self-attention calculation process. This significantly reduces the computational complexity while ensuring that key geometric information is not lost, effectively solving the real-time bottleneck problem caused by radar and ensuring that control commands can still be generated efficiently in complex scenarios.

[0104] In some embodiments, the method further includes: for the original data lost during the dimensionality reduction process, associating it with a high-dimensional vector space using a character mapping method; the character mapping is implemented based on a predefined character dictionary and is used to call the corresponding original data in subsequent tasks according to the usage scenario.

[0105] The core objective of dimensionality reduction is to compress high-dimensional raw data into low-dimensional feature vectors. This process, while removing redundant information, inevitably leads to the loss of some original detailed information. For example, when image data is segmented into image blocks and linearly mapped, the fine texture details within the image blocks may be averaged or abstracted; when point cloud data is voxelized and encoded, the precise spatial distribution of multiple points within a voxel is aggregated into a single feature vector, resulting in the loss of the coordinate information of individual points; when radar data undergoes target detection and point cloud processing, fine-grained information such as noise distribution and signal intensity variations in the original echo may also be filtered out. This lost raw data is crucial in certain specific scenarios. For example, in tasks such as accurately measuring obstacle edge distances, identifying small obstacles, or analyzing road surface materials, the detailed information of the raw data is often more critical than the abstracted features. Therefore, a mechanism is needed to preserve the accessibility of this raw data without affecting the computational efficiency of the main process.

[0106] This embodiment addresses the aforementioned issues using character mapping. For raw data discarded or compressed during dimensionality reduction, the device does not directly input it into the main computation process. Instead, it establishes an index-based association between the raw data and its corresponding feature vectors in the high-dimensional vector space through character mapping. This design is analogous to the relationship between catalog cards and physical books in a library: feature vectors are like cards placed in the reading room for quick retrieval in the main process; raw data is like books stored in the stacks, which can be retrieved at any time using their call numbers. This maintains the lightweight and efficient nature of the main process while reserving a backup plan for special scenarios requiring raw data.

[0107] In its implementation, the device first constructs a predefined character dictionary, with a size of, for example, 65536, where each character corresponds to a Unicode code point. This dictionary acts as an encoding rule, mapping various types of raw data fragments to specific character identifiers. For each image patch in the image data, the device generates a character encoding containing information such as the source sensor ID, acquisition timestamp, and image patch coordinates; for each voxel in the point cloud data, the device generates a character encoding containing spatial grid coordinates and scan time; and for each target in the radar data, the device generates a character encoding containing the target ID and detection time. Furthermore, for raw data attributes lost during dimensionality reduction, such as weather classification, road surface quality, material, and texture information, these are also encoded into strings with a fixed format and mapped to corresponding character IDs.

[0108] For example, for a given image frame, the accompanying weather information is "rain_heavy", the road surface information is "wet_asphalt", and the material information is "asphalt". The device concatenates these strings into "weather=rain_heavy&road=wet_asphalt&material=asphalt", and then maps it to the Unicode character U+F001 using a hash function. During dimensionality reduction, when the device slices the original data and generates a token sequence, each token's corresponding original data fragment is assigned such a character encoding, which is used as metadata and bound to the corresponding feature vector, stored together in an associated storage area near the high-dimensional vector space. When the feature vectors enter subsequent vector aggregation and scene reconstruction processes, these character encodings do not participate in vector calculations; they simply remain attached to the feature vectors, like an "ID number" for each vector.

[0109] When subsequent tasks require detailed information from the original data, the device can retrieve the corresponding original data by reverse-engineering character encoding, based on the needs of the usage scenario. For example, in the 3D scene reconstruction stage, if the model needs to accurately determine the edge position of an obstacle, but the aggregated feature vectors only provide a rough outline, the device can read the character encoding attached to the point cloud token corresponding to the obstacle, and retrieve all the original point cloud data within that voxel from the original point cloud storage based on the spatial grid coordinate information in the encoding, for fine edge calculation. Another example is in the decision-making stage, where adjusting the braking strategy is necessary when encountering a slippery road surface. The model can use the character U+F001 to reverse-look up the dictionary to obtain the information "road surface = slippery," thus increasing safety redundancy during decision-making. This on-demand retrieval mechanism ensures that the device remains efficient in routine processing while obtaining complete support from the original data when detailed information is needed, achieving a dynamic balance between efficiency and accuracy.

[0110] This solution addresses the unavoidable loss of original data information during dimensionality reduction by introducing a character mapping mechanism based on a predefined character dictionary. This mechanism associates the lost key original data with characters in a high-dimensional vector space. This compensation design essentially creates an external knowledge index for the model, allowing for on-demand retrieval of the corresponding original data in subsequent tasks, based on specific usage scenarios (such as requiring precise geometry of the original point cloud or detailed texture of an image). This mechanism effectively preserves the traceability of the original data without increasing the computational burden on the main model architecture, providing flexible data retrieval capabilities for decision-making scenarios requiring high-fidelity information.

[0111] In some embodiments, in order to enable the large model to have accurate instruction generation capabilities, it needs to be pre-trained. It is understood that this training process can be performed by the control instruction generation device mentioned above, or by other independent electronic devices (such as dedicated model training devices), and this application does not specifically limit it in this regard.

[0112] As one feasible approach, please refer to Figure 3 The large model is trained in the following way: S201. Acquire multiple samples, including multidimensional sensing data and their corresponding real control commands.

[0113] Multi-dimensional sensor data samples are multimodal sensor data collected in actual road testing or simulation environments, containing at least two dimensions from image data, point cloud data, radar data, GPS / IMU data, and map data. The corresponding real control commands are control signals such as throttle, brake, and steering generated by professional drivers actually operating the vehicle during data collection, or ideal control commands obtained through post-processing annotation and simulation verification. These paired data sets form the basis of large-scale supervised learning, and the diversity and coverage of the samples directly affect the generalization ability of the trained model.

[0114] Large-scale models need to learn the mapping patterns from perception to decision-making from massive amounts of data; therefore, the quality and quantity of training samples (multidimensional perception data and their corresponding real control commands) are crucial. Thus, as an implementation approach, the multidimensional perception data needs to cover various driving scenarios, including urban roads, highways, complex intersections, inclement weather, and various long-tail scenarios, ensuring the model can learn sufficiently rich environmental features. Simultaneously, the real control commands need to accurately reflect the operating habits and decision-making logic of human drivers in different scenarios, providing the model with correct supervisory signals.

[0115] When acquiring training samples, the sources of sample data need to be as diverse as possible to cover various situations in real-world driving scenarios. Specifically, training samples can come from publicly available multimodal autonomous driving datasets, which typically already contain labeled images, point clouds, and localization data, providing a foundation for initial model training. Simultaneously, extreme scenario data accumulated from past real-world road test projects can also be collected, such as driving records under adverse weather conditions, traffic flow data at complex urban intersections, and perception data in environments with drastic changes in lighting, such as at night or in tunnels. These real-world extreme cases can help the model learn to cope with rare but dangerous situations. Furthermore, simulation tools can generate a large amount of boundary condition and extreme corner case scenario data, such as vehicles suddenly cutting in, pedestrians crossing, and traffic accident scenes—dangerous scenarios that are difficult to collect frequently in actual road tests. Inputting this simulation data into the model for extensive pre-training and validation can effectively compensate for the lack of coverage in real-world data.

[0116] Regarding the composition of sample data, training samples need to fully consider the dynamic characteristics of real traffic environments. Single static images or single-frame point cloud annotation data are far from sufficient for training autonomous driving models, because driving is essentially a continuous spatiotemporal process. The model needs to understand how objects move along the timeline and how they interact with other traffic participants. Therefore, training samples should contain multiple frames of continuous temporal information. Each set of samples consists of multimodal data frames in a continuous time series, not just a single static snapshot. Such temporal data allows the model to learn motion patterns, such as the normal driving trajectory and acceleration / deceleration patterns of vehicles; to learn trajectory prediction capabilities, i.e., inferring future positions based on the motion states of past frames; to learn velocity and acceleration estimation, not just instantaneous position; to learn how to handle occlusion and reappearance phenomena, such as how to maintain tracking when a vehicle reappears after being occluded by a larger vehicle; and to learn the understanding of inter-object interactions, such as how vehicles yield at intersections and how they engage in strategic maneuvering. Currently, many publicly available multimodal datasets and research works have incorporated temporal dynamic modeling as an important consideration, providing continuous frame sequences with timestamps and corresponding annotations, thus providing a data foundation for training the model's temporal learning capabilities.

[0117] In some embodiments, the sample labels corresponding to the multidimensional sensing data may include: original sensing labels, temporal association labels, and control command labels. The original sensing labels are used to supervise intermediate feature learning. Image data employs panoramic segmentation annotation, assigning a semantic category (e.g., vehicle, pedestrian, lane line, drivable area) and instance ID to each pixel; point cloud data uses 3D target detection bounding boxes, each containing the category, center point coordinates, length, width, height dimensions, and orientation angle; radar data is labeled with the ID, position, and velocity of each target.

[0118] Temporal association labels are used to supervise dynamic feature learning. The same target in 10 consecutive frames of data is associated as a trajectory, and each trajectory is labeled with its predicted position and velocity for the next 3 seconds. For example, for a vehicle ahead, its historical trajectory in frames 1-10 is labeled, along with its expected predicted position in frames 11-20.

[0119] Control command labels (i.e., real control commands) are the actual control signals that the model ultimately needs to output. These include: steering wheel angle (-540°~540° continuous value), throttle opening (0~100%), brake pressure (0~100%), headlight control (high beam / low beam / off), and intelligent driving status (normal / degraded / exited). For example, in a following scenario, when the distance between the vehicle and the vehicle in front is less than 20 meters, the real control command label is brake pressure 30% and steering wheel angle maintained at 0°; when the distance is greater than 30 meters, the label is throttle opening 25% maintaining constant speed. During training, the model simultaneously predicts the labels at these three levels and calculates the loss for each, which will not be elaborated upon in this application.

[0120] S202. Input the multidimensional sensing data of the sample into the initial large model, and the large model outputs predictive control commands and executes multiple sub-tasks.

[0121] Among them, multiple sub-tasks include at least perception detection, map segmentation, and occupancy prediction tasks.

[0122] The acquired multidimensional sensory data is input into the initial large model. Upon receiving the input, the large model simultaneously performs two core tasks: first, it outputs predictive control commands, meaning the model directly predicts the throttle, brake, and steering control signals to be applied to the vehicle based on the input sensory data; second, it executes multiple auxiliary sub-tasks, including at least perception detection, map segmentation, and occupancy prediction. Perception detection involves identifying dynamic targets such as vehicles and pedestrians in an image or point cloud and outputting their location and category. Map segmentation involves performing pixel-level or point-level semantic segmentation of static road elements such as drivable areas, lane lines, and curbs. Occupancy prediction involves predicting the probability of each location being occupied in the future, i.e., predicting the probability of each grid (voxel) in three-dimensional space being occupied, that is, determining whether an object (such as a vehicle, pedestrian, or obstacle) exists at a certain location, or predicting the occupancy status in the future.

[0123] In other words, large models not only learn the direct mapping from perception to control, but also achieve a comprehensive understanding of the environment at the internal feature level. This design forces the model's intermediate layer features to not only support the generation of the final control commands, but also to possess multiple capabilities such as perception detection, map understanding, and motion prediction, thereby learning more robust and general feature representations. Multiple tasks sharing the same backbone network can mutually promote and improve each other during training.

[0124] S203. Determine the subtask loss value corresponding to each subtask.

[0125] The loss value is a mathematical metric that measures the difference between the model's predictions and the ground truth labels. For perception detection tasks, cross-entropy loss or bounding box regression loss, commonly used in object detection, can be used to measure the deviation between the target location and category detected by the model and the real target. For map segmentation tasks, pixel-level cross-entropy loss or Dice loss can be used to measure the model's accuracy in classifying pixels such as drivable areas and lane lines. For occupancy prediction tasks, binary cross-entropy loss or Focal loss can be used to measure the difference between the model's prediction of future space occupancy probability and the actual occupancy.

[0126] From an execution logic perspective, the loss value for each subtask is calculated independently, reflecting the model's performance on that specific perceptual dimension. These loss values ​​provide performance feedback for each task in subsequent joint optimization, enabling the model to improve its performance on different perceptual tasks in a targeted manner. The losses from multiple subtasks collectively constitute constraints on the model's intermediate layer features, ensuring that the features learned by the model can support both the perceptual tasks and the final decision-making task.

[0127] As one implementation method, the BEV image segmentation loss L_map corresponding to the map segmentation task can be the cross-entropy loss, which calculates the pixel-level difference between the BEV lane line segmentation map predicted by the model and the ground truth map.

[0128] As one implementation method, the occupancy prediction loss L_occ corresponding to the occupancy prediction task can be Focal loss, which calculates the difference between the probability of each grid being occupied in the next 3 seconds predicted by the model and the true value.

[0129] As one implementation method, the target detection loss L_det for the perception detection task can be calculated using SmoothL1 loss to calculate the bounding box position deviation and cross-entropy loss to calculate the category deviation.

[0130] S204. Determine the control command loss value between the predictive control command and the actual control command.

[0131] Control command loss directly measures the difference between the control signals output by the model (such as accelerator, brake, and steering) and the actual operation signals of a human driver. Mean squared error or L1 loss is typically used to calculate the deviation of continuous control values, ensuring that the model's decision output closely approximates real driving behavior. Control command loss reflects the model's performance on the final task and is a core indicator for measuring the accuracy of the model's driving decisions.

[0132] As one implementation method, the control command loss value L_ctrl can be calculated using mean square error to determine the deviation between the predicted steering wheel angle, throttle opening, and brake pressure and the actual command.

[0133] S205. Construct a comprehensive loss function based on the subtask loss value and the control instruction loss value.

[0134] The control instruction loss is the ultimate optimization goal in training large models. Although the model performs multiple sub-tasks simultaneously, generating accurate control instructions is the ultimate training objective. Sub-task loss can be understood as a regularization or auxiliary supervision mechanism, helping the model learn better intermediate features, while the control instruction loss directly drives the model to optimize in the correct decision direction. The two work together to guide the model to converge to the ideal parameter space. Therefore, a comprehensive loss function needs to be constructed based on the sub-task loss and the control instruction loss.

[0135] The overall loss function is not simply an sum of losses, but rather needs to balance the contributions of each task. A weighted summation approach is typically used, assigning appropriate weight coefficients to the losses of each sub-task and the main task. The setting of these weight coefficients needs to consider the differences in scale, convergence speed, and importance of each task, ensuring that the model can balance the optimization objectives of both the main and auxiliary tasks during training.

[0136] As a feasible implementation method, S205 can be implemented as follows: S2051. Based on the gradient changes or performance indicators of each subtask, dynamically adjust the weights of each subtask so that the weighted loss values ​​of each subtask converge.

[0137] In multi-task joint training, the convergence speed and difficulty of different sub-tasks often vary. Some tasks may have large gradients and converge quickly, easily dominating the model's update direction; others may have small gradients and converge slowly, easily being ignored. To address this issue, the device needs to monitor the gradient changes or performance metrics of each sub-task in real time during training. When a sub-task's gradient is detected to be too large or too small, its weight coefficient is adjusted accordingly; or, based on the current loss value of each sub-task, the weights are dynamically adjusted to keep the weighted loss values ​​of different tasks at similar magnitudes. The purpose of this dynamic adjustment mechanism is to maintain balance among all sub-tasks during training, preventing any single task from dominating the model update due to excessively large gradients, and ensuring that the model converges synchronously across multiple perceptual tasks.

[0138] S2052. Based on the dynamically adjusted weights, perform a weighted summation of the sub-task loss values; After dynamic adjustment by S2051, each subtask is assigned a weight coefficient that matches its current training state. The device multiplies these weight coefficients by the corresponding subtask loss value, and then sums all the weighted subtask loss values ​​to obtain a weighted sum of subtask loss values. This weighted sum comprehensively reflects the model's overall performance across all perceptual subtasks in the current training phase, and because the weights are dynamically adjusted, it avoids the problem of some tasks being ignored or over-optimized.

[0139] S2053. Based on the weighted summation of the subtask loss value and the control instruction loss value, construct a comprehensive loss function.

[0140] The control command loss value represents the loss of the main task, directly measuring the deviation between the model's output control commands and the actual commands. The weighted sum of the sub-task losses reflects the model's overall performance across multiple perception tasks. The device combines these two losses according to a preset ratio, such as by further weighting and summing or using other fusion methods, to form the final comprehensive loss function used for parameter updates. This comprehensive loss function incorporates both the decision objective of the main task and the constraints of multiple perception sub-tasks, enabling the model to simultaneously optimize decision performance and perception capabilities during training, ultimately learning feature representations that can accurately control the vehicle and possess deep environmental understanding capabilities.

[0141] For example, as a feasible implementation, in the early stages of training, initial weights are set based on expert experience: w_map=0.2, w_occ=0.3, w_det=0.3, w_ctrl=0.2. Every preset number of iterations (e.g., 1000 steps), the gradient norm of the loss value for each subtask is calculated. If the gradient norm of a task exceeds 1.5 times the average gradient, its weight is reduced; if it is less than 0.5 times the average gradient, its weight is increased. The goal of dynamic adjustment is to make the weighted loss values ​​of each subtask converge. The final comprehensive loss function is: L_total = w_map·L_map + w_occ·L_occ + w_det·L_det + w_ctrl·L_ctrl Where L_total is the comprehensive loss function; w_map is the weight corresponding to the map segmentation task, and L_map is the loss value corresponding to the map segmentation task; w_occ is the weight corresponding to the occupancy prediction task, and L_occ is the loss value corresponding to the occupancy prediction task; w_det is the weight corresponding to the perception detection task, and L_det is the loss value corresponding to the perception detection task; w_ctrl is the weight corresponding to the control command task, and L_ctrl is the loss value corresponding to the control command prediction task.

[0142] By assigning weights (w) to each task, the model can simultaneously learn environmental perception (L_map, L_occ, L_det) and driving decisions (L_ctrl) during training, thereby improving the model's generalization ability and robustness.

[0143] S206. Update the parameters of the initial large model based on the comprehensive loss function until the training termination condition is met, and obtain the trained large model.

[0144] The parameters of the initial large model are updated based on the constructed comprehensive loss function. The gradient of the loss function with respect to the parameters of each layer of the model is calculated using the backpropagation algorithm, and the parameters are iteratively adjusted using optimizers such as Adam or SGD. This process is repeated on the training set, with each iteration using a batch of training samples to calculate the loss and update the parameters. The training process terminates when the model's performance on the validation set no longer improves or reaches the preset number of training epochs, at which point the trained large model is obtained.

[0145] Parameter updating is the core mechanism of model learning. Through repeated gradient calculations and parameter adjustments, the model gradually learns the mapping pattern from perceived data to control commands, while simultaneously developing a deep understanding of the environment at the internal feature level. Joint optimization of multi-task losses enables the model to continuously enhance its perception capabilities while learning control commands. The resulting large model can accurately generate control commands and contains rich environmental perception information within its internal features, providing a solid model foundation for subsequent end-to-end autonomous driving decisions.

[0146] In some embodiments, to achieve sequential correlation and global optimization in the temporal dimension, a corresponding correlation parameter factor γ is set to realize the sequential linkage of the model. These correlation factors are preset to specific initial values ​​based on expert experience in the early stage of training to constrain the model's behavior in the early stage of training so that it conforms to basic driving common sense; in the subsequent model training iteration optimization process, these parameters will be automatically fine-tuned and corrected with the gradient propagation of the overall network, so that it gradually adapts to the data distribution and tends to the optimum.

[0147] Specifically, these parameter factors are embedded in the raw data from each sensor during the labeling process as a continuous extended sequence. For example, γ1 is embedded in the image sequence acquired by the camera to adjust the decay rate of the temporal features; γ2 is embedded in the point cloud sequence acquired by the lidar to control the fusion weights of historical frame information; and γ3 is embedded in the fused multimodal features to balance the contribution of different modalities to the final decision. These γ1, γ2, and γ3, as learnable network parameters, participate in forward and backward propagation along with other weights of the model, and are ultimately mapped directly to the model's output, outputting vehicle control commands such as steering wheel angle, accelerator pedal opening, and brake pressure.

[0148] In the specific implementation, γ1 is embedded in the image token sequence. When five consecutive frames of image tokens are input into the Transformer, γ1 serves as a scaling factor for positional encoding, controlling the degree of preservation of historical frame information. Its initial value is set to 0.8. γ2 is embedded in the point cloud token sequence. In the sparse convolutional network, it serves as a bias term in the gating mechanism, controlling the influence of distant voxels on the central voxel. Its initial value is set to 0.5. γ3 is embedded in the multimodal fusion layer. After feature concatenation and before inputting into the fully connected layer, each modal feature is multiplied by its corresponding component of γ3. Its initial values ​​are set to [0.7, 0.2, 0.1], corresponding to the contribution weights of the image, point cloud, and radar, respectively. During training, these three parameters, as learnable parameters, participate in backpropagation and gradient update along with other model weights. After 100,000 iterations, γ1 converged to 0.65, indicating that the model tends to decay old frame information faster; γ2 converged to 0.72, indicating that the model found that distant point clouds still have reference value; γ3 converged to [0.6, 0.25, 0.15], indicating that the model makes more balanced use of modal information after training.

[0149] Through this design, the entire model achieves systematic, differentiable, and interconnected joint optimization throughout the entire process from multimodal sensing input to final control command output. Because the correlation parameter factors run throughout the entire time-series processing chain, the model can simultaneously consider the impact of historical information and future control requirements when updating parameters, achieving true time-series linkage. Under this architecture, for any sub-module or intermediate link within the model, separate loss correction and local fine-tuning are no longer required. Instead, end-to-end global optimization directly learns the mapping relationship from the original sensing sequence to the final control command. This system-level optimization approach avoids the error accumulation and local optima problems of traditional modular architectures, facilitating the learning of truly system-optimal control strategies.

[0150] To illustrate the solution provided in this application, the complete implementation process from data acquisition to the generation of control commands is described in detail below with reference to an overall embodiment.

[0151] S11. Acquire multidimensional sensing data.

[0152] During vehicle operation, the controller collects data from various sensors in real time. Cameras capture images of the vehicle ahead at 30 frames per second, generating image data; LiDAR scans the surrounding environment at 10 Hz, generating 3D point cloud data containing hundreds of thousands of points; 4D millimeter-wave radar outputs a target list at 20 Hz, including the distance, speed, horizontal angle, and pitch angle for each target; GPS / IMU outputs the vehicle's latitude, longitude, elevation, heading angle, speed, and acceleration information at 100 Hz; and the controller retrieves high-precision map tiles of the vehicle's surroundings from local storage based on the current GPS location, using this as map data. All data is tagged with a unified global timestamp and time synchronization is used to ensure frame alignment before being temporarily stored in the controller's shared memory.

[0153] S12. Perform dimensionality reduction processing on the multidimensional sensing data and project it onto a shared high-dimensional vector space.

[0154] The controller slices various types of raw data to generate corresponding token sequences. For image data, the controller divides the raw image into 16×16 pixel image blocks. Each image block is linearly mapped and converted into an image token vector, forming an image token sequence. For point cloud data, the controller divides the 3D space into a 0.1-meter square voxel grid. Features are extracted from the point cloud within each non-empty voxel to generate voxel features, forming a point cloud token sequence. For radar data, the controller extracts features from the raw radar targets, obtaining the center position, radial velocity, and radar cross-section of each target. These are then input into a multilayer perceptron to generate a radar token sequence. For GPS / IMU data, the controller slices the continuous pose stream into 100-millisecond time windows. Data within each window is input into a long short-term memory network to extract temporal features, generating a pose token sequence. For map data, the controller performs rasterization segmentation on the local high-precision map. Each raster is input into a convolutional neural network encoder to generate a map token sequence.

[0155] After slicing is completed, the controller attaches a spatiotemporal label to each generated token, recording the time and spatial location information of the original data corresponding to the token in the real world. This information is used to restore the token sequence to its original geometric coordinates during subsequent 3D scene reconstruction.

[0156] Subsequently, the controller inputs the token sequences of all modalities into the corresponding dimensionality reduction networks. Image token sequences are input into a visual Transformer encoder, point cloud token sequences into a sparse convolutional network, radar features into a multilayer perceptron, pose features into a temporal network, and map features into a convolutional network, each outputting a dimensionality-reduced feature vector for its respective modality. Finally, the feature vectors of all modalities are input into a shared projection head, which consists of multiple fully connected layers, uniformly mapping all features into a shared 768-dimensional high-dimensional vector space.

[0157] Combination Figure 4 As shown, the camera acquires image data of the road ahead at a frame rate of 30 frames per second. This image data is first input to the camera encoder module for encoding processing. The camera encoder divides each frame into fixed-size image blocks and generates image features through linear mapping, ultimately outputting camera voxels, i.e., image token sequences. Simultaneously, the LiDAR scans the surrounding environment at a frequency of 10 Hz, generating 3D point cloud data. The point cloud data is voxelized by the LiDAR processing module, dividing the 3D space into 0.1-meter square grids. The point cloud features within each non-empty grid are extracted by a point cloud feature extraction network to generate LiDAR voxels, i.e., radar token sequences. Both voxel sequences are attached with 3D spatial coordinates (X, Y, Z) during the generation process, providing a foundation for subsequent spatiotemporal alignment and geometric position reconstruction.

[0158] S13. Perform vector aggregation in a high-dimensional vector space.

[0159] The controller concatenates all the modal feature vectors obtained after projection into a long feature sequence, which is then input into a multi-layer Transformer encoder for vector aggregation. Under the Transformer's self-attention mechanism, each feature vector calculates its similarity to all other vectors; vector pairs with high similarity attract each other and move closer together during the update process. After multiple iterations, the distance between feature vectors originally describing the same physical object but from different modalities gradually decreases in high-dimensional space. For example, image features, point cloud features, and radar features describing the same vehicle ahead are eventually aggregated together to form a feature cluster. When the cosine distance between feature vectors of the same object is less than a preset threshold of 0.3, the aggregation process is complete, and the aggregated multi-modal fusion feature is output.

[0160] Combination Figure 4As shown, camera voxels (image token sequences) and LiDAR voxel sequences (radar token sequences) with spatial coordinates are input into a shared backbone network. The backbone network employs a 3D sparse convolutional neural network structure to deeply extract and fuse multimodal voxel features, outputting a fused high-dimensional feature map. This feature map has two flows: one flows to the bird's eye view (BEV) module, converting the 3D spatial features into a 2D feature representation from the bird's eye view perspective, facilitating subsequent path planning and scene understanding; the other flows to the focus inference module, which uses an attention mechanism to focus inference on key areas in the scene (such as vehicles, pedestrians, intersections, etc.) to extract more refined local features.

[0161] S14. Generate target control instructions based on the restored 3D scene information.

[0162] The controller inputs the aggregated multimodal fusion features into the trained large model. The large model first performs scene reconstruction on the fusion features to obtain the structured information of the 3D scene. During the scene reconstruction process, the large model assigns different weights to features of different modalities, with image features having the highest weight because image data provides the richest semantic information. The reconstructed 3D scene information includes: the vehicle is currently in the middle lane of a three-lane road, there is a stationary disabled vehicle occupying the right lane 50 meters ahead, a car is rapidly approaching from the left lane about 30 meters behind, the current lane is a tidal flow lane allowing straight travel, and the traffic light at the intersection is green.

[0163] Based on this scenario information, the large model continues to make driving decisions. For example, the large model first evaluates several possible behaviors: maintaining the current lane but needing to avoid the disabled vehicle, changing lanes to the left to overtake the disabled vehicle, and slowing down to wait for the vehicle on the left to pass before changing lanes. After comprehensive reasoning, the large model determines that the vehicle on the left is traveling at a relatively high speed, and changing lanes immediately would pose a collision risk; therefore, it chooses to slow down and wait. The specific plan is as follows: first, decelerate at a rate of 2 meters per square second to 30 kilometers per hour, wait for the vehicle on the left to pass, then accelerate at a rate of 3 meters per square second and change lanes to the left to overtake the disabled vehicle before returning to the original lane. The large model translates this planned trajectory into specific control commands: outputting a throttle opening of 0% (actually coasting deceleration) or braking pressure corresponding to a deceleration of 2 meters per square second, and maintaining a steering wheel angle of 0 degrees. These control commands are sent via the CAN bus to the vehicle's drive-by-wire, brake-by-wire, and steering-by-wire units, and the actuators control the vehicle to complete the deceleration operation according to the commands.

[0164] Combination Figure 4As shown, in the decision output and optimization stage, the voxel generation module dynamically generates additional query voxels based on the current scene, which are then input into subsequent networks along with the features output by the backbone network. These features are weighted and adjusted by the parameter factor module, which is a set of learnable correlation parameter factors γ used to balance the contribution of different modal features and different time frames to the final decision. The features adjusted by the parameter factor are finally input into the vehicle control module, which consists of a multi-layer fully connected network and directly outputs vehicle control commands, including steering wheel angle, throttle opening, and brake pressure.

[0165] The entire network training process combines supervised learning and reinforcement learning mechanisms: gradient backpropagation is performed by calculating the deviation between predicted and actual commands using a focus loss function, enabling the model to focus on key, difficult-to-identify areas (such as distant pedestrians or occluded vehicles). Simultaneously, reinforcement learning algorithms are introduced to iteratively optimize the decision-making strategy, allowing the model to learn better control strategies in complex scenarios. During feature processing, the focus inference module performs deep logical analysis of key areas in the scene based on an attention mechanism, understanding the interaction relationships and potential intentions between objects. Finally, the model outputs prediction results, including the future trajectories of surrounding targets and the vehicle's next action, thereby controlling the vehicle's movement.

[0166] As can be seen from the above embodiments, the solution provided in this application uses a unified deep learning model to directly map the raw data from multimodal sensors to the final vehicle control signal, achieving integrated representation learning from perception to decision-making. This end-to-end algorithm architecture omits the complex intermediate modules found in related solutions, eliminating the need for independent detectors, trackers, planners, etc. Instead, it uses a large model based on the Transformer architecture, leveraging its powerful representation learning and contextual understanding capabilities to transform the pipelined processing from "perception to decision-making" into integrated representation learning.

[0167] It can be seen that the technical solution provided in this application has the following advantages: First, combining Figure 5As shown, the design based on linear group RNN can feed image data (2D planar data provided by image sensors, including coordinates u, v and width and height w, h), radar data (2D radial distance data provided by radar, including distance r and angle θ), point cloud data (3D spatial data (X, Y, Z coordinates) provided by LiDAR), as well as navigation maps, vehicle kinematics information, and GPS / IMU positioning data into a unified 3D backbone network. The network performs multimodal feature extraction, spatiotemporal fusion, and depth processing, ultimately directly outputting the vehicle's next control command, achieving a complete end-to-end mapping from multi-source perception input to driving decisions. This design eliminates the complex spatiotemporal alignment process, multi-sensor accuracy matching, and redundant data processing required in traditional architectures, significantly simplifying the system architecture.

[0168] Second, the overall architecture adopts a direct input of raw data, without intermediate pooling layers, convolutional extraction layers, or information compression processes within the model. A reverse correlation mechanism is implemented through parameter factors, enabling the model to maintain the integrity of its input information perception while allowing for error retrospective adjustment during training. This design significantly improves the model's generalization ability, alleviates overfitting, and enhances the efficiency of utilizing effective data.

[0169] Third, the architecture achieves a direct mapping from raw perception data to planning decisions, avoiding the error accumulation problem caused by the serial connection of multiple independent modules in traditional modular architectures. For vehicle control decisions in assisted intelligent driving, this end-to-end direct mapping can effectively balance decision speed and decision accuracy, providing a relatively optimal control solution while ensuring real-time performance, and significantly improving the reliability of the system in extreme scenarios.

[0170] In an exemplary embodiment, this application also provides a control command generation device. This control command generation device can be the vehicle controller described above, or it can be a command generation device; this application does not limit its scope. Please refer to [link to relevant documentation]. Figure 6 The control command generation device 60 includes: an acquisition module 61, a projection module 62, an aggregation module 63, and a generation module 64.

[0171] The acquisition module 61 is used to acquire multi-dimensional perception data during vehicle operation; the multi-dimensional perception data includes data in at least two of the following dimensions: image data, point cloud data, radar data, GPS / IMU data, and map data; Projection module 62 is used to reduce the dimensionality of multidimensional sensing data and project it onto a shared high-dimensional vector space. The aggregation module 63 is used to perform vector aggregation in a high-dimensional vector space so that the distance between the feature vectors describing the same physical object in different dimensions of the multidimensional perceptual data in the high-dimensional vector space is less than a preset threshold. The generation module 64 is used to generate target control commands adapted to the three-dimensional scene based on the scene information of the three-dimensional scene reconstructed from the high-dimensional vector space; the scene information of the three-dimensional scene is used to characterize the driving environment in which the vehicle is currently located; the target control commands are used to control the vehicle to achieve driving behavior that conforms to the scene geometry and dynamic constraints in the environment corresponding to the three-dimensional scene description.

[0172] In some embodiments, the projection module 62 is specifically used to: slice the data of each dimension in the multidimensional sensing data to generate a corresponding token sequence; and perform dimensionality reduction processing on the token sequence corresponding to each dimension of the multidimensional sensing data.

[0173] In some embodiments, the projection module 62 is specifically used to: for image data, divide it into multiple image blocks and perform linear mapping to generate an image token sequence; and / or, for point cloud data, perform voxel encoding to generate a point cloud token sequence; and / or, for radar data, perform target detection and point cloudification processing to generate a radar token sequence; and / or, for GPS / IMU data, perform data slicing processing based on preset slicing parameters to generate a pose token sequence; and / or, for map data, perform raster segmentation or vector slicing processing to generate a map token sequence.

[0174] In some embodiments, the control instruction generation apparatus further includes: a resolution processing module, configured to reduce the sampling frequency of point cloud data when the amount of point cloud data exceeds a preset data amount threshold or the resolution exceeds a preset resolution threshold; and / or to employ an attention sparsity mechanism on the point cloud token sequence generated after point cloud data conversion, so as to reduce the computational load of self-attention calculation.

[0175] In some embodiments, the generation module 64 is further configured to attach a spatiotemporal label to the token sequence generated for each slice when processing the multidimensional sensing data into slices; the spatiotemporal label is used to restore the token sequence to the original spatiotemporal corresponding geometric coordinates when restoring the scene information of the three-dimensional scene.

[0176] In some embodiments, the generation module 64 is specifically used for: inputting the multimodal fusion features obtained through vector aggregation into a large model, and having the large model perform scene reconstruction on the multimodal fusion features to obtain scene information of the three-dimensional scene; wherein, during the scene reconstruction process, the large model assigns weights to the features corresponding to different modal data respectively; the features corresponding to image data are assigned higher weights than other features; and the large model generates target control instructions adapted to the three-dimensional scene based on the scene information of the three-dimensional scene.

[0177] In some embodiments, the control instruction generation apparatus further includes: an association module, used to associate the original data lost during the dimensionality reduction process with a high-dimensional vector space using a character mapping method; the character mapping is implemented based on a predefined character dictionary and is used to call the corresponding original data according to the usage scenario in subsequent tasks.

[0178] In some embodiments, this application also provides a model training apparatus for training the large model mentioned in the above embodiments. This model training apparatus can be the same as the control command generation apparatus, or it can be a separate apparatus; this application does not impose any limitations on this.

[0179] In some embodiments, the model training apparatus includes: a sample acquisition module, an input module, a determination module, a loss construction module, and a training module. The sample acquisition module acquires multiple sample multidimensional perception data and their corresponding real control commands. The input module inputs the sample multidimensional perception data into an initial large model, which outputs predicted control commands and executes multiple sub-tasks. These sub-tasks include at least a perception detection task, a map segmentation task, and an occupancy prediction task. The determination module determines the sub-task loss value for each sub-task and the control command loss value between the predicted control commands and the real control commands. The loss construction module constructs a comprehensive loss function based on the sub-task loss values ​​and the control command loss values. The training module updates the parameters of the initial large model based on the comprehensive loss function until the training termination condition is met, resulting in a trained large model.

[0180] In some embodiments, the loss construction module is specifically used to: dynamically adjust the weights of each subtask according to the gradient changes or performance indicators of each subtask, so that the weighted loss values ​​of each subtask are close to each other; perform a weighted summation of the subtask loss values ​​based on the dynamically adjusted weights; and construct a comprehensive loss function based on the weighted summation of the subtask loss values ​​and the control instruction loss values.

[0181] In an exemplary embodiment, this application also provides an autonomous driving system, including: a control command generation device and a data acquisition device provided in the above embodiments; the data acquisition device is used to acquire multi-dimensional perception data during vehicle operation and send the multi-dimensional perception data to the control command generation device, so that the control command generation device can implement autonomous driving function using the control command generation method provided in the above embodiments.

[0182] In an exemplary embodiment, this application also provides a vehicle including the control command generation device provided in the above embodiments; or, including the autonomous driving system provided in the above embodiments.

[0183] In an exemplary embodiment, this application also provides an electronic device that may include a processor and a memory. The processor may be a computing cluster composed of multiple computing nodes, and the memory may adopt a distributed memory architecture. The processor integrated into the computing device is configured to execute the task implementation method of any of the above embodiments.

[0184] Figure 7 This is a schematic diagram of the architecture of an instruction generation device provided in an embodiment of this application. Figure 7 As shown, the instruction generation device 201 includes: one or more memories 120, one or more processors 110, a communication bus 140, and a communication interface 130. The processors 110 and memories 120 are connected via the communication bus 140; the one or more memories 120 are used to store computer program code, which includes computer instructions; when the one or more processors 110 execute the computer instructions, the instruction generation device 201 executes the control instruction generation method provided in this embodiment.

[0185] Optionally, the memory 120 may be a non-transitory computer-readable storage medium, such as a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. The embodiments of this application do not impose any limitations on this.

[0186] The processor 110 may be a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof, and the embodiments of this application do not impose any limitations on this.

[0187] The communication bus 140 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. This communication bus 140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7It is represented by a single thick line, but this does not mean that there is only one bus or one type of communication bus.

[0188] Communication interface 130 uses any transceiver-like device for communicating with other devices or communication networks, such as control systems, radio access networks (RAN), wireless local area networks (WLAN), etc.

[0189] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware; for example, the related hardware can be a processor of a computing device. The program instructions can be stored in the above-described computer-readable storage medium, and when executed, they can implement the processes of the above method embodiments. The computer-readable storage medium can be memory. The above-described computer-readable storage medium can also be an external storage device, such as a hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Further, the above-described computer-readable storage medium can include both memory and external storage devices. The above-described computer-readable storage medium is used to store the above-described computer program instructions and other programs and data required for the above-described control instruction generation method.

[0190] This application also provides a computer program product comprising a computer program that, when run on a computing device, causes the computing device to execute any of the control instruction generation methods provided in the above embodiments.

[0191] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0192] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0193] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating control commands, characterized in that, include: Acquire multi-dimensional perception data during vehicle operation; The multidimensional sensing data includes data from at least two of the following dimensions: image data, point cloud data, radar data, GPS / IMU data, and map data; The multidimensional sensing data is then dimensionality reduced and projected onto a shared high-dimensional vector space. Vector aggregation is performed in the high-dimensional vector space so that the distance between feature vectors describing the same physical object in different dimensions of the multidimensional sensing data in the high-dimensional vector space is less than a preset threshold. Based on the scene information of the three-dimensional scene reconstructed from the high-dimensional vector space, a target control command adapted to the three-dimensional scene is generated; the scene information of the three-dimensional scene is used to characterize the driving environment in which the vehicle is currently located; the target control command is used to control the vehicle to achieve driving behavior that conforms to the scene geometry and dynamic constraints in the environment corresponding to the description of the three-dimensional scene.

2. The generation method according to claim 1, characterized in that, The step of reducing the dimensionality of the multidimensional sensing data includes: The data of each dimension in the multidimensional sensing data is sliced ​​to generate a corresponding token sequence; The dimensionality reduction process is performed on the token sequence corresponding to each dimension in the multidimensional sensing data.

3. The generation method according to claim 2, characterized in that, The step of slicing the data in each dimension of the multidimensional sensing data to generate a corresponding token sequence includes: For the image data, it is divided into multiple image blocks and linearly mapped to generate an image token sequence; and / or, For the point cloud data, voxelization encoding is performed to generate a point cloud token sequence; and / or, For the radar data, target detection and point cloudification processing are performed to generate a radar token sequence; and / or, For the GPS / IMU data, data slicing processing is performed based on preset slicing parameters to generate a pose token sequence; and / or, The map data is rasterized or tiled to generate a map token sequence.

4. The generation method according to claim 2, characterized in that, If the amount of point cloud data exceeds a preset data amount threshold or the resolution exceeds a preset resolution threshold, the method further includes: Reduce the sampling frequency of the point cloud data; and / or, The point cloud token sequence generated after the point cloud data conversion adopts an attention sparsity mechanism to reduce the computational cost of self-attention calculation.

5. The generation method according to claim 1, characterized in that, The method further includes: When slicing the multidimensional sensing data, a spatiotemporal label is attached to the token sequence generated for each slice; the spatiotemporal label is used to restore the token sequence to the original spatiotemporal corresponding geometric coordinates when restoring the scene information of the three-dimensional scene.

6. The generation method according to any one of claims 1-5, characterized in that, The step of generating target control commands adapted to the three-dimensional scene based on scene information reconstructed from the high-dimensional vector space includes: The multimodal fusion features obtained through vector aggregation are input into a large model, which then performs scene reconstruction on the multimodal fusion features to obtain the scene information of the 3D scene. During the scene reconstruction process, the large model assigns weights to the features corresponding to different modal data. The features corresponding to the image data are assigned higher weights than other features. The large model generates target control commands adapted to the three-dimensional scene based on the scene information of the three-dimensional scene.

7. The generation method according to claim 6, characterized in that, The large model was trained in the following manner: Acquire multidimensional sensing data from multiple samples and their corresponding real control commands; The multidimensional sensing data of the samples is input into the initial large model, which outputs prediction control commands and executes multiple sub-tasks; the multiple sub-tasks include at least a sensing detection task, a map segmentation task, and an occupancy prediction task. Determine the subtask loss value for each subtask; Determine the control command loss value between the predicted control command and the actual control command; A comprehensive loss function is constructed based on the subtask loss value and the control command loss value; The parameters of the initial large model are updated based on the comprehensive loss function until the training termination condition is met, resulting in a trained large model.

8. The generation method according to claim 7, characterized in that, The construction of the comprehensive loss function based on the subtask loss value and the control command loss value includes: Based on the gradient changes or performance indicators of each subtask, the weights of each subtask are dynamically adjusted so that the weighted loss values ​​of each subtask converge. The sub-task loss values ​​are weighted and summed based on the dynamically adjusted weights. The comprehensive loss function is constructed based on the weighted summation of the subtask loss value and the control command loss value.

9. The generation method according to claim 1, characterized in that, The method further includes: For the original data lost during the dimensionality reduction process, a character mapping method is used to associate it with the high-dimensional vector space; The character mapping is implemented based on a predefined character dictionary and is used to call the corresponding original data in subsequent tasks according to the usage scenario.

10. A control command generation device, characterized in that, include: The acquisition module is used to acquire multi-dimensional perception data during vehicle operation. The multidimensional sensing data includes data from at least two of the following dimensions: image data, point cloud data, radar data, GPS / IMU data, and map data; The projection module is used to reduce the dimensionality of the multidimensional sensing data and project it onto a shared high-dimensional vector space. An aggregation module is used to perform vector aggregation in the high-dimensional vector space, so that the distance between feature vectors describing the same physical object in different dimensions of the multidimensional sensing data in the high-dimensional vector space is less than a preset threshold. The generation module is used to generate target control instructions adapted to the three-dimensional scene based on the scene information of the three-dimensional scene reconstructed from the high-dimensional vector space; the scene information of the three-dimensional scene is used to characterize the driving environment in which the vehicle is currently located; the target control instructions are used to control the vehicle to achieve driving behavior that conforms to the scene geometry and dynamic constraints in the environment corresponding to the description of the three-dimensional scene.

11. An autonomous driving system, characterized in that, include: The data acquisition device and the control command generation device as described in claim 10; The acquisition device is used to acquire multi-dimensional perception data during vehicle operation and send the multi-dimensional perception data to the control command generation device so that the control command generation device can implement the autonomous driving function using the control command generation method as described in any one of claims 1-9.

12. A vehicle, characterized in that, It includes the control command generation device as described in claim 10; or, it includes the autonomous driving system as described in claim 11.

Images