Big language model driven integrated space-earth automatic driving intelligent decision system and method

Abstract

The application discloses a space-ground integrated automatic driving intelligent decision system and method driven by a large language model, which comprises a satellite layer, a vehicle end layer and a cloud end layer. Through efficient data interaction and collaborative processing, precise, intelligent and efficient automatic driving decisions are realized. The satellite layer comprises a low-orbit satellite constellation, which transmits high-precision positioning information to vehicles in real time and provides a space-time reference. The vehicle end layer is an executor, which collects environmental data in real time through sensors and uses high-precision positioning data. An end-to-end rapid decision model built in the vehicle end layer generates preliminary driving decisions. At the same time, driving data, environmental perception data and decision requirements are uploaded to the cloud end. The cloud end remotely monitors and manages the system, evaluates vehicle status and issues early warnings. When the vehicle is safe, the vehicle end decision is adopted. When there is a potential risk or an abnormal situation, the large language model decision is adopted. The optimized instruction, updated model parameter or emergency disposal scheme are pushed to the vehicle end to assist the vehicle end in correcting the decision.

Description

Technical Field

[0001] This invention relates to the fields of intelligent transportation systems and autonomous driving, specifically to a large language model-driven integrated space-ground autonomous driving intelligent decision-making system and method. Background Technology

[0002] Autonomous driving technology, as a revolutionary breakthrough in the transportation field, has received widespread attention and made significant progress. Despite these achievements, existing methods often suffer from limitations in sensor field of view and accuracy when faced with complex road conditions, unexpected situations, and long-tail events. These limitations often expose problems such as insufficient learning ability and poor generalization performance, making it difficult to cope with the diversity and uncertainty of the real world.

[0003] With the rapid development of artificial intelligence technology, large language models, with their powerful language understanding, reasoning, and generation capabilities, have demonstrated enormous application potential in the field of autonomous driving decision-making. Large language models can process and understand massive amounts of text information, possessing the ability to logically analyze and judge complex scenarios, providing rich knowledge support and intelligent guidance for autonomous driving decision-making. However, deploying large language models on the vehicle side, processing massive amounts of sensor data in real time, places extremely high demands on the computing power of the onboard computing unit, increasing cost and power consumption. Meanwhile, the integrated space-ground network has brought new opportunities for autonomous driving. Low-Earth orbit satellites, as a key component of the integrated space-ground network, can provide high-precision positioning information with wide coverage, unaffected by ground obstacles, effectively compensating for the limitations of ground sensors in complex environments, providing autonomous vehicles with more comprehensive and accurate spatiotemporal information, and helping vehicles understand their own location and surrounding situation in real time. Against this backdrop, this invention proposes a large language model-driven integrated space-ground autonomous driving intelligent decision-making system, encompassing the satellite layer, vehicle layer, and cloud layer. By providing wide-area spatiotemporal information through low-orbit satellites, the vehicle collects and analyzes environmental data in real time, and the cloud utilizes the knowledge reserves of large language models and powerful computing power to optimize decisions, providing innovative solutions to the decision-making dilemma of autonomous driving. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a large language model-driven integrated space-ground autonomous driving intelligent decision-making system, which fully leverages the advantages of space-ground collaboration, effectively integrates multi-source information, and achieves high-precision and high-reliability autonomous driving decisions.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0006] This invention designs a large language model-driven integrated space-ground autonomous driving intelligent decision-making system (hereinafter referred to as the integrated space-ground decision-making system). The implementation method of this decision-making system includes the following steps:

[0007] Step 1: Construct a satellite layer as a high-altitude information node for the integrated space-ground system. The satellite layer consists of a low-Earth orbit satellite constellation.

[0008] Step 2: Construct the vehicle-mounted layer as the direct executor of the integrated space-ground system. The vehicle-mounted layer is equipped with multiple sensors and has preliminary decision-making capabilities.

[0009] Step 3: Construct a cloud layer as the central brain of the integrated space-ground system, possessing super computing power and massive data storage capabilities;

[0010] Step 4: Form a tight closed-loop interactive process by combining the layers built in Steps 1, 2 and 3.

[0011] Furthermore, the satellite layer in step 1 consists of a low-Earth orbit satellite constellation, which can provide centimeter-level high-precision positioning information.

[0012] The location information includes both dynamic and static features:

[0013] Dynamic characteristics include vehicle speed, acceleration, and heading angle changes. Real-time monitoring of vehicle speed provides a clear picture of its current travel rate. Acceleration characteristics are closely related to the dynamic changes in the vehicle's power performance and driving status. The heading angle change rate reflects the rate of change in the vehicle's direction of travel and is crucial for determining the vehicle's steering intentions and handling agility.

[0014] Static features include the vehicle's absolute position information and relative position features. With the help of high-precision maps, vehicles can accurately know their own lane, road segment, and relative positional relationship with surrounding landmarks and traffic facilities, as well as the spatial distribution of other traffic participants in the vicinity.

[0015] The positioning information composed of dynamic and static features is interconnected and works synergistically to provide comprehensive and multi-level information support for autonomous driving decisions.

[0016] Furthermore, the vehicle-side layer in step 2 is equipped with a variety of sensors, including lidar, cameras, and millimeter-wave radar. These sensors collect environmental data about the vehicle's surroundings in real time from different dimensions.

[0017] LiDAR can accurately construct a three-dimensional point cloud map of the area surrounding a vehicle.

[0018] Cameras capture visual images to identify targets such as traffic signs, traffic lights, pedestrians, and other vehicles.

[0019] Millimeter-wave radar focuses on detecting the speed and distance information of target objects.

[0020] The vehicle-side layer incorporates an end-to-end rapid decision-making system. This system utilizes an efficient end-to-end model to deeply fuse, analyze, and infer multi-source sensor data. Combined with high-precision positioning information provided by low-Earth orbit satellites, it rapidly generates driving decisions adapted to complex road conditions, including speed and steering. The end-to-end rapid decision-making system is implemented using a multi-modal fusion Transformer architecture, specifically:

[0021] The input features include image data I∈R H×W×C LiDAR point cloud data P1∈R N×C Millimeter-wave radar point cloud data P2∈R N×C And high-precision positioning information M, where H and W are the length and width of the image, C is the number of channels, and N is the number of point clouds.

[0022] For image data I, it is divided into multiple small blocks, and each small block is regarded as a sequence element F. I It is mapped to a fixed-dimensional feature vector through linear projection, and positional encoding is added to preserve spatial location information.

[0023] For features P1 and P2 of lidar and millimeter-wave radar, convert them into feature vector sequences F. P And add corresponding sensor embeddings to distinguish features from different sensor sources.

[0024] For high-precision positioning information, based on latitude, longitude, heading angle, etc., it is encoded into a position feature sequence F. M .

[0025] The above sequence is then used as input to a multimodal fusion Transformer. In the Transformer encoder, a multi-head self-attention mechanism is used to interactively fuse feature sequences from different sensors. This mechanism can simultaneously focus on different subspaces of the input sequence, capturing the complex relationships between features from different sensors to obtain the fused features. Specifically:

[0026] First, the input sequence is linearly transformed to generate query (Q), key (K), and value (V) matrices, with each head corresponding to an independent weight matrix. The Q, K, and V matrices are then partitioned according to the number of heads, with each head having a processing dimension of d. model The subvector of / h. During attention computation, each head performs a scaled dot product attention operation in parallel: for the i-th head, its attention score is:

[0027]

[0028] Here, softmax is the activation function. The fused features are then obtained, which can be represented as:

[0029] output i=score i V i

[0030] The features fused by the encoder are then input into the Transformer decoder, and combined with the preset driving target, continuous decision control commands such as speed and steering angle are generated.

[0031] Furthermore, the cloud layer in step 3 is responsible for managing and maintaining the massive database, providing data support for both vehicle and satellite terminals. Specifically:

[0032] The cloud storage architecture employs a distributed file system, providing high scalability, high availability, and high-performance data storage capabilities. These distributed file systems distribute data across multiple nodes, employing data redundancy backup strategies to ensure that data can still be retrieved intact even if some nodes fail, guaranteeing continuous system operation.

[0033] Based on their characteristics and uses, stored data can be categorized into several types: high-precision maps, historical traffic data, real-time traffic information, and various updated traffic regulations.

[0034] The high-precision map data includes detailed information such as road topology, lane line information, and traffic sign locations, and is stored in vector format for easy querying and precise navigation.

[0035] Historical traffic data includes traffic flow, congestion, and accident records for different time periods and road sections, providing a reference for road condition prediction and driving route planning. It is stored in a time-series database to support efficient time range queries.

[0036] Real-time traffic information focuses on the dynamic changes in the road, such as road construction, temporary traffic control, and traffic congestion caused by emergencies. It is obtained through multiple sources, including real-time reporting from vehicles and satellite remote sensing monitoring, and is updated in real time in the form of a message queue to ensure the timeliness of the data.

[0037] Various traffic regulations and driving knowledge documents are stored in text format, making it easy for large language models to retrieve and learn at any time, and update their knowledge reserves.

[0038] When data grows in massive quantities, distributed systems can expand their capacity by adding storage nodes to meet the ever-increasing data storage demands.

[0039] Furthermore, the powerful cloud-based large language model running in step 3 continuously optimizes its knowledge system through deep mining and learning of massive amounts of data, providing knowledge distillation for the vehicle-side model and enhancing its decision-making capabilities. Specifically:

[0040] The large language model adopts a GPT-based architecture, and its input is massive multi-source data. On the one hand, it includes massive text knowledge, traffic rules, accident cases, and scenario response logic. On the other hand, it integrates actual driving data uploaded by the vehicle, including sensor environmental data and decision sequences, so that the model closely fits the real driving situation.

[0041] In terms of model training algorithms, a pre-training-fine-tuning paradigm based on the Transformer architecture is adopted.

[0042] The pre-training phase utilizes a large-scale unsupervised corpus for pre-training, enabling the model to learn general language patterns and semantic understanding capabilities.

[0043] The fine-tuning phase combines specific autonomous driving tasks with real-world scenario data for fine-tuning, enhancing the model's ability to identify risks and optimize decisions.

[0044] Furthermore, the cloud layer in step 3 undertakes the task of remote monitoring and management of the entire system, collects driving data, sensor status and decision execution feedback information uploaded by the vehicle in real time, evaluates and warns the vehicle's operating status, follows the vehicle's decision when the driving scenario is safe, adopts the decision of the large language model when potential risks or abnormal situations are found, pushes optimization instructions, updated model parameters or emergency response plans to the vehicle in real time to assist the vehicle in correcting its decision, and initiates the final decision of emergency braking and pulling over to the side of the road when the system is unable to make a decision in extremely complex emergency scenarios (such as traffic accidents).

[0045] Furthermore, the specific details of the closed-loop interaction process in step 4 are as follows:

[0046] The satellite layer continuously transmits high-precision positioning information and macroscopic environmental data to the vehicle and the cloud.

[0047] Based on this information, the vehicle collects and processes local sensor data in real time, uses an end-to-end rapid decision-making system to generate preliminary driving decisions, and uploads driving data, environmental perception data, and decision requirements to the cloud in real time.

[0048] The cloud platform combines its own stored data with powerful computing capabilities to comprehensively analyze the information uploaded by the vehicle and assess the vehicle's operating status. When the driving scenario is safe, the vehicle's decision is adopted directly. When the driving scenario is abnormal, the cloud platform uses a large language model to provide knowledge enhancement and model optimization suggestions to the vehicle and feeds back the processing results to the vehicle in real time to help the vehicle correct its decisions.

[0049] The vehicle further optimizes driving operations based on feedback from the cloud and feeds back the results to the cloud, enabling the system to learn and optimize continuously, thereby ensuring the safe and efficient operation of autonomous vehicles in various complex scenarios.

[0050] The beneficial effects of this invention are:

[0051] 1) This invention proposes a space-ground integrated autonomous driving intelligent decision-making system driven by a large language model. By making full use of the high-precision positioning information of low-orbit satellites, it organically combines the ground vehicle-side rapid decision-making system and the cloud-based large language model, enabling the autonomous driving decision-making system to perceive the environment more comprehensively and accurately, and to respond to complex road conditions more intelligently and flexibly, thereby improving the accuracy and reliability of decision-making.

[0052] 2) This invention constructs an end-to-end decision optimization closed-loop system, realizing full-process automation and intelligence from input information collection and model decision generation to execution result feedback. Based on the actual driving status of the vehicle fed back by the actuator, the system evaluates the decision-making effect in real time. Once a deviation or anomaly is detected, it immediately triggers the re-decision mechanism of the large language model and combines integrated space-ground information for targeted optimization and adjustment, ensuring that the vehicle always travels along the optimal trajectory, comprehensively improving the safety and stability of autonomous driving.

[0053] 3) With the support of high-precision positioning, this invention effectively reduces the configuration requirements of vehicle-side sensors, which can not only reduce the hardware cost of the vehicle, but also reduce the complexity of vehicle-side data processing, and further improve the overall operating efficiency of the system.

[0054] 4) The multi-node parallel data processing design of the cloud layer in this invention significantly improves data access speed, reduces latency, and meets the stringent real-time requirements of autonomous driving systems. Simultaneously, the redundant backup mechanism effectively mitigates risks such as hardware failures and network interruptions, ensuring data security and integrity, and laying a solid foundation for the stable operation of the autonomous driving decision-making system. Attached Figure Description

[0055] Figure 1 This invention provides an architecture diagram of a large language model-driven integrated space-ground autonomous driving intelligent decision-making system.

[0056] Figure 2 A cloud-based distributed file storage architecture diagram provided by this invention;

[0057] Figure 3 The vehicle decision-making process flowchart provided by this invention; Detailed Implementation

[0058] The present invention will be further described below with reference to the accompanying drawings. The drawings are provided to further understand the embodiments of the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the present disclosure and do not constitute a limitation thereof.

[0059] This invention proposes a large language model-driven integrated space-ground autonomous driving intelligent decision-making system, such as... Figure 1The diagram shown illustrates an architecture of a large language model-driven integrated space-ground autonomous driving intelligent decision-making system for implementation of this application, which mainly includes the following steps:

[0060] Step 1: Construct a satellite layer as a high-altitude information node for the integrated space-ground system. The satellite layer consists of a low-Earth orbit satellite constellation.

[0061] Furthermore, the satellite layer consists of a constellation of low-Earth orbit satellites, which can provide centimeter-level high-precision positioning information.

[0062] The location information includes both dynamic and static features:

[0063] The dynamic characteristics include the vehicle's speed, acceleration, and heading angle changes.

[0064] Real-time monitoring of vehicle speed can provide a direct indication of how fast or slow the vehicle is moving.

[0065] Acceleration characteristics are closely related to the dynamic changes in a vehicle's power performance and driving status.

[0066] The rate of change of heading angle reflects the rate of change of the vehicle's direction of travel and is crucial for judging the vehicle's steering intention and handling agility.

[0067] Static features include the vehicle's absolute position information and relative position features. With the help of high-precision maps, vehicles can accurately know their own lane, road segment, and relative positional relationship with surrounding landmarks and traffic facilities, as well as the spatial distribution of other traffic participants in the vicinity.

[0068] The positioning information composed of dynamic and static features is interconnected and works synergistically to provide comprehensive and multi-level information support for autonomous driving decisions.

[0069] Step 2: Construct the vehicle-mounted layer as the direct executor of the integrated space-ground system. The vehicle-mounted layer is equipped with multiple sensors and has preliminary decision-making capabilities.

[0070] Furthermore, the vehicle-mounted layer is equipped with a variety of sensors, including lidar, cameras, and millimeter-wave radar. These sensors collect environmental data about the vehicle's surroundings in real time from different dimensions.

[0071] LiDAR can accurately construct a three-dimensional point cloud map of the area surrounding a vehicle.

[0072] Cameras capture visual images to identify targets such as traffic signs, traffic lights, pedestrians, and other vehicles.

[0073] Millimeter-wave radar focuses on detecting the speed and distance information of target objects.

[0074] The vehicle-side layer incorporates an end-to-end rapid decision-making system. This system utilizes an efficient end-to-end model to deeply fuse, analyze, and infer multi-source sensor data. Combined with high-precision positioning information provided by low-Earth orbit satellites, it rapidly generates driving decisions adapted to complex road conditions, including speed and steering. The end-to-end rapid decision-making system is implemented using a multi-modal fusion Transformer architecture, specifically:

[0075] The input features include image data I∈R H×W×C LiDAR point cloud data P1∈R N×C Millimeter-wave radar point cloud data P2∈R N×C And high-precision positioning information M, where H and W are the length and width of the image, C is the number of channels, and N is the number of point clouds.

[0076] For image data I, it is divided into multiple small blocks, and each small block is regarded as a sequence element F. I It is mapped to a fixed-dimensional feature vector through linear projection, and positional encoding is added to preserve spatial location information.

[0077] For features P1 and P2 of lidar and millimeter-wave radar, convert them into feature vector sequences F. P And add corresponding sensor embeddings to distinguish features from different sensor sources.

[0078] For high-precision positioning information, based on latitude, longitude, heading angle, etc., it is encoded into a position feature sequence F. M .

[0079] The above sequence is then used as input to a multimodal fusion Transformer. In the Transformer encoder, a multi-head self-attention mechanism is used to interactively fuse the feature sequences from different sensors. The multi-head self-attention mechanism can simultaneously focus on different subspaces of the input sequence, capture the complex correlation between features from different sensors, and obtain the fused features.

[0080] Here, we analyze a specific example: When processing a sequence containing vehicle image features, LiDAR point cloud features, and millimeter-wave radar target features, the model can learn the intrinsic relationship between vehicle appearance features, vehicle contours measured by LiDAR, and vehicle speed detected by millimeter-wave radar through an attention mechanism, thereby achieving more accurate object perception.

[0081] Specifically:

[0082] First, the input sequence is linearly transformed to generate query (Q), key (K), and value (V) matrices. A multi-head self-attention mechanism is then used for processing, with each head corresponding to an independent weight matrix. The Q, K, and V matrices are partitioned according to the number of heads, with each head processing a dimension of d. model / h subvectors, where d model Let represent the overall dimension of the input, and h represent the number of heads. During attention computation, each head performs a scaled dot product attention operation in parallel: for the (i)th head, its attention score is:

[0083]

[0084] Here, softmax is the activation function.

[0085] Subsequently, the fused feature output is obtained. i , can be represented as:

[0086] output i =score i V i

[0087] The features fused by the encoder are then input into the Transformer decoder, and combined with the preset driving target, continuous decision control commands such as speed and steering angle are generated.

[0088] Step 3: Construct a cloud layer as the central brain of the integrated space-ground system, possessing super computing power and massive data storage capabilities;

[0089] Furthermore, the cloud layer is responsible for managing and maintaining the massive database, providing data support for both vehicle and satellite terminals. Specifically:

[0090] like Figure 2 As shown in the cloud-based distributed file storage architecture diagram, the cloud storage architecture adopts a distributed file system, which has high scalability, high availability and high performance data storage capabilities.

[0091] These distributed file systems distribute data across multiple nodes and employ data redundancy backup strategies to ensure that data can still be retrieved intact when some nodes fail, thus guaranteeing the continuous operation of the system.

[0092] Based on their characteristics and uses, stored data can be categorized into several types: high-precision maps, historical traffic data, real-time traffic information, and various updated traffic regulations.

[0093] The high-precision map data includes detailed information such as road topology, lane line information, and traffic sign locations, and is stored in vector format for easy querying and precise navigation.

[0094] Historical traffic data includes traffic flow, congestion, and accident records for different time periods and road sections, providing a reference for road condition prediction and driving route planning. It is stored in a time-series database to support efficient time range queries.

[0095] Real-time traffic information focuses on the dynamic changes in the road, such as road construction, temporary traffic control, and traffic congestion caused by emergencies. It is obtained through multiple sources, including real-time reporting from vehicles and satellite remote sensing monitoring, and is updated in real time in the form of a message queue to ensure the timeliness of the data.

[0096] Various traffic regulations and driving knowledge documents are stored in text format, making it easy for large language models to retrieve and learn at any time, and update their knowledge reserves.

[0097] When data grows in massive quantities, distributed systems can expand their capacity by adding storage nodes to meet the ever-increasing data storage demands.

[0098] Furthermore, a powerful cloud-based large language model runs in the cloud layer. Through deep mining and learning of massive amounts of data, it continuously optimizes its own knowledge system, providing knowledge distillation for the vehicle-side model and improving the vehicle-side model's decision-making capabilities. Specifically:

[0099] The large language model adopts a GPT-based architecture, and its input is massive amounts of multi-source data.

[0100] On the one hand, it contains a massive amount of textual knowledge, traffic rules, accident cases, and scenario response logic.

[0101] On the other hand, it integrates actual driving data uploaded from the vehicle, including sensor environmental data and decision sequences, so that the model closely matches real-world driving scenarios.

[0102] In terms of model training algorithms, a pre-training-fine-tuning paradigm based on the Transformer architecture is adopted.

[0103] The pre-training phase utilizes a large-scale unsupervised corpus for pre-training, enabling the model to learn general language patterns and semantic understanding capabilities.

[0104] The fine-tuning phase combines specific autonomous driving tasks with real-world scenario data for fine-tuning, enhancing the model's ability to identify risks and optimize decisions.

[0105] Furthermore, the cloud layer undertakes the task of remote monitoring and management of the entire system, collecting driving data, sensor status, and decision execution feedback information uploaded by the vehicle in real time, evaluating and issuing warnings on the vehicle's operating status, following the vehicle's decisions when the driving scenario is safe, and adopting the decision of the large language model when potential risks or abnormal situations are detected, pushing optimization instructions, updated model parameters, or emergency response plans to the vehicle in real time to assist the vehicle in correcting its decisions, and in the event of an extremely complex emergency scenario (such as a traffic accident) where the system cannot make a decision, initiating the final decision of emergency braking and pulling over to the side of the road.

[0106] Step 4: Form a tight closed-loop interactive process by combining the layers built in Steps 1, 2 and 3.

[0107] Furthermore, such as Figure 3 The vehicle decision-making process flowchart shows that the closed-loop interaction and vehicle decision-making process of the system are as follows:

[0108] The satellite layer continuously transmits high-precision positioning information and macroscopic environmental data to the vehicle and the cloud.

[0109] Based on this information, the vehicle collects and processes local sensor data in real time, uses an end-to-end rapid decision-making system to generate preliminary driving decisions, and uploads driving data, environmental perception data, and decision requirements to the cloud in real time.

[0110] The cloud platform combines its own stored data with powerful computing capabilities to comprehensively analyze the information uploaded by the vehicle and assess the vehicle's operating status. When the driving scenario is safe, the vehicle's decision is adopted directly. When the driving scenario is abnormal, the cloud platform uses a large language model to provide knowledge enhancement and model optimization suggestions to the vehicle and feeds back the processing results to the vehicle in real time to help the vehicle correct its decisions.

[0111] The vehicle further optimizes driving operations based on feedback from the cloud and feeds the results back to the cloud, enabling the system to continuously learn and dynamically optimize, thereby ensuring the safe and efficient operation of autonomous vehicles in various complex scenarios.

[0112] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent methods or modifications that do not depart from the technology of the present invention should be included within the scope of protection of the present invention.

Claims

1. A large language model-driven intelligent decision-making method for integrated space-ground autonomous driving, characterized in that, Includes the following steps: Step 1: Construct a satellite layer as a high-altitude information node in the integrated space-ground network system. The satellite layer consists of a low-Earth orbit satellite constellation. The high-precision positioning information transmitted to ground vehicles includes dynamic and static features. Dynamic features include information on vehicle speed, acceleration, and heading angle changes. Real-time monitoring of vehicle speed provides an intuitive reflection of the vehicle's current speed. Acceleration features reflect the dynamic changes in the vehicle's power performance and driving status. Heading angle changes reflect the rate of change in the vehicle's driving direction, allowing for the determination of the vehicle's steering intention and achieving maneuverability. Static features include the vehicle's absolute and relative position information. Combined with high-precision maps, vehicles can accurately know their lane, road segment, relative position to surrounding landmarks and traffic facilities, as well as the spatial distribution of other traffic participants. The positioning information composed of dynamic and static features is interconnected and works synergistically to provide comprehensive and multi-level information support for autonomous driving decisions. Step 2: Construct the vehicle-side layer as the executor of the integrated space-ground network system. The vehicle-side layer is equipped with multiple sensors, including: LiDAR, cameras, and millimeter-wave radar, to collect environmental data around the vehicle in real time from different dimensions; the LiDAR accurately constructs a three-dimensional point cloud map of the vehicle's surroundings. The camera captures visual images for identifying traffic signs, traffic lights, pedestrians, and other vehicle targets. The millimeter-wave radar is used to detect the speed and distance information of target objects; The vehicle-side layer has an end-to-end rapid decision-making system built in. The end-to-end decision-making system performs deep fusion, analysis and reasoning on multi-source sensor data. Combined with high-precision positioning information provided by low-orbit satellites, it quickly generates driving decisions that are adapted to the current complex road conditions, including speed and steering. The end-to-end rapid decision-making system adopts a multimodal fusion Transformer architecture, as detailed below: Input features include image data LiDAR point cloud data Millimeter-wave radar point cloud data and high-precision positioning information ,in and For the length and width of the image, For the number of channels, The number of point clouds; For image data Divide it into multiple small blocks, and treat each small block as a sequence element. It is mapped to a fixed-dimensional feature vector through linear projection, and positional encoding is added to preserve spatial location information; Point cloud features for lidar and millimeter-wave radar and Convert it into a sequence of feature vectors And add corresponding sensor embeddings to distinguish features from different sensor sources; For high-precision positioning information, latitude, longitude, and heading angle are used as the basis for encoding into a position feature sequence. ; The above sequence is then used as input to the multimodal fusion Transformer. In the Transformer encoder, a multi-head self-attention mechanism is used to interactively fuse the feature sequences of different sensors. The multi-head self-attention mechanism can simultaneously focus on different subspaces of the input sequence, capture the complex correlation between features of different sensors, and obtain the fused features. The features fused by the encoder are input into the Transformer decoder, and combined with the preset driving target, continuous decision instructions for speed and steering angle are generated. Step 3: Construct a cloud layer as the central brain of the integrated space-ground network system, possessing super computing power and massive data storage capabilities; the cloud layer remotely monitors and manages the entire decision-making system, collects driving data, perception data, and decision execution feedback information uploaded by the vehicle in real time, evaluates and warns of vehicle operating status, follows the vehicle's rapid decision-making when the vehicle driving scenario is safe, and adopts a decision model based on a large language model when potential risks or abnormal situations are detected, pushes optimization instructions, updated model parameters, or emergency response plans to the vehicle in real time to assist the vehicle in correcting its decisions; Steps 4, 1, 2, and 3 form a tightly integrated closed-loop interaction process; specifically as follows: The satellite layer continuously transmits high-precision positioning information and macroscopic environmental data to the vehicle layer and the cloud layer; Based on this information, the vehicle-side layer collects and processes local sensor data in real time, uses an end-to-end rapid decision-making system to generate preliminary driving decisions, and uploads driving data, environmental perception data and decision requirements to the cloud in real time. The cloud layer combines its own stored data with powerful computing capabilities to comprehensively analyze the information uploaded by the vehicle and evaluate the vehicle's operating status. When the driving scenario is safe, the vehicle's decision is directly adopted. When the driving scenario is abnormal, the cloud-based large language model provides knowledge enhancement and model optimization suggestions to the vehicle and feeds back the processing results to the vehicle in real time to assist the vehicle in correcting its decisions. Based on feedback from the cloud, the vehicle further optimizes driving decisions and feeds the results back to the cloud, enabling continuous learning and dynamic optimization to ensure the safe and efficient operation of autonomous vehicles in various complex scenarios.

2. The large language model-driven intelligent decision-making method for integrated space-ground autonomous driving as described in claim 1, characterized in that, In step 2, the Transformer encoder at the vehicle end layer generates the fused features, and the specific method is as follows: First, the input sequence is linearly transformed to generate a query. ),key( ),value( ) matrix, each head corresponds to an independent weight matrix; by , , The matrix is ​​partitioned by the number of heads, with each head having a processing dimension of [dimension number missing]. For the subvectors, during attention computation, each head performs a scaled dot product attention operation in parallel: for the... Based on height, their attention score is: in, For activation functions; The fused features are then obtained and can be represented as follows: ； The cloud layer storage architecture in step 3 adopts a distributed file system. The distributed file system stores data across multiple nodes. Through a data redundancy backup strategy, it ensures that the data can still be completely retrieved when some nodes fail, thus ensuring the continuous operation of the system. Based on their characteristics and uses, stored data can be categorized as follows: high-precision maps, historical traffic data, real-time traffic information, and various updated traffic regulations. Among them, the high-precision map data covers road topology, lane line information, and traffic sign locations, and is stored in vector form for easy quick query and accurate navigation; Historical traffic data includes traffic flow, congestion, and accident records for different time periods and road sections, providing a reference for road condition prediction and driving route planning. It is stored in a time-series database to support efficient time range queries. Real-time traffic information focuses on the dynamic changes in the road at present, such as road construction, temporary traffic control, and traffic congestion caused by emergencies. It is obtained through real-time reporting from vehicles and satellite remote sensing monitoring from multiple sources, and is updated in real time in the form of message queues to ensure the timeliness of the data. Traffic regulations are stored in text format, which facilitates retrieval and learning by large language models at any time, allowing for updates to the knowledge base. The cloud layer in step 3 uses a large language model to continuously optimize its knowledge system through deep mining and learning of massive amounts of data, providing knowledge distillation for the vehicle-side decision-making model and improving its decision-making capabilities, as detailed below: The large language model adopts a GPT-based architecture, and its input is massive multi-source data. On the one hand, it includes massive text knowledge, traffic rules, accident cases, and scenario response logic. On the other hand, it integrates actual driving data uploaded by the vehicle, including sensor environmental data and decision sequences, so that the model closely fits the real driving situation. For training large language models, a pre-training-fine-tuning paradigm based on the Transformer architecture is adopted. In the pre-training stage, a large-scale unsupervised corpus is used for pre-training, allowing the model to learn general language patterns and semantic understanding capabilities. In the fine-tuning stage, specific autonomous driving tasks are combined with real-world scenario data for fine-tuning, enhancing the model's ability to identify risks and optimize decisions.

3. A space-ground integrated autonomous driving intelligent decision-making system driven by a large language model based on the autonomous driving intelligent decision-making method of claim 1, characterized in that, Including the satellite layer, vehicle layer, and cloud layer; The satellite layer, as a high-altitude information node of the integrated space-ground network, is composed of a low-orbit satellite constellation and transmits high-precision positioning information to ground vehicles in real time, providing a spatiotemporal reference for the vehicles. The vehicle-side layer is the direct executor of the decision-making system. It is equipped with a variety of sensors to collect environmental data around the vehicle in real time. At the same time, it uses high-precision positioning data provided by low-orbit satellites and the end-to-end fast decision-making model built into the vehicle-side layer to generate preliminary driving decisions. It also uploads driving data, environmental perception data and decision requirements to the cloud layer. The cloud layer is responsible for remote monitoring and management of the entire decision-making system. It collects driving data, perception data, and decision execution feedback information uploaded by the vehicle in real time, evaluates and warns about the vehicle's operating status, and follows the vehicle's rapid decision-making when the driving scenario is safe. When potential risks or abnormal situations are detected, it adopts a decision model based on a large language model and pushes optimization instructions, updated model parameters, or emergency response plans to the vehicle in real time to assist the vehicle in correcting its decisions.

4. The large language model-driven integrated space-ground autonomous driving intelligent decision-making system according to claim 3, characterized in that, The high-precision positioning information transmitted from the satellite layer to the ground vehicle includes dynamic and static features. Dynamic features include information on vehicle speed, acceleration, and heading angle changes. Real-time monitoring of vehicle speed provides an intuitive reflection of the vehicle's current speed. Acceleration features reflect the dynamic changes in the vehicle's power performance and driving status. Heading angle changes reflect the rate of change in the vehicle's driving direction, allowing for the determination of the vehicle's steering intention and achieving maneuverability. Static features include the vehicle's absolute and relative position information. Combined with high-precision maps, vehicles can accurately know their lane, road segment, relative position to surrounding landmarks and traffic facilities, as well as the spatial distribution of other traffic participants. The positioning information composed of dynamic and static features is interconnected and works synergistically to provide comprehensive and multi-level information support for autonomous driving decisions.

5. The large language model-driven integrated space-ground autonomous driving intelligent decision-making system according to claim 3, characterized in that, The vehicle-mounted layer is equipped with a variety of sensors, including lidar, cameras, and millimeter-wave radar, to collect environmental data around the vehicle in real time from different dimensions. The lidar accurately constructs a three-dimensional point cloud map of the vehicle's surroundings; The camera captures visual images for identifying traffic signs, traffic lights, pedestrians, and other vehicle targets. The millimeter-wave radar is used to detect the speed and distance information of target objects.

6. The large language model-driven integrated space-ground autonomous driving intelligent decision-making system according to claim 3, characterized in that, The vehicle-side layer has an end-to-end rapid decision-making system built in. It uses an end-to-end model to deeply fuse, analyze and reason about multi-source sensor data, and combines it with high-precision positioning information provided by low-orbit satellites to quickly generate driving decisions that are adapted to the current complex road conditions, including speed and steering.

7. The large language model-driven integrated space-ground autonomous driving intelligent decision-making system according to claim 6, characterized in that, The end-to-end rapid decision-making system adopts a multimodal fusion Transformer architecture, as detailed below: Input features include image data LiDAR point cloud data Millimeter-wave radar point cloud data and high-precision positioning information ,in and For the length and width of the image, For the number of channels, The number of point clouds; For image data Divide it into multiple small blocks, and treat each small block as a sequence element. It is mapped to a fixed-dimensional feature vector through linear projection, and positional encoding is added to preserve spatial location information; Point cloud features for lidar and millimeter-wave radar and Convert it into a sequence of feature vectors And add corresponding sensor embeddings to distinguish features from different sensor sources; For high-precision positioning information, latitude, longitude, and heading angle are used as the basis for encoding into a position feature sequence. ; The above sequence is then used as input to the multimodal fusion Transformer. In the Transformer encoder, a multi-head self-attention mechanism is used to interactively fuse the feature sequences of different sensors. The multi-head self-attention mechanism can simultaneously focus on different subspaces of the input sequence, capture the complex correlation between features of different sensors, and obtain the fused features. The features fused by the encoder are input into the Transformer decoder, and combined with the preset driving target, continuous decision instructions for speed and steering angle are generated.

8. The large language model-driven integrated space-ground autonomous driving intelligent decision-making system according to claim 7, characterized in that, The Transformer encoder generates the fused features as follows: First, the input sequence is linearly transformed to generate a query. ),key( ),value( ) matrix, each head corresponds to an independent weight matrix; by , , The matrix is ​​partitioned by the number of heads, with each head having a processing dimension of [dimension number missing]. For the subvectors, during attention computation, each head performs a scaled dot product attention operation in parallel: for the... Based on height, their attention score is: in, For activation functions; The fused features are then obtained and can be represented as follows: 。 9. The large language model-driven integrated space-ground autonomous driving intelligent decision-making system according to claim 3, characterized in that, The cloud layer storage architecture adopts a distributed file system, which distributes data across multiple nodes. Through a data redundancy backup strategy, it ensures that the data can still be retrieved completely when some nodes fail, thus guaranteeing the continuous operation of the system. Based on their characteristics and uses, stored data can be categorized as follows: high-precision maps, historical traffic data, real-time traffic information, and various updated traffic regulations. Among them, the high-precision map data covers road topology, lane line information, and traffic sign locations, and is stored in vector form for easy quick query and accurate navigation; Historical traffic data includes traffic flow, congestion, and accident records for different time periods and road sections, providing a reference for road condition prediction and driving route planning. It is stored in a time-series database to support efficient time range queries. Real-time traffic information focuses on the dynamic changes in the road at present, such as road construction, temporary traffic control, and traffic congestion caused by emergencies. It is obtained through real-time reporting from vehicles and satellite remote sensing monitoring from multiple sources, and is updated in real time in the form of message queues to ensure the timeliness of the data. Traffic regulations are stored in text format, which facilitates retrieval and learning by large language models at any time, allowing for the updating of knowledge reserves.

10. The large language model-driven integrated space-ground autonomous driving intelligent decision-making system according to claim 3, characterized in that, The cloud layer contains a large language model. Through deep mining and learning of massive amounts of data, it continuously optimizes its own knowledge system, providing knowledge distillation for the vehicle-side decision-making model and improving the decision-making ability of the vehicle-side decision-making model, as detailed below: The large language model adopts a GPT-based architecture, and its input is massive multi-source data. On the one hand, it includes massive text knowledge, traffic rules, accident cases, and scenario response logic. On the other hand, it integrates actual driving data uploaded by the vehicle, including sensor environmental data and decision sequences, so that the model closely fits the real driving situation. For training large language models, a pre-training-fine-tuning paradigm based on the Transformer architecture is adopted; in the pre-training stage, a large-scale unsupervised corpus is used for pre-training, allowing the model to learn general language patterns and semantic understanding capabilities. The fine-tuning phase combines specific autonomous driving tasks with real-world scenario data for fine-tuning, enhancing the model's ability to identify risks and optimize decisions.

Images