Global information-driven automatic driving control method and system

Abstract

The invention provides a global information-driven automatic driving control method and system, which can make up the sight distance limit of a ground sensor by using the aerial view angle of an unmanned aerial vehicle through combining the ground sensing information and the aerial sensing information of the unmanned aerial vehicle. By aligning unmanned aerial vehicle instance query and ground instance query, the blind area problem of traditional ground visual perception is solved. Moreover, by combining the natural language navigation instruction with the visual information, the decision process of automatic driving is optimized, and the navigation and planning capabilities in a complex scene are improved. Besides, the BEV features and the language features are fused, the navigation decision is made through an optimization strategy, an accurate control instruction can be generated according to environment information and a navigation target when a complex driving task is processed, and the decision making efficiency and reliability of the system are further improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and more specifically, to a globally information-driven autonomous driving control method and system. Background Technology

[0002] With the rapid development of autonomous driving technology, more and more systems rely on multi-sensor fusion and deep learning models to improve perception accuracy and decision-making capabilities. Current autonomous driving systems mainly rely on onboard sensors (such as cameras and LiDAR) for real-time perception to understand the surrounding environment. However, these systems can usually only process information within a limited visual range. In complex environments, especially when remote targets and the environment cannot be perceived visually, the system's performance and safety are limited.

[0003] In recent years, research on air-ground cooperative perception using drones has gradually emerged, aiming to overcome the limitations of traditional autonomous driving systems in areas such as long-range detection and obstruction recognition. Air-ground cooperative autonomous driving systems enhance the global field of vision and intelligent decision-making capabilities of the perception system by integrating multi-view information from the ground and the air. However, these systems still face challenges such as how to efficiently fuse data from different perspectives and sensors, and how to process global navigation information.

[0004] In summary, multi-sensor fusion is one of the key technologies for improving perception capabilities and intelligent decision-making in the field of autonomous driving. Existing related technologies mainly focus on the following aspects: 1. Traditional sensor fusion technology: This generally uses data from ground cameras, radar, lidar, and other sensors for fusion, primarily addressing local environmental perception problems, but its ability to identify distant targets and occluded objects is limited. 2. Vision-language models: These are used to understand natural language commands and combine visual information for decision-making, but these models are still limited to scenes within the line of sight and struggle with global planning. 3. Air-ground cooperative perception: Some emerging research has proposed air-ground cooperative perception methods, which combine UAV perspectives and vehicle-mounted perception information to improve the system's perception range and solve the problem of blind spots in the field of view of traditional ground sensors. However, how to efficiently process information from different perspectives and achieve efficient feature alignment and matching remains a technical challenge. Summary of the Invention

[0005] The purpose of this invention is to provide a globally information-driven autonomous driving control method and system to improve navigation accuracy, decision-making efficiency, and reliability.

[0006] In a first aspect, the present invention provides a globally information-driven autonomous driving control method, the method comprising: The original images captured by the vehicle-mounted camera and the drone are acquired, and the original images are subjected to multi-task restoration processing to obtain the restored images. The restored image is processed using an encoder to generate a BEV feature map, and an instance query is generated based on the BEV feature map. The instance query includes ground instance query and UAV instance query. The UAV instance query and the ground instance query are aligned, the aligned UAV instance query and the ground instance query are matched, and the successfully matched UAV instance query and the ground instance query are fused to obtain the fused perception result. The fused perception results and the restored image are used as visual inputs, and the generated natural language navigation instructions and questions are used as text inputs. The visual language model is used to output decision suggestions that are consistent with the navigation target. The decision suggestions, fused perception results, and historical control signals are spliced ​​together to obtain a global multimodal representation; The global multimodal representation is fused with the BEV feature map to generate input features. Based on the constraints of the input features and the natural language navigation instruction representation, a planning model is used to output control actions for autonomous driving.

[0007] In an optional implementation, the step of performing multi-task restoration processing on the original image to obtain the restored image includes: The original image is encoded into a low-dimensional latent representation; Based on the low-dimensional latent representation and the pre-trained latent diffusion model, multi-task restoration processing is performed, outputting the noise prediction results corresponding to each task, and weighted and fused multiple noise prediction results. Denoising is performed on the weighted fusion noise prediction results to generate a potentially clear representation; The restored image is obtained by decoding the potential sharp representation.

[0008] In an optional implementation, the step of processing the restored image using an encoder to generate a BEV feature map, and generating an instance query based on the BEV feature map, includes: The restored image is input into an encoder with shared parameters to generate a ground BEV feature map and a UAV BEV feature map; A ground instance query is generated from the ground BEV feature map, and a drone instance query is generated from the drone BEV feature map. The ground instance query and the UAV instance query include semantic features and 3D reference points.

[0009] In an optional implementation, the step of aligning the UAV instance query with the ground instance query includes: A rigid transformation matrix is ​​constructed using the position and attitude angle data of the UAV. The rigid transformation matrix is ​​then used to project the 3D reference points in the UAV instance query onto the vehicle coordinate system to achieve spatial alignment. The semantic features in the UAV instance query are concatenated with the rotation matrix, and the concatenation result is refined using a perceptron to achieve feature alignment.

[0010] In an optional implementation, the step of matching the aligned UAV instance query with the ground instance query includes: The spatial distance and feature similarity of each aligned UAV instance query and each ground instance query are calculated. The matching score is obtained by weighted fusion of spatial distance and feature similarity; Based on the matching scores between the UAV instance query and each ground instance query, the ground instance query that matches the UAV instance query is determined.

[0011] In an optional implementation, the step of using the fused perception result and the restored image as visual input, and the generated natural language navigation instructions and questions as text input, and using a visual language model to output decision suggestions consistent with the navigation target, includes: Visual input is obtained by feature stitching of the fused perception result and the restored image, and text input is obtained by text stitching of the generated natural language navigation instructions and questions; Based on the visual and text inputs, an initial decision suggestion is generated using a visual language model, and an online summary information is generated from the initial decision suggestion. The visual language model is fine-tuned based on the summary information; Using the fine-tuned visual language model, along with the visual and text inputs, the system outputs decision recommendations consistent with the navigation target.

[0012] In an optional implementation, the step of fine-tuning the visual language model based on the summary information includes: A mutual information function is constructed based on the summary information and natural language navigation instructions, and a summary reward item and an initial decision reward item are defined; The visual language model is fine-tuned based on the mutual information function, the summary reward term, and the initial decision reward term.

[0013] In an optional implementation, the step of concatenating the decision recommendations, fused perception results, and historical control signals to obtain a global multimodal representation includes: Extract the high-dimensional language features of the decision suggestion, and compress the high-dimensional language features into low-dimensional language features that are compatible with the fusion perception result; The weights of each source feature are learned by a multilayer perceptron, and the dimensionality-reduced language features, fused perception results, and historical control information are concatenated according to the learned weights to generate a global multimodal representation.

[0014] In an optional implementation, the step of outputting control actions for autonomous driving using a planning model under the constraints of the input features and the natural language navigation instruction representation includes: The natural language navigation instructions are parsed into hard constraint terms and soft reward terms; The input features are input into the planning model, candidate actions are filtered based on the hard constraints, and the policy network is optimized based on the soft reward terms to output the control action for autonomous driving that meets the hard constraints and has the highest total reward.

[0015] Secondly, the present invention provides a globally information-driven autonomous driving control system, the system comprising: The restoration processing module is used to acquire the original images captured by the vehicle-mounted camera and the drone, and to perform multi-task restoration processing on the original images to obtain the restored images. The generation module is used to process the restored image using an encoder to generate a BEV feature map, and to generate an instance query based on the BEV feature map. The instance query includes a ground instance query and a UAV instance query. The alignment and matching module is used to align the UAV instance query with the ground instance query, match the aligned UAV instance query with the ground instance query, and fuse the successfully matched UAV instance query and ground instance query to obtain the fused perception result. The model processing module is used to take the fused perception result and the restored image as visual input, and the generated natural language navigation instructions and questions as text input, and use the visual language model to output decision suggestions consistent with the navigation target. The splicing module is used to splice the decision suggestions, fused perception results and historical control signals to obtain a global multimodal representation; The control action generation module is used to fuse the global multimodal representation with the BEV feature map to generate input features, and under the constraints of the input features and the natural language navigation instruction representation, use a planning model to output control actions for autonomous driving.

[0016] This invention provides a globally information-driven autonomous driving control method and system. It combines ground and UAV aerial perception information, utilizing the UAV's aerial perspective to compensate for the line-of-sight limitations of ground sensors. By aligning UAV instance queries with ground instance queries, it solves the blind spot problem of traditional ground visual perception. Furthermore, by combining natural language navigation commands with visual information, it optimizes the autonomous driving decision-making process, improving navigation and planning capabilities in complex scenarios. Moreover, by fusing BEV features with language features and optimizing navigation decisions, it can generate precise control commands based on environmental information and navigation targets when handling complex driving tasks, further improving the system's decision-making efficiency and reliability. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a globally information-driven autonomous driving control method provided in an embodiment of the present invention; Figure 2 for Figure 1 A flowchart of the sub-steps included in S11; Figure 3 for Figure 1 A flowchart of the sub-steps included in S12; Figure 4 for Figure 1 A flowchart of the sub-steps included in S13; Figure 5 for Figure 1 Another flowchart of the sub-steps included in S13; Figure 6 for Figure 1 A flowchart of the sub-steps included in S14; Figure 7 for Figure 1 A flowchart of the sub-steps included in S15; Figure 8 for Figure 1 A flowchart of the sub-steps included in S16; Figure 9 A functional block diagram of a globally information-driven autonomous driving control system provided in an embodiment of the present invention; Figure 10 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0020] Please see Figure 1 The following is a flowchart of a global information-driven autonomous driving control method provided in an embodiment of the present invention. The detailed steps of the global information-driven autonomous driving control method are described below.

[0021] S11: Acquire the original images captured by the vehicle-mounted camera and the drone, perform multi-task restoration processing on the original images, and obtain the restored images; S12, use the encoder to process the restored image to generate BEV feature map, and generate instance query based on BEV feature map. Instance query includes ground instance query and UAV instance query. S13, Align the UAV instance query with the ground instance query, Match the aligned UAV instance query with the ground instance query, and fuse the successfully matched UAV instance query and ground instance query to obtain the fused perception result; S14, take the fused perception results and restored image as visual input, and the generated natural language navigation instructions and questions as text input, and use the visual language model to output decision suggestions consistent with the navigation target; S15, the decision suggestions, fused perception results and historical control signals are spliced ​​together to obtain a global multimodal representation; S16, the global multimodal representation is fused with the BEV feature map to generate input features. Based on the constraints of the input features and the natural language navigation command representation, the planning model is used to output control actions for autonomous driving.

[0022] The global information-driven autonomous driving control method provided in this embodiment improves the perception capability, decision-making accuracy, and navigation efficiency of the autonomous driving control system in complex environments by integrating aerial and ground collaborative perception, multimodal information, and reinforcement learning planning.

[0023] The specific implementation methods of each of the above steps will be explained in detail below.

[0024] In this embodiment, after obtaining the original images captured by the vehicle-mounted camera and the drone respectively, the original images are processed to output a high-quality, non-degradable restored image.

[0025] Please see Figure 2 The steps of performing multi-task restoration processing on the original image to obtain the restored image can be implemented in the following way: S111, encodes the original image into a low-dimensional latent representation; S112 performs multi-task restoration based on low-dimensional latent representation and pre-trained latent diffusion model, outputs noise prediction results for each task, and weights and fuses multiple noise prediction results. S113, Denoise the noise prediction results after weighted fusion to generate a potentially clear representation; S114, decode the potential sharp representation to obtain the restored image.

[0026] In this embodiment, the raw images captured by the vehicle-mounted camera and the drone are denoted as x. These raw images may be affected by various factors such as blurring and noise, resulting in low image quality. The raw images can first be encoded into a low-dimensional latent representation. This reduces computational overhead while preserving key visual features.

[0027] Using a pre-trained latent diffusion model as a foundation, multiple image restoration tasks (super-resolution, motion deblurring, defocusing deblurring, denoising, etc.) are trained simultaneously through multi-task learning. Each task is conditionally controlled by different text prompts to guide the latent diffusion model in generating the corresponding restoration result. The noise prediction function of the latent diffusion model is:

[0028] in, for A potential representation with noise at any given moment; This indicates a splicing operation, which transforms the low-dimensional latent representation... With noisy latent representation Stitching ensures that the potential diffusion model predicts noise based on the input degraded image; The text prompts corresponding to the task; It is the noise vector predicted by UNet (a convolutional neural network), and the actual noise. ( Alignment. The training objective of this latent diffusion model is to minimize the following loss function:

[0029] After training the latent diffusion model using the above method, the latent diffusion model is used to process the low-quality original image.

[0030] For original images with mixed degradation levels, multi-task noise prediction results are weighted and fused, adaptively invoking the restoration capabilities of different tasks without requiring manual specification of degradation types. Assume there are... There are several restoration tasks, each corresponding to a noise prediction function. The weighted fusion formula is shown below:

[0031] This represents the noise prediction result after weighted fusion. Represent the weight of each task, satisfying =1, the optimal weight combination can be found through grid search.

[0032] To control the level of detail generated, a "downsampling-upsampling" process is performed on the original input image to generate a low-detail latent representation. To balance the weights of restoration quality and detail fidelity, the model is designed to avoid generating excessive false details (fidelity) or insufficient details (quality). Therefore, the final weighted fusion noise prediction result is as follows:

[0033] in, Control the intensity of detail generation; the higher the intensity, the richer the detail.

[0034] Finally, the DDIM sampling algorithm is used to perform progressive denoising on the weighted fused noise prediction results to generate a potentially clear representation. The final decoded image is a high-quality, non-degradable restored image. :

[0035]

[0036] After obtaining high-quality restored images, BEV (Bird's-Eye View) feature maps are generated based on the restored images. Ground instance queries and UAV instance queries are then generated based on the BEV feature maps. For details, please refer to [link to relevant documentation]. Figure 3 This step can be achieved in the following way: S121, Input the restored image into the encoder with shared parameters to generate ground BEV feature map and UAV BEV feature map; S122, generate ground instance queries from the ground BEV feature map and drone instance queries from the drone BEV feature map respectively; the ground instance query and drone instance query include semantic features and 3D reference points.

[0037] In this embodiment, the generated high-quality restored image is input into a BEV encoder with shared parameters to generate a BEV feature map. This ensures that both reside in the same feature space.

[0038] From BEV feature map The system dynamically generates instance queries, with the number of queries and initial location varying based on BEV features; more targets result in more queries. Each query instance contains semantic features (appearance, category, and other semantic information) and a 3D reference point (spatial coordinates). This enables target-level feature representation.

[0039] For ground vehicles, the generated ground instance query representation is as follows:

[0040] For drones, the generated drone instance query representation is as follows:

[0041] Based on the above, the UAV instance query and ground instance query will be aligned. For details, please refer to [link to relevant documentation]. Figure 4 This can be achieved in the following ways: S131, a rigid transformation matrix is ​​constructed using the position and attitude angle data of the UAV, and the 3D reference points in the UAV instance query are projected into the vehicle coordinate system using the rigid transformation matrix to achieve spatial alignment; S132, the semantic features in the drone instance query are concatenated with the rotation matrix, and the concatenation result is refined using a perceptron to achieve feature alignment.

[0042] In this embodiment, spatial alignment is first performed, and a rigid transformation matrix is ​​constructed using the position and attitude angle data of the UAV. This involves projecting the 3D reference points of drone instance queries onto the vehicle coordinate system to achieve spatial uniformity. Each drone instance query from the drone's perspective contains an explicit 3D reference point (usually the target center point), denoted as... To align with the ground vehicle coordinate system, this point needs to be projected into the vehicle coordinate system:

[0043] in, It is a 3D reference point in the UAV coordinate system. , It is a 3D reference point in the projected vehicle coordinate system, which is consistent with the spatial coordinate dimension of the ground instance query.

[0044] Then, feature alignment is performed, refining the semantic features of drone instance queries through a lightweight three-layer perceptron, and incorporating rotation matrix information to compensate for feature differences caused by viewpoint rotation: ) in, These are semantic features from the perspective of drones. These are semantic features after feature alignment. It is a rotation matrix (used for implicit alignment). This indicates a splicing operation. It is a three-layer perceptron, with the output dimension being the same as the input dimension.

[0045] After performing spatial alignment and feature alignment, the aligned UAV instance queries are matched with ground instance queries. Please refer to [link to relevant documentation]. Figure 5 This step can be achieved in the following way: S133, calculate the spatial distance and feature similarity between the aligned UAV instance queries and the ground instance queries; S133, the matching score is obtained by weighted fusion of spatial distance and feature similarity; S135, Based on the matching scores between the UAV instance query and each ground instance query, determine the ground instance query that matches the UAV instance query.

[0046] In this embodiment, cross-view matching and fusion are performed on UAV instance queries and ground instance queries. The aligned UAV instance queries need to be matched with candidate queries from the perspective of ground vehicles (ground instance queries), based on spatial distance. (Euclidean distance) and feature similarity (Cosine similarity) is used to calculate the matching score. The final matching score is a weighted sum of spatial distance and feature similarity, represented as follows:

[0047]

[0048]

[0049] in, This represents the 3D reference point in the drone instance query. This represents the 3D reference point in the ground instance query. Indicates spatial distance. This represents the semantic features in drone instance queries. This represents the semantic features in ground instance queries. This represents the weighting coefficient.

[0050] The Hungarian algorithm is used for optimal matching. Successfully matched query pairs are then subjected to feature fusion through another three-layer perceptron to obtain the fused perception result. Unmatched confidence level Queries in version 0.7 are retained.

[0051] ) In this embodiment, the fused perception results and high-quality restored images are used as visual input, and natural language navigation instructions and questions are used as text input. A visual language model is used to output decision suggestions consistent with the navigation objective. For details, please refer to... Figure 6 This step can be achieved in the following way: S141, the visual input is obtained by feature stitching of the fused perception results and the restored image, and the text input is obtained by text stitching of the generated natural language navigation instructions and questions; S142, using a visual language model to generate initial decision suggestions based on visual and text input, and then generating summary information online from the initial decision suggestions; S143, Fine-tuning of the visual language model based on summary information; S144 utilizes a finely tuned visual language model, along with visual and text inputs, to output decision recommendations consistent with the navigation objective.

[0052] In this embodiment, multimodal input preprocessing is first performed, and the obtained fused perception result is visually feature-stitched with the high-quality restored image to obtain the visual input. Natural language navigation instructions As a global context, and related to specific issues Perform text concatenation to obtain the text input. .

[0053] The preprocessed multimodal inputs (including visual and text inputs) are fed into a navigation-guided supervised fine-tuning visual-language model to generate initial decision suggestions. This ensures that the visual language model possesses the basic ability to correlate driving scenarios with navigation commands. Next, initial decision suggestions are made. Create an online summary and generate a concise summary. This reduces redundant information while retaining key navigation-related content.

[0054]

[0055] in, These are summary prompts; VLM stands for Visual Language Model.

[0056] Fine-tuning of the visual language model based on summary information is specifically achieved through the following methods: A mutual information function is constructed based on summary information and natural language navigation instructions, and a summary reward term and an initial decision reward term are defined; the visual language model is then fine-tuned based on the mutual information function, the summary reward term, and the initial decision reward term.

[0057] Mutual information functions are defined based on summary information and natural language navigation instructions. Simultaneously optimize two objectives: summary Concise and compatible with navigation commands Highly relevant and aligned with the overall navigation intent:

[0058] in, This is a summary S The generation probability, It is a navigation command. g Given a summary S The conditional probability. The smaller the value, the more concise and relevant the summary is to navigation.

[0059] To quantify the output quality of the visual language model, two reward terms are defined: a summary reward term. (Taking into account both the performance and mutual information of the visual language model), initial decision reward item (Measuring the basic quality of decision recommendations), characterized as follows:

[0060]

[0061] in, It is a visual language model that needs to be optimized after navigation-guided supervision and fine-tuning. It is a pre-trained, un-fine-tuned visual language model. These correspond to the summary information generated by the two models, To balance the performance of the visual language model with the weight of mutual information.

[0062] Abstract reward items Superior to the initial decision reward item To achieve the desired outcome, the visual language model is fine-tuned using the NPO objective function to generate decision suggestions that better align with navigation intent. The optimization objective is to minimize... That is to let Maximize the probability:

[0063] in, It is the sigmoid function, which... Map to [0,1] to ensure the loss is within a reasonable range; It is a preference dataset containing sample pairs of "visual features + cue words + initial decision + summary to be optimized + reference summary".

[0064] The optimized visual-language model takes multimodal input (visual and text input) as input and outputs decision suggestions consistent with the navigation objective. .

[0065] Then, the decision recommendations, fused perception results, and historical control signals are concatenated to obtain a global multimodal representation. Please refer to [link / reference]. Figure 7 This step can be achieved in the following way: S151, extract high-dimensional language features for decision-making suggestions, and compress the high-dimensional language features into low-dimensional language features that are compatible with the fusion perception results; S152 learns the weights of each source feature through a multilayer perceptron, and then concatenates the dimensionality-reduced language features, fused perception results, and historical control information according to the learned weights to generate a global multimodal representation.

[0066] In this embodiment, the generated decision suggestions are first extracted. Corresponding high-dimensional language features High-dimensional language features are extracted using a lightweight sparse dimensionality reduction multilayer perceptron. Compressed to a low dimension compatible with the features of the BEV fusion perception results, denoted as dimensionality-reduced language features. At the same time, it retains key navigation semantics (such as actions and distances) to avoid information loss.

[0067] The reduced-dimensional language features, fused perception results, and historical control information are concatenated. During the concatenation process, the weights of each source feature are learned through a feature fusion multilayer perceptron. The multi-source features are concatenated according to the weights, and the output is a unified global multimodal representation with the same dimension as the BEV features, ensuring that it is compatible with the input format of the subsequent planning model.

[0068] Building upon this, the global multimodal representation is fused with the BEV feature map to generate input features. Under the constraints of these input features and natural language navigation command representations, a planning model is used to output control actions for autonomous driving. For details, please refer to [link to relevant documentation]. Figure 8 This step can be achieved in the following way: S161, parses natural language navigation instructions into hard constraint terms and soft reward terms; S162 inputs the input features into the planning model, filters candidate actions based on hard constraints, optimizes the policy network based on soft reward terms, and outputs the control action that meets the hard constraints and has the highest total reward for autonomous driving.

[0069] In this embodiment, the global multimodal representation is first fused with the aforementioned BEV feature map to generate input features that adapt to the planning model, consisting of "local geometry + global semantics". .

[0070] Natural language navigation instructions The constraints are categorized into two types: hard constraints, which are rules that must be followed; and soft rewards, which are objectives that guide the optimization of the strategy.

[0071] Hard constraints are candidate actions output by the planning model. ( Steering wheel angle Throttle opening, (Brake opening), define constraint functions:

[0072] Soft rewards include navigation rewards. Safety Rewards Comfort Rewards The formula is:

[0073] Among them, the weight settings satisfy + .

[0074] Employing a near-end strategy optimization algorithm, to As input, after filtering candidate actions by hard constraints, soft rewards are applied. Optimize the policy network to generate actions that offer "high rewards and align with navigation". The optimized policy network will then input... The output controls that satisfy the hard constraints and have the highest total reward.

[0075] In summary, the global information-driven autonomous driving control method provided in this embodiment improves the perception range and accuracy of driving control. Specifically, through air-ground cooperative perception, the vehicle can acquire remote images from the perspective of the UAV, making up for the line-of-sight limitations of traditional ground sensors and improving the system's global perception capability, especially in complex environments where potential obstacles can be detected earlier.

[0076] Furthermore, this solution enhances decision-making capabilities and safety. Specifically, by integrating global navigation information and a visual language model, the system can make decisions based on richer context, avoiding the decision-making errors inherent in traditional autonomous driving systems that rely solely on local visual information. For example, the system can predict and plan routes in advance, preventing judgment errors at intersections or complex road sections.

[0077] Furthermore, this scheme improves the efficiency of multimodal information fusion. Specifically, by effectively fusing BEV features, linguistic features, and historical control information, it significantly enhances the efficiency of information fusion and the accuracy of decision-making. This method can flexibly adapt to different types of environments and tasks, reduce computational burden, and improve real-time decision-making capabilities.

[0078] Through these innovations, this invention provides a more intelligent and holistic solution in the field of autonomous driving, improving the adaptability, efficiency, and safety of autonomous driving systems in complex environments.

[0079] Based on the same inventive concept, please refer to Figure 9This invention also provides a functional module diagram of a globally information-driven autonomous driving control system. This embodiment can divide the globally information-driven autonomous driving control system into functional modules based on the above method embodiments. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0080] For example, when dividing functional modules according to their respective functions, Figure 9 The illustrated global information-driven autonomous driving control system is merely a system diagram. This system may include a restoration processing module, a generation module, an alignment and matching module, a model processing module, a stitching module, and a control action generation module. The functions of each module of this global information-driven autonomous driving control system will be described in detail below.

[0081] The restoration processing module is used to acquire the original images captured by the vehicle-mounted camera and the drone, perform multi-task restoration processing on the original images, and obtain the restored images. The generation module is used to process the restored image using the encoder to generate BEV feature maps, and to generate instance queries based on the BEV feature maps. Instance queries include ground instance queries and UAV instance queries. The alignment and matching module is used to align UAV instance queries with ground instance queries, match the aligned UAV instance queries with ground instance queries, and fuse the successfully matched UAV instance queries and ground instance queries to obtain the fused perception result. The model processing module takes the fused perception results and the restored image as visual input, and the generated natural language navigation instructions and questions as text input, and uses the visual language model to output decision suggestions consistent with the navigation target. The splicing module is used to splice together decision suggestions, fused perception results, and historical control signals to obtain a global multimodal representation; The control action generation module is used to fuse the global multimodal representation with the BEV feature map to generate input features. Based on the constraints of the input features and the natural language navigation command representation, the module uses a planning model to output control actions for autonomous driving.

[0082] The global information-driven autonomous driving control system provided in this embodiment can be used to execute the global information-driven autonomous driving control method under any of the above embodiments. For details not covered in this embodiment, please refer to the corresponding descriptions in the above embodiments. This embodiment will not elaborate further here.

[0083] Please see Figure 10 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device can be an in-vehicle controller or a cloud server. The electronic device includes a memory, a processor, and a communication module. The memory, processor, and communication module are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0084] The memory is used to store computer programs or data. Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0085] The processor is used to read / write data or programs stored in the memory and execute the global information-driven autonomous driving control method provided in any embodiment of the present invention.

[0086] The communication module is used to establish communication connections between electronic devices and other communication terminals via a network, and to send and receive data via the network.

[0087] It should be understood that, Figure 10 The structure shown is only a schematic diagram of an electronic device; the electronic device may also include components that are larger than those shown. Figure 10 The more or fewer components shown, or having the same Figure 10 The different configurations shown.

[0088] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing machine-executable instructions, which, when executed, implement the global information-driven autonomous driving control method provided in the above embodiments.

[0089] Specifically, the computer-readable storage medium can be a general-purpose storage medium, such as a removable disk or hard disk. When the computer program on the computer-readable storage medium is executed, it can perform the aforementioned globally information-driven autonomous driving control method. The processes involved in the execution of the executable instructions on the computer-readable storage medium can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0090] In summary, the globally information-driven autonomous driving control method and system provided in this invention integrates visual perception, language commands, and global navigation information, and has the following innovative features: 1. Multi-view perception and air-ground collaboration: By combining ground and air perception information, the aerial perspective of the UAV is used to compensate for the line-of-sight limitations of ground sensors. This method introduces a cross-view feature fusion mechanism into the perception system and uses dynamic decoding instance queries for spatial alignment, solving the blind spot problem of traditional ground-view perception systems. 2. Visual and language fusion: By using natural language navigation commands as a global context and combining them with visual information, the decision-making process of the autonomous driving system is optimized. This method uses a mutual information reward mechanism to optimize the coordination between language and vision, effectively improving the system's navigation and planning capabilities in complex scenarios. 3. Multimodal information fusion and reinforcement learning planning: An innovative method is proposed that integrates BEV features with language features and optimizes navigation decisions through reinforcement learning. This method can generate precise control commands based on environmental information and navigation targets when handling complex driving tasks, further improving the system's decision-making efficiency and reliability.

[0091] By integrating air-ground collaborative perception, multimodal information, and reinforcement learning planning, the perception capability, decision-making accuracy, and navigation efficiency of autonomous driving systems in complex environments are improved, providing a sound theoretical basis for remote perception and decision-making of autonomous vehicles.

[0092] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0093] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] Furthermore, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0095] It should be noted that if the functionality is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] The above are merely embodiments of the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A globally information-driven autonomous driving control method, characterized in that, The method includes: The original images captured by the vehicle-mounted camera and the drone are acquired, and the original images are subjected to multi-task restoration processing to obtain the restored images. The restored image is processed using an encoder to generate a BEV feature map, and an instance query is generated based on the BEV feature map. The instance query includes ground instance query and UAV instance query. The UAV instance query and the ground instance query are aligned, the aligned UAV instance query and the ground instance query are matched, and the successfully matched UAV instance query and the ground instance query are fused to obtain the fused perception result. The fused perception results and the restored image are used as visual inputs, and the generated natural language navigation instructions and questions are used as text inputs. The visual language model is used to output decision suggestions that are consistent with the navigation target. The decision suggestions, fused perception results, and historical control signals are spliced ​​together to obtain a global multimodal representation; The global multimodal representation is fused with the BEV feature map to generate input features. Based on the constraints of the input features and the natural language navigation instruction representation, a planning model is used to output control actions for autonomous driving.

2. The global information-driven autonomous driving control method according to claim 1, characterized in that, The step of performing multi-task restoration processing on the original image to obtain the restored image includes: The original image is encoded into a low-dimensional latent representation; Based on the low-dimensional latent representation and the pre-trained latent diffusion model, multi-task restoration processing is performed, outputting the noise prediction results corresponding to each task, and weighted and fused multiple noise prediction results. Denoising is performed on the weighted fusion noise prediction results to generate a potentially clear representation; The restored image is obtained by decoding the potential sharp representation.

3. The globally information-driven autonomous driving control method according to claim 1, characterized in that, The step of processing the restored image using an encoder to generate a BEV feature map, and generating an instance query based on the BEV feature map, includes: The restored image is input into an encoder with shared parameters to generate a ground BEV feature map and a UAV BEV feature map; A ground instance query is generated from the ground BEV feature map, and a drone instance query is generated from the drone BEV feature map. The ground instance query and the UAV instance query include semantic features and 3D reference points.

4. The globally information-driven autonomous driving control method according to claim 3, characterized in that, The step of aligning the UAV instance query with the ground instance query includes: A rigid transformation matrix is ​​constructed using the position and attitude angle data of the UAV. The rigid transformation matrix is ​​then used to project the 3D reference points in the UAV instance query onto the vehicle coordinate system to achieve spatial alignment. The semantic features in the UAV instance query are concatenated with the rotation matrix, and the concatenation result is refined using a perceptron to achieve feature alignment.

5. The globally information-driven autonomous driving control method according to claim 1, characterized in that, The step of matching the aligned UAV instance query with the ground instance query includes: The spatial distance and feature similarity of each aligned UAV instance query and each ground instance query are calculated. The matching score is obtained by weighted fusion of spatial distance and feature similarity; Based on the matching scores between the UAV instance query and each ground instance query, the ground instance query that matches the UAV instance query is determined.

6. The globally information-driven autonomous driving control method according to claim 1, characterized in that, The step of using the fused perception result and the restored image as visual input, and the generated natural language navigation instructions and questions as text input, and using a visual language model to output decision suggestions consistent with the navigation target, includes: Visual input is obtained by feature stitching of the fused perception result and the restored image, and text input is obtained by text stitching of the generated natural language navigation instructions and questions; Based on the visual and text inputs, an initial decision suggestion is generated using a visual language model, and an online summary information is generated from the initial decision suggestion. The visual language model is fine-tuned based on the summary information; Using the fine-tuned visual language model, along with the visual and text inputs, the system outputs decision recommendations consistent with the navigation target.

7. The globally information-driven autonomous driving control method according to claim 6, characterized in that, The step of fine-tuning the visual language model based on the summary information includes: A mutual information function is constructed based on the summary information and natural language navigation instructions, and a summary reward item and an initial decision reward item are defined; The visual language model is fine-tuned based on the mutual information function, the summary reward term, and the initial decision reward term.

8. The global information-driven autonomous driving control method according to claim 1, characterized in that, The step of concatenating the decision recommendations, fused perception results, and historical control signals to obtain a global multimodal representation includes: Extract the high-dimensional language features of the decision suggestion, and compress the high-dimensional language features into low-dimensional language features that are compatible with the fusion perception result; The weights of each source feature are learned by a multilayer perceptron, and the dimensionality-reduced language features, fused perception results, and historical control information are concatenated according to the learned weights to generate a global multimodal representation.

9. The global information-driven autonomous driving control method according to claim 1, characterized in that, The steps of outputting control actions for autonomous driving using a planning model under the constraints of the input features and the natural language navigation command representation include: The natural language navigation instructions are parsed into hard constraint terms and soft reward terms; The input features are input into the planning model, candidate actions are filtered based on the hard constraints, and the policy network is optimized based on the soft reward terms to output the control action for autonomous driving that meets the hard constraints and has the highest total reward.

10. A globally information-driven automatic driving control system, characterized in that, The system includes: The restoration processing module is used to acquire the original images captured by the vehicle-mounted camera and the drone, and to perform multi-task restoration processing on the original images to obtain the restored images. The generation module is used to process the restored image using an encoder to generate a BEV feature map, and to generate an instance query based on the BEV feature map. The instance query includes a ground instance query and a UAV instance query. The alignment and matching module is used to align the UAV instance query with the ground instance query, match the aligned UAV instance query with the ground instance query, and fuse the successfully matched UAV instance query and ground instance query to obtain the fused perception result. The model processing module is used to take the fused perception result and the restored image as visual input, and the generated natural language navigation instructions and questions as text input, and use the visual language model to output decision suggestions consistent with the navigation target. The splicing module is used to splice the decision suggestions, fused perception results and historical control signals to obtain a global multimodal representation; The control action generation module is used to fuse the global multimodal representation with the BEV feature map to generate input features, and under the constraints of the input features and the natural language navigation instruction representation, use a planning model to output control actions for autonomous driving.

Images