End-to-end perception decision regulation architecture determination method and device

Abstract

The application provides an end-to-end perception decision regulation architecture determination method and device, belongs to the technical field of automatic driving, and is based on a first multi-modal data set constructed on the basis of a basic traffic signal image, road map data and basic driving rules of an automatic driving scene, a second multi-modal data set constructed on the basis of dynamic traffic element image sequence data, complex environment element image sequence data and complex driving rules of the automatic driving scene, and a third multi-modal data set constructed on the basis of an automatic driving scene image, an automatic driving target and sensor time sequence data, and the multi-modal large model is trained in stages to obtain an end-to-end perception decision regulation architecture suitable for the automatic driving scene. The multi-modal large model is trained through the staged training strategy, and the training gradually realizes basic element recognition, complex element recognition and decision training, enhances the real-time performance of the model, and meets the real-time requirement of the automatic driving task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to an end-to-end perception decision regulation architecture determination method and device. BACKGROUND

[0002] The existing automatic driving technology mainly adopts a layered architecture. The layered architecture divides the automatic driving system into multiple modules, including a perception module, a prediction module, a planning module, and a control module. The perception module identifies static and dynamic objects in the surrounding environment through sensors such as cameras, lidar, radar, etc., and generates an environment map. The prediction module analyzes the behavior of dynamic objects to understand the movement trend of pedestrians and vehicles. The planning module formulates a reasonable path based on the perception and prediction information. The control module is responsible for executing the planning action. The layered architecture has a clear structure and is the mainstream of many current automatic driving solutions.

[0003] However, the independence between the modules may cause information loss or limited expression during transmission, making it difficult to optimize the overall system. In addition, due to the need for multiple modules to work together, the system complexity is high, the design cycle is long, and it may also cause certain computational delay, which is not ideal for real-time automatic driving tasks. SUMMARY

[0004] The present application provides an end-to-end perception decision regulation architecture determination method and device, which solves the defect that the layered architecture is not ideal for real-time automatic driving tasks.

[0005] The present application provides an end-to-end perception decision regulation architecture determination method, comprising:

[0006] A first multi-modal data set is constructed based on basic traffic signal images, road map data, and basic driving rules of an automatic driving scene. A pre-set multi-modal large model is trained.

[0007] A second multi-modal data set is constructed based on dynamic traffic element image sequence data, complex environment element image sequence data, and complex driving rules of an automatic driving scene. The multi-modal large model trained by the first multi-modal data set is trained.

[0008] A third multi-modal data set is constructed based on automatic driving scene images, automatic driving targets, and sensor time sequence data. The multi-modal large model trained by the second multi-modal data set is end-to-end trained to obtain an end-to-end perception decision regulation architecture suitable for an automatic driving scene.

[0009] The automatic driving scene images include one or more of basic traffic signals, road map data, dynamic traffic elements, and complex environment elements.

[0010] As an embodiment, the multi-modal large model comprises a cross-modal feature alignment module and a rule understanding module;

[0011] The cross-modal feature alignment module is configured to extract respective basic semantic features of the basic traffic signal image, the road map data, and the basic driving rules, map the respective basic semantic features to a same semantic embedding space to complete semantic feature alignment and semantic feature fusion, and obtain basic semantic fusion features.

[0012] The rule understanding module is configured to convert the basic driving rules into a structured semantic representation.

[0013] As an embodiment, the road map data comprises environment map information.

[0014] As an embodiment, the multi-modal large model further comprises a multi-modal perception module and a modal feature cross-decoding module.

[0015] The multi-modal perception module is configured to extract respective time sequence features of the dynamic traffic element image sequence data, the complex environment element image sequence data, and the complex driving rules.

[0016] The modal feature cross-decoding module is configured to interactively decode and globally aggregate the respective time sequence features to generate a unified high-order representation.

[0017] As an embodiment, the multi-modal large model further comprises a decision module.

[0018] The decision module is configured to encode the autonomous driving scene image, the autonomous driving target, and the sensor time sequence data into a joint representation, input the structured semantic representation, the unified high-order representation, and the joint representation into a reinforcement learning model, and generate a decision, which is used to represent a set of autonomous driving control instructions.

[0019] As an embodiment, the third multi-modal data set comprises a simulation multi-modal data subset and a real multi-modal data subset, and correspondingly, the multi-modal large model trained based on the second multi-modal data set is end-to-end trained to obtain an end-to-end perception decision regulation and control architecture suitable for an autonomous driving scene, comprising:

[0020] Based on the simulation multi-modal data subset, the multi-modal large model trained for complex recognition is simulated and trained, and in the simulation training process, the parameters of the multi-modal large model are optimized based on an end-to-end back propagation mechanism.

[0021] Based on the real multi-modal data subset, the multi-modal large model after simulation training is parameter adjusted to obtain the end-to-end perception decision regulation and control architecture.

[0022] The application also provides an end-to-end perception decision-making control architecture determination device, comprising:

[0023] A first training module is configured to train a preset multi-modal large model based on a first multi-modal data set constructed based on basic traffic signal images, road map data, and basic driving rules of an autonomous driving scene;

[0024] A second training module is configured to train the multi-modal large model trained based on the first multi-modal data set based on a second multi-modal data set constructed based on dynamic traffic element image sequence data, complex environment element image sequence data, and complex driving rules of the autonomous driving scene;

[0025] A third training module is configured to perform end-to-end training on the multi-modal large model trained based on the second multi-modal data set based on a third multi-modal data set constructed based on autonomous driving scene images, autonomous driving targets, and sensor time sequence data, to obtain an end-to-end perception decision-making control architecture suitable for the autonomous driving scene.

[0026] The autonomous driving scene images comprise one or more of the following: basic traffic signals, road map data, dynamic traffic elements, and complex environment elements.

[0027] The application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the end-to-end perception decision-making control architecture determination method of any of the above when executing the computer program.

[0028] The application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the end-to-end perception decision-making control architecture determination method of any of the above.

[0029] The application also provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the end-to-end perception decision-making control architecture determination method of any of the above.

[0030] The application provides an end-to-end perception decision-making control architecture determination method and device, which trains a multi-modal large model through a phased training strategy, gradually realizes identification of basic elements, identification of complex elements, and decision-making training, enhances the real-time performance of the model, avoids information loss in the traditional hierarchical architecture through an end-to-end process, significantly improves reaction efficiency and anti-interference ability, and meets the real-time requirements of autonomous driving tasks. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0032] Figure 1 is one of the flowcharts of the method for determining an end-to-end perception decision regulation architecture provided by the present application.

[0033] Figure 2 is another flowchart of the method for determining an end-to-end perception decision regulation architecture provided by the present application.

[0034] Figure 3 is a structural schematic diagram of a multi-modal large model provided by the present application.

[0035] Figure 4 is a structural schematic diagram of the apparatus for determining an end-to-end perception decision regulation architecture provided by the present application.

[0036] Figure 5 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION

[0037] In order to make the objects, technical solutions and advantages of the present application clearer, the following will combine the drawings in the present application to clearly and completely describe the technical solutions in the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the protection scope of the present application.

[0038] The existing automatic driving technology has many deficiencies in practical application. For the hierarchical architecture, although the multi-module design has the advantage of clear structure, the independence between the modules may cause information loss or limited expression in the transmission process, making it difficult to optimize the whole system. At the same time, due to the need for multiple modules to work together, the system complexity is high, the design cycle is long, and a certain calculation delay is brought, which is not ideal for real-time automatic driving tasks. In addition, although the end-to-end deep learning method can simplify the system architecture and directly map the input to the control instruction, the current mainstream end-to-end model mainly relies on single-modal data (such as camera images), which cannot utilize the rich information of multi-modal data, making the model's perception of the environment insufficient and having strong unexplainability, which makes it difficult to verify the driving decision logic and has poor generalization ability. Once encountering a scene not seen in the training data, the system performance may decrease significantly. In addition, although multi-modal fusion technology has been used to improve the perception ability, the current technology mainly focuses on the perception level and cannot be deeply integrated into the end-to-end driving decision process, resulting in challenges in robustness and real-time performance in practical applications. Overall, these shortcomings make it difficult for existing technologies to achieve ideal performance levels in complex and diverse real driving scenarios.

[0039] Therefore, the present application provides an end-to-end perception decision control architecture determination method and device, which will be described in detail below with reference to the accompanying drawings.

[0040] Figure 1 is one of the flowcharts of the end-to-end perception decision control architecture determination method provided by the present application, as shown in Figure 1 The present application provides an end-to-end perception decision control architecture determination method, which comprises steps S100-S300.

[0041] Step S100, based on the first multi-modal data set constructed by the basic traffic signal image, the road map data and the basic driving rules of the automatic driving scene, the pre-set multi-modal large model is trained.

[0042] Optionally, the road map data includes environmental map information, such as position data and lane distribution related to the position data,

[0043] The basic traffic signal image is used to represent an image containing traffic signals (such as traffic lights, turning signs, speed limit signs, etc.), and the basic driving rules are used to represent the driving rules set for the basic traffic signal image and the road map data, such as red light stop green light go, and careful driving is required at accident-prone locations.

[0044] The first multi-modal dataset includes basic traffic signal images of an image modality, road map data of a spatial modality, and basic driving rules of a text modality, uses image data as a main modality input, combines high-precision road map data and basic driving rules set by humans, and enhances the understanding of traffic signals, rules, and scenes of the multi-modal large model through a combination of vision, space, and language.

[0045] The preset multi-modal large model can be trained in a weakly supervised or semi-supervised learning manner, so that the multi-modal large model trained based on basic recognition can accurately classify and semantically label traffic basic elements in a scene, and generate a preliminary high-level semantic representation of the scene, such as a straight green light or a dangerous sharp turn.

[0046] In step S200, the multi-modal large model trained based on the first multi-modal dataset is trained based on a second multi-modal dataset constructed based on dynamic traffic element image sequence data, complex environment element image sequence data, and complex driving rules of an autonomous driving scene.

[0047] The dynamic traffic elements include dynamic participants such as pedestrians, vehicles, and bicycles on the road, the complex environment elements include complex structure roads (such as intersections and roundabouts) and driving environments (such as bad weather), and the complex driving rules are used to represent driving rules set for the dynamic traffic element image sequence data and the complex environment element image sequence data (such as a left pedestrian about to enter the lane and slow down to avoid). The complex driving rules contain language description of scene information and can be used as a supplementary feature.

[0048] The multi-modal large model trained based on the first multi-modal dataset can be trained in a weakly supervised learning manner to achieve complex recognition, real-time detection and recognition of the position, category, and behavior trend of dynamic traffic participants (for example, a pedestrian approaches from the right and slows down to brake), and also can achieve global semantic graph construction to form a complete dynamic scene understanding capability.

[0049] In step S300, the multi-modal large model trained based on the second multi-modal dataset is end-to-end trained based on a third multi-modal dataset constructed based on autonomous driving scene images, autonomous driving targets, and sensor time sequence data, to obtain an end-to-end perception decision rule control architecture suitable for autonomous driving scenes.

[0050] The autonomous driving scene images include one or more of basic traffic signals, road map data, dynamic traffic elements, and complex environment elements.

[0051] The automatic driving target is used to represent a natural language instruction set for an automatic driving scene image, such as turning left at the next intersection, and the sensor time sequence data includes time sequence data collected by a radar sensor, which is used to help the vehicle identify roads, obstacles, pedestrians, and the like, and enhance adaptability to extreme scenes (such as low visibility and irregular moving objects).

[0052] As shown in Figure 2 The training phase of the multi-modal large model provided by the present application includes a basic traffic signal recognition training step (i.e., step S100), a road traffic element recognition training step (i.e., step S200), and an end-to-end decision training step (i.e., step S300). Through a progressive training strategy, an efficient, reliable, and interpretable end-to-end automatic driving model based on a multi-modal large model (i.e., an end-to-end perception decision regulation architecture) is constructed to realize the complete process of autonomous behavior through perceiving the environment, making decisions, and performing control.

[0053] In the application phase, real-time image data collected by a high-definition camera, position information collected by a positioning system, and radar data collected by a sensor are input into the end-to-end perception decision regulation architecture to obtain driving control instructions (such as acceleration, steering, braking, forward movement, lane change, and deceleration) output by the end-to-end perception decision regulation architecture. In various complex scenarios, reliable real-time driving control decisions can be output to ensure that the logical path leading to the driving behavior has complete link interpretability.

[0054] It can be understood that the present application effectively improves the perception accuracy and scene understanding ability by introducing a multi-modal large model and adding deep fusion of multi-source information such as language description of the scene information. At the same time, through a phased training strategy, optimization from basic traffic signal recognition to dynamic traffic element perception to driving decision is gradually realized, significantly enhancing the generalization ability, robustness, and real-time performance. The end-to-end perception decision regulation architecture avoids the information loss problem in the traditional hierarchical architecture through an end-to-end process, significantly improves the reaction efficiency and anti-interference ability, and the deep fusion of multi-modal information makes the decision process more stable and reasonable in complex scenarios.

[0055] On the basis of the above-mentioned embodiments, as an optional embodiment, the multi-modal large model includes a cross-modal feature alignment module and a rule understanding module.

[0056] The cross-modal feature alignment module is configured to extract respective basic semantic features of the basic traffic signal image, the road map data, and the basic driving rules, map each of the basic semantic features to a same semantic embedding space to complete semantic feature alignment and semantic feature fusion, and obtain basic semantic fusion features.

[0057] The rule understanding module is configured to convert the basic driving rules into a structured semantic representation.

[0058] The cross-modal feature alignment module mainly includes a multi-channel encoder and an aggregation network. Each modality data (such as visual information, environmental structure, text rules, etc.) in the first multi-modal data set extracts basic semantic features through an independent dedicated neural network. The features of all modalities are mapped to the same semantic embedding space. Through cross-modal attention mechanism, fusion Transformer structure or contrast learning algorithm, semantic alignment and fusion are realized to establish deep semantic connection between different modalities.

[0059] The rule understanding module adopts a deep understanding model based on Transformer or a large language model. After feature fusion, the text rules are parsed and reasoned in context, and the text rules are expressed as structured semantic representations. The multi-modal environmental perception features are combined and input into the decision module to realize intelligent understanding and application of complex scene text rules.

[0060] It can be understood that the present application performs basic recognition training on the multi-modal large model, that is, the cross-modal feature alignment module and the rule understanding module are trained. Unlike the independent processing or simple splicing of single modality data in existing models, the traditional method is difficult to capture the complex semantic dependence between modalities. Through the cross-modal feature alignment module and the rule understanding module, semantic interaction and scene correlation understanding of different modalities are realized.

[0061] On the basis of the above-mentioned embodiments, as an optional embodiment, the multi-modal large model further comprises a multi-modal perception module and a modal feature cross-decoding module.

[0062] The multi-modal perception module is configured to extract time sequence features of the dynamic traffic element image sequence data, the complex environmental element image sequence data and the complex driving rules respectively.

[0063] The modal feature cross-decoding module is configured to interactively decode and globally aggregate the time sequence features to generate a unified high-order representation.

[0064] Optionally, the multi-modal perception module can also be connected with the cross-modal feature alignment module to transmit the extracted time sequence features to the cross-modal feature alignment module for feature alignment. The cross-modal feature alignment module transmits the aligned time sequence features to the rule understanding module, and the rule understanding module converts the time sequence features into structured semantic representation time sequence features. Correspondingly, the modal feature cross-decoding module interactively decodes and globally aggregates the structured semantic representation time sequence features to generate a unified high-order representation.

[0065] Optionally, the multi-modal large model further comprises a dynamic prediction network, which adopts a time modeling structure such as a neural module based on a Transformer, a time convolution network (TCN) or a long short-term memory network (LSTM), is used for modeling the time dynamics of the time series data in the second multi-modal data set, effectively captures the evolution law of the behavior of environmental elements and participants (pedestrians, vehicles, etc.) through the input of continuous time perception data flow, and realizes the early perception and state prediction of motion, change and emergencies in the scene.

[0066] Optionally, the modal feature cross decoding module comprises a heterogeneous information decoder or a gate fusion unit, and through a cross attention mechanism, the time series data from different modalities are deeply decoupled and re-integrated in time and semantic space, so as to make up for the limitations of a single modality in capturing global dynamics and sudden situations.

[0067] It can be understood that, by means of the multi-modal perception module and the modal feature cross decoding module, the limitations of a single modality in capturing global dynamics and sudden situations can be made up, and robust perception of dynamic road elements can be realized.

[0068] On the basis of the above-mentioned embodiments, as an optional embodiment, the multi-modal large model further comprises a decision module.

[0069] The decision module is configured to encode the automatic driving scene image, the automatic driving target and the sensor time series data into a joint representation, input the structured semantic representation, the unified high-order representation and the joint representation into a reinforcement learning model, and generate a decision, which is used to represent a set of automatic driving control instructions.

[0070] It can be understood that, by means of the end-to-end reinforcement learning driving, the decision module can realize the collaborative optimization of different targets (such as efficiency, stability, energy consumption, etc.).

[0071] On the basis of the above-mentioned embodiments, as an optional embodiment, the third multi-modal data set comprises a simulation multi-modal data subset and a real multi-modal data subset, and correspondingly, the multi-modal large model trained based on the second multi-modal data set is subjected to end-to-end training to obtain an end-to-end perception decision regulation and control architecture suitable for an automatic driving scene, which comprises:

[0072] Based on the simulation multi-modal data subset, the multi-modal large model trained for complex recognition is subjected to simulation training, and in the simulation training process, the parameters of the multi-modal large model are optimized based on an end-to-end back propagation mechanism.

[0073] Based on the real multi-modal data subset, the multi-modal large model subjected to simulation training is subjected to parameter adjustment to obtain the end-to-end perception decision regulation and control architecture.

[0074] It can be understood that the application carries out end-to-end reinforcement learning in a simulation environment, continuously adjusts the driving strategy to realize optimization of path planning and driving control, and then further fine-tunes in a real data environment to deal with noise and diversified terrain, which is beneficial to reliably output real-time driving control decisions in various complex scenes and ensures that the logical path leading to the driving behavior has complete link explainability.

[0075] Figure 3 is a structural schematic diagram of a preferred embodiment of the multi-modal large model provided by the application, and the arrows in the diagram are signal flow directions.

[0076] In the basic traffic signal recognition training step, the training process is organically penetrated by environment perception, rule processing and multi-modal feature alignment, the whole process parameters are optimized through end-to-end back propagation, so that the intelligent agent can realize comprehensive understanding of multi-modal information, context reasoning and action planning, and the generalization ability and adaptability of the multi-modal large model to complex environments are greatly improved.

[0077] The loss function expression of the basic traffic signal recognition training step is as follows:

[0078] L = λ1L cross + λ2L align + λ3L rule ;

[0079] Wherein, L represents the loss function of the basic traffic signal recognition training step; λ1, λ2, λ3 represent the weight coefficients of different sub-task losses respectively, for balancing the contributions of each part in the overall training target; L cross represents the main task loss of the basic recognition (such as classification, detection, etc.) in the multi-modal scene; L align represents the alignment degree of different modalities (such as vision, structure, text, etc.) in the public semantic space by contrast learning or embedding similarity loss, for example, using contrast loss (such as InfoNCE), mean square error (MSE) or triplet loss (Triplet Loss) method; L rule represents the context analysis loss of the rule understanding module, for constraining the understanding and reasoning accuracy of the model to the text rule expression (such as the cross-entropy loss of rule prediction).

[0080] When the multi-modal large model is trained, all parameters θ (including the weights and biases of the multi-modal encoder, the feature aggregation network, the cross-modal fusion module and the rule analysis network, etc.) are updated synchronously through the end-to-end back propagation algorithm (such as Adam or SGD optimizer), and the optimization target is to minimize the total loss L. The parameter update expression is as follows:

[0081] θ ← θ - η L / θ

[0082] wherein θ represents the set of trainable parameters of the full model; η represents the learning rate; L / θ represents the gradient of the total loss L with respect to the model parameters θ.

[0083] Through the above end-to-end loss joint and parameter optimization process, the unified fusion and generalization modeling of multi-modal features, context rules and environmental perception information can be efficiently realized, thereby significantly improving the comprehensive intelligent capability of the multi-modal large model in complex scenarios.

[0084] In the road traffic element recognition training step, training and optimization are performed through an end-to-end deep learning method, and algorithm frameworks such as sequence-to-sequence loss functions, dynamic event detection loss, or multi-task loss are used to realize accurate perception and intelligent prediction of events and states in complex and variable environments, thereby significantly improving the global understanding and dynamic adaptation capability of the intelligent agent in complex scenarios.

[0085] The decision module is composed of an input perception coding unit, a global scene perception module, a strategy network, a multi-task optimization part and an execution output layer. The input perception coding unit encodes the original image, target information and real-time dynamic data stream (including multi-modal time sequence features of sensors, state of the agent and the environment, etc.) into a joint representation, which is input to the global scene perception module. The global scene perception module models the fusion attention mechanism and the context association, dynamically understands the multi-target and environmental elements in the scene, and outputs high-dimensional features of the current state. The downstream multi-task strategy optimization network takes reinforcement learning as the basic framework, usually uses deep reinforcement learning algorithms (such as DQN, DDPG, PPO, SAC, etc.) or their multi-task variants, makes autonomous decisions on the environment state through the strategy network, and outputs a series of candidate actions and their probability distribution, realizing the collaborative optimization of different targets (such as efficiency, stability, energy consumption, etc.). During the training process, the parameters of the strategy network are continuously optimized through end-to-end back propagation using environmental feedback (reward signal), and the ability of adaptive dynamic adjustment and multi-target task weighting is achieved. The overall process of the module is perception, fusion, global understanding, strategy decision, execution feedback and continuous optimization, which can realize dynamic prediction and path planning of the agent behavior under complex dynamic environment and multi-target constraints, and continuously improve the strategy robustness and generalization ability. In addition, in order to improve the efficiency of multi-target optimization, mechanisms such as hierarchical reinforcement learning, multi-target reward function design and adaptive experience replay can be introduced to realize the efficient adaptation and optimal behavior output of the strategy network to complex scenes and dynamic targets. By introducing the global scene perception module and the multi-task strategy optimization network, the dynamic prediction and multi-target optimization path planning of behavior are realized (different from the existing end-to-end methods which do not use reinforcement learning and lack dynamic modeling of real targets). Finally, an end-to-end perception decision regulation model that can adapt to complex dynamic environments can be obtained.

[0086] Next, the hardware and software environment of the experiment of the present application, the data set used for the experiment, the experimental setting and the experimental evaluation index are introduced in detail.

[0087] (1) Experimental environment.

[0088] The detailed information of the environment configuration is shown in Table 1.

[0089] Table 1 Experimental environment configuration

[0090]

[0091] (2) Experimental data set.

[0092] The proposed method is verified on classical data sets, namely NuScenes and Waymo data sets, and various experimental settings are tried.

[0093] (3) Experimental setting.

[0094] The application compares the results with traditional models UniAD, VAD, BEV-Planner, pure multi-modal large models: LLava, Vicuna, and end-to-end models based on multi-modal large models: EMMA, Omnidrive. The application uses the indicators provided by nuScenes for overall evaluation, including L2, collision rate and boundary rate. Through experiments, the end-to-end perception decision-making and control architecture provided by the application is superior to existing models.

[0095] In summary, the application introduces a multi-modal large model, integrates multi-modal data into an end-to-end architecture, combines a weakly supervised learning method, adds language description of scene information and other multi-source information deep fusion, effectively improves the perception accuracy and scene understanding ability, gradually from accurate recognition of basic traffic signals, to robust perception of dynamic road elements, and finally completes driving decision training, to improve the adaptability, real-time performance and safety of the automatic driving system in complex open environments, while enhancing the explainability of driving decisions, overcoming the black box defect of existing end-to-end models, and meeting the actual needs of high-complexity driving scenarios. Through multi-modal cross-training, the generalization ability of multi-source traffic rule data is achieved. The pre-training ability of the language model is used to embed abstract rules into the model to deepen traffic semantic understanding. Rich sensor fusion information enhances the adaptability of the system to extreme scenarios such as low visibility and irregular moving objects. Time series dynamic modeling improves the prediction ability for complex driving scenarios. The end-to-end process avoids the information loss problem in traditional hierarchical architecture, significantly improves the reaction efficiency and anti-interference ability. Deep fusion of multi-modal information makes the decision-making process more stable and reasonable in complex scenarios.

[0096] The end-to-end perception decision-making and control architecture determination device provided by the application is described below, and the end-to-end perception decision-making and control architecture determination device described below can be mutually corresponding with the end-to-end perception decision-making and control architecture determination method described above.

[0097] Figure 4 is a structural schematic diagram of the end-to-end perception decision-making and control architecture determination device provided by the application, as Figure 4 shown, the application further provides an end-to-end perception decision-making and control architecture determination device, comprising the following modules.

[0098] The first training module 410 is configured to train a preset multi-modal large model based on a first multi-modal data set constructed based on basic traffic signal images, road map data and basic driving rules of an automatic driving scene;

[0099] The second training module 420 is configured to train the multi-modal large model trained based on the first multi-modal data set, based on a second multi-modal data set constructed based on dynamic traffic element image sequence data, complex environment element image sequence data, and complex driving rules of the autonomous driving scene;

[0100] The third training module 430 is configured to perform end-to-end training on the multi-modal large model trained based on the second multi-modal data set, based on a third multi-modal data set constructed based on autonomous driving scene images, autonomous driving targets, and sensor time sequence data, to obtain an end-to-end perception decision rule control architecture suitable for the autonomous driving scene.

[0101] The autonomous driving scene images include one or more of basic traffic signals, road map data, dynamic traffic elements, and complex environment elements.

[0102] As an embodiment, the multi-modal large model includes a cross-modal feature alignment module and a rule understanding module.

[0103] The cross-modal feature alignment module is configured to extract respective basic semantic features of the basic traffic signal images, the road map data, and the basic driving rules, map the respective basic semantic features to a same semantic embedding space to complete semantic feature alignment and semantic feature fusion, and obtain basic semantic fusion features.

[0104] The rule understanding module is configured to convert the basic driving rules into a structured semantic representation.

[0105] As an embodiment, the road map data includes environment map information.

[0106] As an embodiment, the multi-modal large model further includes a multi-modal perception module and a modal feature cross-decoding module.

[0107] The multi-modal perception module is configured to extract respective time sequence features of the dynamic traffic element image sequence data, the complex environment element image sequence data, and the complex driving rules.

[0108] The modal feature cross-decoding module is configured to perform interactive decoding and global aggregation on the respective time sequence features to generate a unified high-order representation.

[0109] As an embodiment, the multi-modal large model further includes a decision module.

[0110] The decision module is used to encode the autonomous driving scene image, the autonomous driving target, and the sensor time-series data into a joint representation, and input the structured semantic representation, the unified high-order representation, and the joint representation into a reinforcement learning model to generate a decision, which is used to represent the set of autonomous driving control commands.

[0111] As one embodiment, the third multimodal dataset includes a subset of simulated multimodal data and a subset of real multimodal data. Correspondingly, the end-to-end training of the large multimodal model trained on the second multimodal dataset to obtain an end-to-end perception, decision-making, and control architecture suitable for autonomous driving scenarios includes:

[0112] Based on the aforementioned subset of simulated multimodal data, a large multimodal model trained on complex recognition is simulated and trained. During the simulation training process, the parameters of the large multimodal model are optimized based on an end-to-end backpropagation mechanism.

[0113] Based on the real multimodal data subset, the parameters of the multimodal large model after simulation training are adjusted to obtain the end-to-end perception, decision-making, and control architecture.

[0114] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute the end-to-end perception decision-making and control architecture determination method, the method including:

[0115] The first multimodal dataset, constructed based on basic traffic signal images, road map data, and basic driving rules for autonomous driving scenarios, is used to train a pre-defined multimodal large model.

[0116] The second multimodal dataset, constructed based on dynamic traffic element image sequence data, complex environment element image sequence data, and complex driving rules in autonomous driving scenarios, is used to train the multimodal large model trained on the first multimodal dataset.

[0117] A third multimodal dataset is constructed based on autonomous driving scene images, autonomous driving targets, and sensor time-series data. The multimodal large model trained on the second multimodal dataset is then trained end-to-end to obtain an end-to-end perception, decision-making, and control architecture suitable for autonomous driving scenarios.

[0118] The automatic driving scene image includes one or more of a basic traffic signal, road map data, a dynamic traffic element, and a complex environment element.

[0119] In addition, the logic instructions in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0120] On the other hand, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor, so that the computer can execute the end-to-end perception decision regulation architecture determination method provided by the above-mentioned method. The method comprises:

[0121] Based on the first multi-modal data set constructed based on the basic traffic signal image, the road map data, and the basic driving rules of the automatic driving scene, the pre-set multi-modal large model is trained;

[0122] Based on the second multi-modal data set constructed based on the dynamic traffic element image sequence data, the complex environment element image sequence data, and the complex driving rules of the automatic driving scene, the multi-modal large model trained based on the first multi-modal data set is trained;

[0123] Based on the third multi-modal data set constructed based on the automatic driving scene image, the automatic driving target, and the sensor time sequence data, the multi-modal large model trained based on the second multi-modal data set is trained in an end-to-end manner, and an end-to-end perception decision regulation architecture suitable for the automatic driving scene is obtained.

[0124] The automatic driving scene image includes one or more of a basic traffic signal, road map data, a dynamic traffic element, and a complex environment element.

[0125] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the end-to-end perception decision-making and control architecture determination method provided by the above method, and the method comprises:

[0126] A first multi-modal data set is constructed based on the basic traffic signal images, road map data and basic driving rules of the autonomous driving scene, and a preset multi-modal large model is trained;

[0127] A second multi-modal data set is constructed based on the dynamic traffic element image sequence data, complex environment element image sequence data and complex driving rules of the autonomous driving scene, and the multi-modal large model trained based on the first multi-modal data set is trained;

[0128] A third multi-modal data set is constructed based on the autonomous driving scene images, autonomous driving targets and sensor time sequence data, and the multi-modal large model trained based on the second multi-modal data set is trained in an end-to-end manner to obtain an end-to-end perception decision-making and control architecture suitable for the autonomous driving scene;

[0129] The autonomous driving scene images include one or more of the basic traffic signals, road map data, dynamic traffic elements and complex environment elements.

[0130] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0131] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in terms of the contribution to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0132] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An end-to-end perception decision regulation architecture determination method, characterized in that, The method comprises the steps of: constructing a first multi-modal data set based on basic traffic signal images of an automatic driving scene, road map data, and basic driving rules, and training a preset multi-modal large model; constructing a second multi-modal data set based on dynamic traffic element image sequence data of an automatic driving scene, complex environment element image sequence data, and complex driving rules, and training the multi-modal large model trained based on the first multi-modal data set; constructing a third multi-modal data set based on automatic driving scene images, automatic driving targets, and sensor time sequence data, and performing end-to-end training on the multi-modal large model trained based on the second multi-modal data set to obtain an end-to-end perception decision rule control architecture suitable for an automatic driving scene; wherein the automatic driving scene images comprise one or more of basic traffic signals, road map data, dynamic traffic elements, and complex environment elements; the multi-modal large model comprises a cross-modal feature alignment module, a rule understanding module, a multi-modal perception module, a modal feature cross-decoding module, and a decision module; the cross-modal feature alignment module is configured to extract basic semantic features of the basic traffic signal images, the road map data, and the basic driving rules respectively, map the basic semantic features to a same semantic embedding space to complete semantic feature alignment and semantic feature fusion, and obtain basic semantic fusion features; the rule understanding module is configured to convert the basic driving rules into a structured semantic representation; the multi-modal perception module is configured to extract time sequence features of the dynamic traffic element image sequence data, the complex environment element image sequence data, and the complex driving rules respectively; the modal feature cross-decoding module is configured to perform interactive decoding and global aggregation on the time sequence features to generate a unified high-order representation; the decision module is configured to encode the automatic driving scene images, the automatic driving targets, and the sensor time sequence data into a joint representation, input the structured semantic representation, the unified high-order representation, and the joint representation into a reinforcement learning model, and generate a decision, which is used to represent a set of automatic driving control instructions.

2. The end-to-end perception decision regulation architecture determination method of claim 1, wherein, The road map data comprises environment map information.

3. The end-to-end perception decision regulation architecture determination method according to any one of claims 1-2, characterized in that, The third multi-modal data set comprises a simulation multi-modal data subset and a real multi-modal data subset, and correspondingly, the end-to-end training of the multi-modal large model trained based on the second multi-modal data set to obtain the end-to-end perception decision rule control architecture suitable for the automatic driving scene comprises: based on the simulation multi-modal data subset, performing simulation training on the multi-modal large model trained based on complex recognition, and in the simulation training process, optimizing parameters of the multi-modal large model based on an end-to-end back propagation mechanism; based on the real multi-modal data subset, adjusting parameters of the multi-modal large model after simulation training to obtain the end-to-end perception decision rule control architecture.

4. An end-to-end perception decision regulation architecture determination apparatus characterized by, The method comprises the steps of: a first training module is configured to construct a first multi-modal data set based on basic traffic signal images of an automatic driving scene, road map data, and basic driving rules, and train a preset multi-modal large model; a second training module configured to train the multi-modal large model trained based on the first multi-modal dataset based on a second multi-modal dataset constructed based on dynamic traffic element image sequence data, complex environment element image sequence data, and complex driving rules of an autonomous driving scene; a third training module configured to train the multi-modal large model trained based on the second multi-modal dataset based on a third multi-modal dataset constructed based on autonomous driving scene images, autonomous driving targets, and sensor time sequence data, to obtain an end-to-end perception decision regulation architecture applicable to the autonomous driving scene; wherein the autonomous driving scene images include one or more of basic traffic signals, road map data, dynamic traffic elements, and complex environment elements; the multi-modal large model includes a cross-modal feature alignment module, a rule understanding module, a multi-modal perception module, a modal feature cross-decoding module, and a decision module; the cross-modal feature alignment module is configured to extract basic semantic features of the basic traffic signal images, the road map data, and the basic driving rules, respectively, map the basic semantic features to a same semantic embedding space to complete semantic feature alignment and semantic feature fusion, and obtain basic semantic fusion features; the rule understanding module is configured to convert the basic driving rules into a structured semantic representation; the multi-modal perception module is configured to extract time sequence features of the dynamic traffic element image sequence data, the complex environment element image sequence data, and the complex driving rules, respectively; the modal feature cross-decoding module is configured to interactively decode and globally aggregate the time sequence features to generate a unified high-order representation; the decision module is configured to encode the autonomous driving scene images, the autonomous driving targets, and the sensor time sequence data into a joint representation, input the structured semantic representation, the unified high-order representation, and the joint representation into a reinforcement learning model, and generate a decision, which is used to represent a set of autonomous driving control instructions.

5. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the end-to-end perception decision regulation architecture determination method of any one of claims 1-3.

6. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the end-to-end perception decision regulation architecture determination method of any one of claims 1-3.

7. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the end-to-end perception decision regulation architecture determination method of any one of claims 1-3.

Images