Automatic driving optimization method, electronic device and storage medium

Abstract

The application discloses an automatic driving optimization method, an electronic device and a storage medium, relates to the technical field of vehicles, and is applied to a first device, wherein a first cognitive decision model is deployed on the first device and comprises the following steps: acquiring in-vehicle and out-vehicle monitoring data, inputting the in-vehicle and out-vehicle monitoring data into the first cognitive decision model to obtain first automatic driving control parameters; determining a target first automatic driving control parameter based on user question information, acquiring a first driving behavior explanation corresponding to the target first automatic driving control parameter generated by the first cognitive decision model, and outputting the first driving behavior explanation; and optimizing the first cognitive decision model based on user feedback information. The application improves the explainability of an end-to-end perception decision architecture, reduces negative emotions such as doubt, panic and unease of users, and improves user experience.

Description

Technical Field

[0001] This application relates to the field of vehicle technology, and in particular to an autonomous driving optimization method, electronic device, and storage medium. Background Technology

[0002] With the continuous development of technology and breakthroughs in computing power, autonomous driving is developing rapidly. Autonomous driving can provide people with a safer, more convenient and efficient way to travel, while also having a profound impact on the entire transportation system and urban planning.

[0003] The mainstream technical solutions for autonomous driving systems can be divided into two main types: rule-based modular architecture and data-driven end-to-end perception and decision-making architecture. While rule-based modular architectures have achieved mass production due to their advantages such as low computational requirements, simple deployment, and interpretable processes, they struggle to cover the entire scenario domain, and the results of modular solutions are not globally optimal. End-to-end solutions, on the other hand, use raw sensor data as input and generate planned and / or low-level control actions as output, enabling joint, global optimization and resulting in higher accuracy in autonomous driving decisions.

[0004] However, the end-to-end perception and decision-making architecture has poor interpretability. Therefore, when controlling the vehicle to perform sudden behaviors, it is easy for users riding in autonomous vehicles to be unable to understand the driving behavior of autonomous vehicles, resulting in negative emotions such as confusion, panic, and anxiety, and a poor user experience. Summary of the Invention

[0005] The main purpose of this application is to provide an autonomous driving optimization method, electronic device and storage medium, which aims to solve the technical problem of poor interpretability of end-to-end perception decision schemes in related technologies.

[0006] To achieve the above objectives, this application provides an autonomous driving optimization method, which is applied to a first device on which a first cognitive decision-making model is deployed. The autonomous driving optimization method includes the following steps:

[0007] Acquire in-vehicle and out-of-vehicle monitoring data, input the in-vehicle and out-of-vehicle monitoring data into the first cognitive decision model, and obtain at least one first autonomous driving control parameter;

[0008] Upon detecting user question information, a target first autonomous driving control parameter is determined from each of the first autonomous driving control parameters based on the user question information. A first driving behavior explanation corresponding to the target first autonomous driving control parameter generated by the first cognitive decision model is obtained, and the first driving behavior explanation is output.

[0009] Upon detecting user feedback information generated in response to the explanation of the first driving behavior, the first cognitive decision-making model is optimized based on the user feedback information.

[0010] This application also provides an autonomous driving optimization method, which is applied to a second device on which a second cognitive decision-making model is deployed. The autonomous driving optimization method includes the following steps:

[0011] Initialize the second cognitive decision model so that the model parameters of the second cognitive decision model are the same as the model parameters of the first cognitive decision model deployed on the first device;

[0012] Receive user feedback information and vehicle interior / exterior monitoring data sent by the first device;

[0013] The in-vehicle and out-of-vehicle monitoring data are input into the second cognitive decision model to obtain at least one second autonomous driving control parameter and the second driving behavior explanation corresponding to each second autonomous driving control parameter.

[0014] Based on the differences between the user feedback information and the second autonomous driving control parameters, and the differences between the user feedback information and the second driving behavior explanation, the second cognitive decision model is optimized.

[0015] The first optimized model parameters of the optimized second cognitive decision model are sent to the first device so that the first device can optimize its own deployed first cognitive decision model based on the first optimized model parameters.

[0016] This application also provides an autonomous driving optimization device, on which a first cognitive decision-making model is deployed, the autonomous driving optimization device comprising:

[0017] The acquisition module is used to acquire in-vehicle and out-of-vehicle monitoring data, input the in-vehicle and out-of-vehicle monitoring data into the first cognitive decision model, and obtain at least one first autonomous driving control parameter;

[0018] The output module is used to determine the target first autonomous driving control parameter from each of the first autonomous driving control parameters based on the user question information when the user question information is detected, obtain the first driving behavior explanation corresponding to the target first autonomous driving control parameter generated by the first cognitive decision model, and output the first driving behavior explanation.

[0019] The first optimization module is used to optimize the first cognitive decision model based on the user feedback information generated in response to the explanation of the first driving behavior.

[0020] This application also provides an autonomous driving optimization device, on which a second cognitive decision-making model is deployed, the autonomous driving optimization device comprising:

[0021] An initialization module is used to initialize the second cognitive decision model so that the model parameters of the second cognitive decision model are the same as the model parameters of the first cognitive decision model deployed on the first device.

[0022] The receiving module is used to receive user feedback information and vehicle interior and exterior monitoring data sent by the first device;

[0023] The cognitive decision module is used to input the in-vehicle and out-of-vehicle monitoring data into the second cognitive decision model to obtain at least one second autonomous driving control parameter and the second driving behavior explanation corresponding to each of the second autonomous driving control parameters.

[0024] The second optimization module is used to optimize the second cognitive decision model based on the difference between the user feedback information and the second autonomous driving control parameters, and the difference between the user feedback information and the second driving behavior explanation.

[0025] The sending module sends the first optimized model parameters of the optimized second cognitive decision model to the first device, so that the first device can optimize its own deployed first cognitive decision model based on the first optimized model parameters.

[0026] This application also provides an electronic device, which is a physical device, comprising: a memory, a processor, and a program of the autonomous driving optimization method stored in the memory and executable on the processor. When the program of the autonomous driving optimization method is executed by the processor, it can implement the steps of the autonomous driving optimization method as described above.

[0027] This application also provides a storage medium, which is a computer-readable storage medium, on which a program for implementing an autonomous driving optimization method is stored. When the program for the autonomous driving optimization method is executed by a processor, it implements the steps of the autonomous driving optimization method as described above.

[0028] This application provides an autonomous driving optimization method, an electronic device, and a storage medium. The autonomous driving optimization method is applied to a first device, on which a first cognitive decision-making model is deployed. First, by acquiring in-vehicle and out-of-vehicle monitoring data, the data is input into the first cognitive decision-making model to obtain at least one first autonomous driving control parameter, realizing the practical application of the first cognitive decision-making model. Then, upon detecting user question information, a target first autonomous driving control parameter is determined from the various first autonomous driving control parameters based on the user question information. A first driving behavior explanation corresponding to the target first autonomous driving control parameter generated by the first cognitive decision-making model is obtained and output. This achieves the purpose of interacting with the user, explaining the driving behavior, and dispelling user doubts when the user has difficulty understanding it. Furthermore, upon detecting user feedback information generated regarding the first driving behavior explanation, the first cognitive decision-making model is optimized based on the user feedback information, realizing online training of the first cognitive decision-making model in practical applications. On the one hand, this makes the cognition and decision-making of the first cognitive decision-making model closer to human thinking logic, thereby making the driving behavior decided by the first cognitive model easier for users to understand. Therefore, this approach overcomes the poor interpretability of end-to-end perception and decision-making architectures. This is because, when controlling a vehicle to perform sudden actions, users may experience confusion, panic, and anxiety due to the inability to understand the driving behavior, resulting in a poor user experience. The approach improves the interpretability of the end-to-end perception and decision-making architecture, reducing negative emotions such as confusion, panic, and anxiety caused by the unknown, thus enhancing the user experience. On the other hand, while a limited number of training samples can efficiently train a first cognitive decision-making model that meets the needs of most scenarios, it is usually difficult to comprehensively cover all scenarios. Therefore, when encountering new boundary scenarios in practical applications, interaction with users can learn their cognitive decision-making methods, enabling further optimization of the first cognitive decision-making model and effectively solving the long-tail problem caused by the inability of training samples to cover all scenarios. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating the first embodiment of the autonomous driving optimization method of this application;

[0032] Figure 2 This is a schematic diagram of a possible implementation scenario for the autonomous driving optimization method in the embodiments of this application;

[0033] Figure 3 This is a flowchart illustrating the second embodiment of the autonomous driving optimization method of this application;

[0034] Figure 4 This is a schematic diagram of the structure of the autonomous driving optimization device in the embodiments of this application;

[0035] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the autonomous driving optimization method in this application embodiment.

[0036] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0037] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Example 1

[0039] This application provides an autonomous driving optimization method, referring to... Figure 1 In a first embodiment of the autonomous driving optimization method of this application, the autonomous driving optimization method is applied to a first device, on which a first cognitive decision-making model is deployed, and the autonomous driving optimization method includes the following steps:

[0040] Step S10: Acquire in-vehicle and out-of-vehicle monitoring data, input the in-vehicle and out-of-vehicle monitoring data into the first cognitive decision model, and obtain at least one first autonomous driving control parameter;

[0041] The execution subject of the method in this embodiment can be an autonomous driving optimization device, an autonomous driving optimization terminal device, or a server. This embodiment takes an autonomous driving optimization device as an example. The autonomous driving optimization device can be integrated into terminal devices such as vehicles, in-vehicle terminals, smartphones, and computers with data processing functions.

[0042] In this embodiment, it should be noted that the first device may be a vehicle, an in-vehicle terminal, etc. A first cognitive decision-making model is deployed on the first device. This model is used to achieve end-to-end driving behavior control of the autonomous vehicle. The cognitive decision-making model uses an attention mechanism and can be a deep learning model based on the Transformer structure, or other deep learning neural network models using attention mechanisms. The specific design can be tailored to the actual situation, and this embodiment does not impose any limitations on this.

[0043] The first cognitive decision-making model includes at least a first cognitive layer, a first decision layer, and a first interpretation layer. The cognitive layer extracts, fuses, encodes, and infers features from the acquired in-vehicle and out-of-vehicle monitoring data to obtain the cognitive layer features required by the decision layer. The decision layer plans the autonomous vehicle's driving behavior in the next time step based on the cognitive layer features transmitted from the cognitive layer, enabling the autonomous vehicle to avoid risks and safely reach its destination. The interpretation layer interprets the autonomous vehicle's driving behavior based on the outputs of the cognitive layer and the decision layer. After inputting the in-vehicle and out-of-vehicle monitoring data into the first cognitive decision-making model, cognitive layer features are first extracted through the first cognitive layer, then input into the first decision layer to determine autonomous driving control parameters, and finally the interpretation layer decodes the decision features used to determine the autonomous driving control parameters. The autonomous driving control parameters and decision features are then aggregated to generate a first driving behavior interpretation.

[0044] The in-vehicle and out-of-vehicle monitoring data refers to the data required for autonomous driving optimization, including external monitoring data and cabin monitoring data. This data can be one or more modalities such as image data, text data, sound data, and signal data. For example, external monitoring data can be external image data collected by a camera, external sound data collected by a microphone, or traffic light countdown data obtained through a communication module. For example, cabin monitoring data can be vehicle speed, acceleration, and other vehicle driving data obtained from the vehicle controller or controller area network bus, or cabin image data collected by a camera, driver image data, cabin sound data collected by a microphone, text data converted from collected speech, or text data extracted from collected images. This in-vehicle and out-of-vehicle monitoring data can be collected in real time, thus carrying real-time information about the status of other road users, the driver's status, road conditions, and other potentially changing information.

[0045] For different modalities of vehicle interior and exterior monitoring data, different models can be used for feature extraction. For example, convolutional neural networks can be used for feature extraction of image data, and long short-term memory networks can be used for feature extraction of time series data. Different models can also be used for feature processing to extract different risk features required by the decision-making layer. Therefore, multiple models can be deployed in the cognitive layer for feature processing. The specific model structure and model type can be determined according to the actual situation. This embodiment does not impose any restrictions on this.

[0046] Cognitive layer features can include at least one of environmental cognitive features and user demand features. The environmental cognitive features characterize the autonomous vehicle's perception of its current environment, such as its perception of traffic facilities, ground obstacles, its own driving status, other traffic participants, the risks of the scene, and risk items within the scene. The main purpose of autonomous vehicle behavior control is to ensure the autonomous vehicle safely reaches its destination; that is, to control the vehicle to its destination while avoiding risks. In one feasible approach, autonomous driving optimization can involve predicting high-risk and / or low-risk driving areas for the autonomous vehicle in the next time step, thereby planning the autonomous vehicle's driving behavior to avoid high-risk areas or pass through low-risk areas in the next time step. Since the environmental cognitive features are an objective representation of the autonomous vehicle's current environment, the more accurate the extracted environmental cognitive features, the more accurate the risk assessment of the autonomous vehicle's current situation, and the higher the safety of the autonomous driving optimization. The environmental cognitive features may include scene cognitive features, map features, risk target detection and tracking features, risk target movement features, low-risk driving area features, and other features. The cognitive layer can be a multi-layered structure, meaning that the input of any cognitive layer can be the output of other cognitive layers, and the output of any cognitive layer can be the input of other cognitive layers. This enables the fusion and reasoning of multiple features, capturing the correlation and dependency between the various features extracted by the cognitive layers, and achieving a more comprehensive understanding and deeper logical reasoning of the current driving situation. The user demand features are used to characterize the needs of users riding in the autonomous vehicle, such as the user's needs for the destination, the user's needs for the route to the destination, the user's need to save time, and the user's need for a smooth ride to reduce motion sickness. The user demand characteristics mentioned above can be captured by detecting user operations, collecting user images, etc., and then the user demand characteristics can be extracted from them. For example, user images can be collected by a camera. If the image recognition technology identifies that the user is currently asleep, the user's sleep state usually implies a need for the vehicle to drive smoothly. For example, user voice information can also be collected by a microphone. If the user says "drive faster", it can be known that the user has a need for acceleration.

[0047] The cognitive layer can employ an attention mechanism, which, in one feasible approach, can utilize a transformer architecture. This attention mechanism effectively extracts, fuses, encodes, and infers features from various modalities and types of vehicle monitoring data. It extracts real-time information that may change at the current moment, combines it with historical information from a previous period to form an information sequence, and captures the correlations and dependencies between different types of information. Through feature inference, it extracts risk features with deep logical information, which the decision layer then uses to optimize autonomous driving. This allows for comprehensive consideration of potentially changing information, multiple risk items, and the interplay between user needs during the prediction of autonomous vehicle driving risks, improving the accuracy of risk prediction. Consequently, autonomous vehicles can avoid more risks while better meeting user needs, enhancing the safety and comfort of autonomous driving optimization. The correlations and dependencies between different types of information can significantly impact autonomous driving optimization. For example, interactions may occur between traffic participants, between traffic participants and traffic facilities, and between traffic participants and the surrounding environment. For example, at an intersection, this vehicle is traveling east-west, and there is a second vehicle traveling north-south and pedestrians walking east-west nearby. Assuming no interaction between traffic participants, the two vehicles will not collide at their current speeds. However, if the second vehicle slows down or stops to avoid a pedestrian, a collision may occur. Therefore, considering interactions between traffic participants, this vehicle still needs to slow down or plan an alternative route. If the second vehicle is in a turning lane, considering interactions between traffic participants and traffic facilities, if the second vehicle's turning trajectory does not intersect with this vehicle's, this vehicle does not need to slow down. Furthermore, considering interactions between traffic participants and the environment, the complexity of an intersection increases the risk to both people and vehicles, potentially requiring the vehicle to slow down.

[0048] Autonomous driving control parameters may include driving trajectory, vehicle control parameters, etc. The driving trajectory refers to the trajectory of the autonomous vehicle after a certain period of time from the current moment, and the autonomous vehicle can be controlled to drive based on the driving trajectory. The vehicle control parameters include speed, accelerator pedal opening, steering wheel angle, etc.

[0049] As an example, step S10 includes: collecting sensor data at the current moment as in-vehicle and out-of-vehicle monitoring data using one or more sensors installed on the autonomous vehicle; and communicating with external devices via a communication module on the autonomous vehicle to obtain external data provided by the external devices as in-vehicle and out-of-vehicle monitoring data, such as traffic light signals, traffic accident information, and roadside monitoring data collected by roadside monitoring equipment. The in-vehicle and out-of-vehicle monitoring data is then input into the first cognitive decision-making model for data processing including cognition, decision-making, and analysis to determine the first autonomous driving control parameters for the autonomous vehicle at the current moment or for a period of time after the current moment.

[0050] Optionally, before the step of acquiring in-vehicle and out-of-vehicle monitoring data, the method further includes:

[0051] Step A10: Obtain the monitoring data inside and outside the training sample vehicle, the labeled values ​​of the first autonomous driving control parameters, and the labeled values ​​of the first driving behavior interpretation;

[0052] Step A20: Input the training sample vehicle interior and exterior monitoring data into the first cognitive decision model to be trained to obtain at least one training sample first autonomous driving control parameter and the training sample first driving behavior explanation corresponding to each training sample first autonomous driving control parameter.

[0053] Step A30: Based on the difference between the first autonomous driving control parameter and the labeled value of the first autonomous driving control parameter in the training sample, and the difference between the first driving behavior explanation and the labeled value of the first driving behavior explanation in the training sample, the first cognitive decision model to be trained is iteratively updated.

[0054] In this embodiment, it should be noted that before applying the cognitive decision-making model, offline training of the model is required. During each iteration of offline training, the first cognitive decision-making model can be updated based on the calculated model loss until it converges. The training process of the first cognitive decision-making model can be performed on the final application device. Considering the computational power requirements for model training, the offline-trained first cognitive decision-making model can also be deployed to the application device after training on other devices. For example, the first cognitive decision-making model can be trained offline on a server and then deployed on an autonomous vehicle. The method of cognitive decision-making based on training samples during offline training is similar to the model application process. Content that is the same as or similar to the above embodiment can be referred to the above description and will not be repeated hereafter.

[0055] As an example, steps A10-A30 include: acquiring training sample in-vehicle and out-of-vehicle monitoring data, first automated driving control parameter annotation values, and first driving behavior interpretation annotation values, wherein the first automated driving control parameter annotation values ​​and the first driving behavior interpretation annotation values ​​can be annotated manually or by an annotation model, etc., and this embodiment does not limit this; inputting the training sample in-vehicle and out-of-vehicle monitoring data into the first cognitive decision model to be trained, to obtain at least one training sample first automated driving control parameter and the training sample first driving behavior interpretation corresponding to each of the training sample first automated driving control parameters; and comparing the difference between the training sample first automated driving control parameter and the first automated driving control parameter annotation values ​​with the training sample first... The differences between the driving behavior explanation and the first driving behavior explanation label value are aggregated to determine the total loss of the first cognitive decision model. Based on the total loss of the first cognitive decision model, it is determined whether the first cognitive decision model has converged. If the first cognitive decision model has converged, it can be determined that the offline training of the model is completed, and the offline trained first cognitive decision model is obtained. If the first cognitive decision model has not converged, the first cognitive decision model can be updated once based on the total loss of the first cognitive decision model using the gradient descent method, and the steps of obtaining the training sample vehicle interior and exterior monitoring data, the first autonomous driving control parameter label value, and the first driving behavior explanation label value are returned to perform the next round of offline training until the first cognitive decision model converges.

[0056] Step S20: When user question information is detected, a target first autonomous driving control parameter is determined from each of the first autonomous driving control parameters based on the user question information, a first driving behavior explanation corresponding to the target first autonomous driving control parameter generated by the first cognitive decision model is obtained, and the first driving behavior explanation is output.

[0057] In this embodiment, it should be noted that after the first decision layer outputs the first autonomous driving control parameters, each of the first autonomous driving control parameters can be input into the first interpretation layer. The first interpretation layer generates a first driving behavior explanation corresponding to each of the first autonomous driving control parameters and can also output each of the first driving behavior explanations, thereby achieving proactive interaction with the user and comprehensively resolving the user's doubts about driving behavior. If this method is adopted, after detecting user question information and determining the target first autonomous driving control parameter, the first driving behavior explanation corresponding to the target first autonomous driving control parameter can also be obtained. However, considering the computing power of the first device, it is also possible to detect user question information, determine the target first autonomous driving control parameter from each of the first autonomous driving control parameters based on the user question information, and then input the target first autonomous driving control parameter into the first interpretation layer. The first driving behavior explanation generated by the first interpretation layer corresponding to the target first autonomous driving control parameter is then output to specifically answer the user's doubts.

[0058] As an example, step S20 includes: during the operation of the autonomous vehicle, user operations can be continuously monitored. The user can ask questions through voice, typing, clicking option buttons, etc. When the first device detects the user's question information, it can extract the target first autonomous driving control parameters corresponding to the user's questioning driving behavior from the user's question information, and then obtain the first driving behavior explanation corresponding to the target first autonomous driving control parameters, and output the first driving behavior explanation to answer the user's questions in a targeted manner.

[0059] Optionally, the vehicle interior and exterior monitoring data includes current vehicle interior and exterior monitoring data at the current moment and historical vehicle interior and exterior monitoring data before the current moment; the first cognitive decision model includes a first cognitive layer, a first decision layer and a first interpretation layer;

[0060] The step of inputting the in-vehicle and out-of-vehicle monitoring data into the first cognitive decision model to obtain at least one first autonomous driving control parameter includes:

[0061] Step B10: Extract the first current cognitive layer features from the current vehicle interior and exterior monitoring data through the first cognitive layer, and extract the first historical cognitive layer features from the historical vehicle interior and exterior monitoring data;

[0062] Step B20: Input the first current cognitive layer features and the first historical cognitive layer features into the first decision layer to determine the first autonomous driving control parameters;

[0063] The step of outputting the first driving behavior explanation corresponding to the target first autonomous driving control parameter generated by the first cognitive decision model includes:

[0064] Step B30: Using a spatiotemporal attention mechanism in the first interpretation layer, the target decision features corresponding to the first autonomous driving control parameters are extracted based on the features of the first current cognitive layer, the features of the first historical cognitive layer, and the first autonomous driving control parameters.

[0065] Step B40: Generate a first driving behavior explanation based on the first autonomous driving control parameters and the target decision features.

[0066] In this embodiment, it's important to note that when users ask questions, it indicates that even with their current knowledge, they may still struggle to understand the driving behavior of the autonomous vehicle. In such cases, a comprehensive spatiotemporal analysis of the driving behavior, providing a detailed explanation, can better fill in any gaps in the user's understanding or those lost due to the passage of time. For example, if the vehicle brakes suddenly because a cat runs across the road, the user might not fully grasp the situation by the time they look up, leading to confusion, panic, or anxiety, resulting in a poor user experience. However, if the user is provided with the prior knowledge of the cat running across the road, they can more easily understand the braking action. After receiving a detailed explanation, users can also understand the decision-making logic of the cognitive decision-making model, allowing them to more accurately judge the correctness of the model's decisions. In this case, the suggestions are more accurate and more conducive to the cognitive decision-making model learning human thought processes rather than simply imitating the final driving behavior.

[0067] The vehicle's internal and external monitoring data includes current internal and external monitoring data at the current moment and historical internal and external monitoring data prior to the current moment. Based on the current internal and external monitoring data, some superficial influencing factors affecting cognitive decision-making can be quickly analyzed, such as traffic lights, intersections, and vehicles ahead. Combined with historical internal and external monitoring data prior to the current moment, the movement trends of moving objects can be analyzed, thereby predicting the trajectory of moving objects within a preset time range after the current moment, achieving risk warning, and improving driving safety.

[0068] As an example, steps B10-B40 include: inputting the current in-vehicle and out-of-vehicle monitoring data into the first cognitive layer of the first cognitive decision model, performing multi-level progressive feature processing such as feature extraction, feature fusion, feature encoding, and feature reasoning to obtain the first current cognitive layer features; inputting the historical in-vehicle and out-of-vehicle monitoring data into the first cognitive layer of the first cognitive decision model, performing multi-level progressive feature processing such as feature extraction, feature fusion, feature encoding, and feature reasoning to obtain the first historical cognitive layer features; then, inputting the first current cognitive layer features and the first historical cognitive layer features into the first decision layer, using an attention mechanism to capture the temporal and spatial correlation and dependency of each feature, performing further feature fusion, reasoning, and decision-making to determine the first autonomous driving control parameters; furthermore, the first current cognitive layer features, the first historical cognitive layer features, and the first autonomous driving control parameters can be input into the first interpretation layer, using a spatiotemporal attention mechanism to perform feature fusion, feature encoding, and feature reasoning analysis on the first current cognitive layer features, the first historical cognitive layer features, and the first autonomous driving control parameters to determine the target decision features corresponding to the first autonomous driving control parameters, and then aggregating the first autonomous driving control parameters and the target decision features to obtain the first driving behavior interpretation.

[0069] In one feasible approach, when there are multiple first autonomous driving control parameters, the target decision features corresponding to each first autonomous driving control parameter can be determined separately, and then each first autonomous driving control parameter and its corresponding target decision features can be aggregated to obtain a step-by-step explanation of driving behavior. Then, the step-by-step explanations of driving behavior can be aggregated in chronological order to form a first driving behavior explanation.

[0070] Step S30: If user feedback information generated in response to the explanation of the first driving behavior is detected, the first cognitive decision model is optimized based on the user feedback information.

[0071] In this embodiment, it should be noted that for long-tail boundary scenarios that the cognitive decision-making model has not experienced before, the cognitive decision-making model makes cognitive decisions based on conventional scenarios. The determined driving behavior may not be able to meet the actual needs of long-tail boundary scenarios well. In this case, user feedback information can be received and used to guide the cognitive decision-making model to learn human coping measures for long-tail boundary scenarios, thereby achieving further optimization of the first cognitive decision-making model.

[0072] As an example, step S30 includes: after outputting the first driving behavior explanation, user operations can be continuously detected. The user can provide feedback on the first driving behavior explanation through voice, typing, clicking option buttons, etc. When user feedback information generated in response to the first driving behavior explanation is detected, the label value corresponding to the output result of the first cognitive decision model can be extracted from the user feedback information. Then, by comparing the difference between the output result of the first cognitive decision model and the label value, the first cognitive decision model can be optimized.

[0073] Optionally, the autonomous driving optimization method involves a second device, on which a second cognitive decision model is deployed, and the model parameters of the second cognitive decision model and the first cognitive decision model are the same;

[0074] The step of optimizing the first cognitive decision-making model based on user feedback information generated in response to the explanation of the first driving behavior includes:

[0075] Step S31: When user feedback information generated in response to the explanation of the first driving behavior is detected, the user feedback information and the in-vehicle and out-of-vehicle monitoring data are sent to the second device so that the second device can optimize the second cognitive decision model based on the user feedback information and the in-vehicle and out-of-vehicle monitoring data, and send the first optimized model parameters of the optimized second cognitive decision model to the first device.

[0076] Step S32: Receive the first optimized model parameters of the optimized second cognitive decision model sent by the second device, and optimize the first cognitive decision model based on the first optimized model parameters.

[0077] In this embodiment, it should be noted that model training typically takes a long time and requires significant computing power. If the first cognitive decision-making model is directly trained on the first device, it may affect the normal operation of the first device. For example, it could cause the first cognitive decision-making model to lag due to insufficient computing resources, impacting the efficiency of autonomous driving decisions. Therefore, this embodiment deploys a second cognitive decision-making model with the same model parameters as the first cognitive decision-making model on the second device. The second cognitive decision-making model is trained using the computing power of the second device. After training, the first device can directly update the first cognitive decision-making model based on the model parameters of the second cognitive decision-making model trained on the second device, without needing to train the model on the first device itself. This allows the resources of the first device to be more effectively used for controlling the driving behavior of the autonomous vehicle. It should be noted that having the same model parameters for the second and / or first cognitive decision-making models means that the model parameters of the second and / or first cognitive decision-making models can be different before each optimization, during the optimization process, or after optimization but before synchronizing the model parameters.

[0078] As an example, steps S31-S32 include: after outputting the first driving behavior explanation, user operations can be continuously detected. The user can provide feedback on the first driving behavior explanation through voice, typing, clicking option buttons, etc. When user feedback information is detected regarding the first driving behavior explanation, the user feedback information and the in-vehicle and out-of-vehicle monitoring data are sent to a second device. The second device then inputs the user feedback information into the second cognitive decision model to obtain at least one second autonomous driving control parameter and a second driving behavior explanation corresponding to each second autonomous driving control parameter. Based on the differences between the user feedback information and the second autonomous driving control parameters, and the differences between the user feedback information and the second driving behavior explanation, the second cognitive decision model is optimized, and the first optimized model parameters of the optimized second cognitive decision model are sent to the first device. After receiving the first optimized model parameters of the optimized second cognitive decision model sent by the second device, the first device can directly update the model parameters of the first cognitive decision model to the first optimized model parameters, or it can first adjust the first optimized model parameters and then update the model parameters of the first cognitive decision model to the adjusted first optimized model parameters.

[0079] In one feasible approach, if the parameters of the first optimized model are adjusted first, and then the parameters of the first cognitive decision model are updated to the adjusted parameters of the first optimized model, the parameters of the optimized first cognitive decision model and the optimized second cognitive decision model may not be the same. Thus, in the next optimization, based on the same in-vehicle and out-of-vehicle monitoring data, the two models will make different cognitions and decisions, producing different outputs. Training the second cognitive decision model will not achieve the purpose of accurately training the first cognitive decision model. Therefore, in this case, after each optimization of the first cognitive decision model, the parameters of the first cognitive decision model need to be sent to the second device so that the second device can adjust the parameters of the second cognitive decision model to be the same as the optimized first cognitive decision model. If the parameters of the first cognitive decision model are directly updated to the parameters of the first optimized model, the parameters of the second cognitive decision model remain the same as those of the first cognitive decision model before and after optimization. Therefore, after optimizing the first cognitive decision model, subsequent model parameter synchronization is unnecessary.

[0080] Optionally, the step of optimizing the first cognitive decision-making model based on the first optimization model parameters includes:

[0081] Step C10: Obtain the phased adjustment parameters and the initial model parameters of the first cognitive decision-making model;

[0082] Step C20: Aggregate the first optimized model parameters and the initial model parameters according to the staged adjustment parameters to obtain the second optimized model parameters;

[0083] Step C30: Adjust the model parameters of the first cognitive decision-making model based on the second optimized model parameters.

[0084] In this embodiment, it should be noted that the scenarios experienced during vehicle operation are constantly changing, and it is rare for two scenarios to be exactly the same. Therefore, if the cognitive decision-making model is made to imitate human processing methods, the model's generalization ability will be poor. Furthermore, since it is difficult to find all scenarios for the cognitive decision-making model to imitate, the model's performance will be poor. Moreover, one or a few user feedback pieces of information may not be entirely accurate. When riding in an autonomous vehicle, users may not be able to pay attention to all features, so their analysis may also be somewhat one-sided. In this case, by introducing phased adjustment parameters, the model can be optimized by combining the results of its original learning and training with user feedback information. The phased adjustment parameter is used to adjust the degree of intervention of user feedback information on the first cognitive decision-making model. It can be the weight ratio of the initial model parameters and the first optimized model parameters. For example, the initial model parameters and the first optimized model parameters are weighted and summed according to the weight ratio to obtain the second optimized model parameters. Alternatively, the phased adjustment parameter can be an adjustment coefficient of the first optimized model parameters. For example, the first optimized model parameters are multiplied by the adjustment coefficient and then summed with the initial model parameters to obtain the second optimized model parameters. The specific settings can be adjusted according to actual circumstances, and this embodiment does not impose any limitations on this.

[0085] As an example, steps C10-C30 include: obtaining pre-set stage adjustment parameters and initial model parameters of the first cognitive decision-making model before optimization; aggregating the first optimized model parameters and the initial model parameters based on the stage adjustment parameters using aggregation methods such as summation, averaging, weighted summation, and weighted averaging to obtain second optimized model parameters; and then assigning the second optimized model parameters to the corresponding model parameters in the first cognitive decision-making model.

[0086] It should be noted that the number of parameters in the second optimized model can be less than or equal to the number of parameters in the first cognitive decision model. That is, all model parameters can be updated, or only some model parameters can be updated.

[0087] Optionally, the step of aggregating the first optimized model parameters and the initial model parameters according to the staged adjustment parameters to obtain the second optimized model parameters includes:

[0088] The staged adjustment parameters, the first optimized model parameters, and the initial model parameters are input into a model parameter aggregation algorithm to obtain the second optimized model parameters, wherein the obtained second optimized model parameters are:

[0089] θ′ t =αθ′ t-1 +(1-α)θt

[0090] Where, θ′ t For the second optimized model parameters, θ′ t-1 Here are the initial model parameters, α is the stage adjustment parameter, and θ is the parameter for each stage. t These are the parameters for the first optimized model.

[0091] In one feasible approach, refer to Figure 2 The training of the first cognitive decision-making model is divided into offline training and online training, ultimately achieving a continuous closed-loop training method for user interaction behavior habits throughout the entire lifecycle. Before practical application, data stored in a cloud database is first utilized. This data mainly comes from images, videos, radar point clouds, positioning information, and vehicle chassis parameters collected from vehicles, experimental vehicles, and mass-produced vehicles. The collected data is uploaded to the cloud database according to a standard format. The cloud database cleans, filters, and performs time-series fusion of multimodal data, encodes the features of the multimodal data, and transmits it to the first cognitive decision-making model for understanding, thus achieving offline training. After offline training is completed, the first cognitive decision-making model can handle most scenarios well and meet the requirements for application. During application, a second cognitive decision-making model is introduced to learn user feedback information and human coping logic in long-tail boundary scenarios. The learned model parameters are then shared with the first cognitive decision-making model. By sharing the parameters learned from the second cognitive decision-making model, the first cognitive decision-making model can quickly learn human coping logic in long-tail boundary scenarios without affecting the normal operation of the autonomous vehicle. This enables online training during model application, allowing the first cognitive decision-making model to continuously iterate and evolve throughout its entire lifecycle.

[0092] In this embodiment, the autonomous driving optimization method is applied to a first device, on which a first cognitive decision-making model is deployed. First, by acquiring in-vehicle and out-of-vehicle monitoring data, this data is input into the first cognitive decision-making model to obtain at least one first autonomous driving control parameter, thus realizing the practical application of the first cognitive decision-making model. Then, upon detecting user question information, a target first autonomous driving control parameter is determined from each of the first autonomous driving control parameters based on the user question information. A first driving behavior explanation corresponding to the target first autonomous driving control parameter generated by the first cognitive decision-making model is obtained and output. This achieves the purpose of interacting with the user, explaining the driving behavior, and dispelling user doubts when the user has difficulty understanding it. Furthermore, upon detecting user feedback information generated regarding the first driving behavior explanation, the first cognitive decision-making model is optimized based on the user feedback information, realizing online training of the first cognitive decision-making model in practical applications. On the one hand, this makes the cognition and decision-making of the first cognitive decision-making model closer to human thinking logic, thereby making the driving behavior decided by the first cognitive model easier for users to understand. Therefore, this approach overcomes the poor interpretability of end-to-end perception and decision-making architectures. This is because, when controlling a vehicle to perform sudden actions, users may experience confusion, panic, and anxiety due to the inability to understand the driving behavior, resulting in a poor user experience. The approach improves the interpretability of the end-to-end perception and decision-making architecture, reducing negative emotions such as confusion, panic, and anxiety caused by the unknown, thus enhancing the user experience. On the other hand, while a limited number of training samples can efficiently train a first cognitive decision-making model that meets the needs of most scenarios, it is usually difficult to comprehensively cover all scenarios. Therefore, when encountering new boundary scenarios in practical applications, interaction with users can learn their cognitive decision-making methods, enabling further optimization of the first cognitive decision-making model and effectively solving the long-tail problem caused by the inability of training samples to cover all scenarios.

[0093] Example 2

[0094] Furthermore, a second embodiment of this application proposes an autonomous driving optimization method, which is applied to a second device on which a second cognitive decision-making model is deployed. Contents identical or similar to those in the above embodiments can be referred to the above description and will not be repeated hereafter. Based on this, refer to... Figure 3 The autonomous driving optimization method includes the following steps:

[0095] Step D10: Initialize the second cognitive decision model so that the model parameters of the second cognitive decision model are the same as the model parameters of the first cognitive decision model deployed on the first device;

[0096] In this embodiment, it should be noted that during the application of the cognitive decision-making model, if inaccurate cognitive decisions are encountered and the model is not optimized in a timely manner, the autonomous vehicle will make more inaccurate decisions, affecting user experience and even posing safety risks. Model training typically takes a long time and requires significant computing power. If the first cognitive decision-making model is trained directly on the first device, it may affect the normal use of the first device, for example, causing the first cognitive decision-making model to lag due to insufficient computing resources, thus affecting the efficiency of autonomous driving decision-making.

[0097] As an example, step D10 includes: before optimization, obtaining the model parameters of the first cognitive decision model, and initializing the second cognitive decision model based on the model parameters of the first cognitive decision model, so that the model parameters of the second cognitive decision model are the same as the model parameters of the first cognitive decision model deployed on the first device.

[0098] Step D20: Receive user feedback information and vehicle / interior monitoring data sent by the first device;

[0099] Step D30: Input the in-vehicle and out-of-vehicle monitoring data into the second cognitive decision model to obtain at least one second autonomous driving control parameter and the second driving behavior explanation corresponding to each second autonomous driving control parameter.

[0100] As an example, steps D20-D30 include: after the autonomous vehicle receives user feedback information, it can use the in-vehicle and out-of-vehicle monitoring data in the current scene as training sample data, and send the user feedback information as labeled values ​​to the second device for model training. After receiving the user feedback information and in-vehicle and out-of-vehicle monitoring data sent by the first device, the second device inputs the in-vehicle and out-of-vehicle monitoring data into the second cognitive decision model, performs data processing such as cognition, decision-making, and analysis, and determines the second autonomous driving control parameters of the autonomous vehicle at the current moment or for a period of time after the current moment, as well as the second driving behavior interpretation corresponding to each of the second autonomous driving control parameters.

[0101] In one feasible embodiment, the second cognitive decision-making model includes a second cognitive layer, a second decision layer, and a second interpretation layer. The step of inputting the in-vehicle and out-of-vehicle monitoring data into the second cognitive decision-making model to obtain at least one second autonomous driving control parameter and the second driving behavior interpretation corresponding to each second autonomous driving control parameter may include: inputting the in-vehicle and out-of-vehicle monitoring data into the second cognitive layer of the second cognitive decision-making model, performing multi-level progressive feature processing such as feature extraction, feature fusion, feature encoding, and feature reasoning to obtain second cognitive layer features; then, inputting the second cognitive layer features into the first decision layer, using an attention mechanism to capture the temporal and spatial correlation and dependency of each feature, performing further feature fusion, reasoning, and decision-making to determine the second autonomous driving control parameter; then, inputting the second cognitive layer features and the second autonomous driving control parameter into the second interpretation layer, using a spatiotemporal attention mechanism to perform feature fusion, feature encoding, and feature reasoning analysis on the second cognitive layer features and the second autonomous driving control parameter to determine the target decision feature corresponding to the second autonomous driving control parameter, and then aggregating the first autonomous driving control parameter and the target decision feature to obtain the second driving behavior interpretation.

[0102] Step D40: Optimize the second cognitive decision model based on the differences between the user feedback information and the second autonomous driving control parameters, and the differences between the user feedback information and the second driving behavior explanation;

[0103] As an example, step D40 includes: aggregating the differences between the user feedback information and the second autonomous driving control parameters and the differences between the user feedback information and the second driving behavior explanation to obtain the total loss of the second cognitive decision model; and using the gradient descent method to optimize some or all of the model parameters in the second cognitive decision model based on the total loss of the second cognitive decision model.

[0104] Optionally, the step of optimizing the second cognitive decision-making model based on the difference between the user feedback information and the second autonomous driving control parameters, and the difference between the user feedback information and the second driving behavior explanation, includes:

[0105] Step D41: Based on the user feedback information, the second autonomous driving control parameters, and the second driving behavior explanation, determine at least one sub-model to be optimized from the second cognitive decision model;

[0106] Step D42: Optimize each of the sub-models to be optimized based on the differences between the user feedback information and the second autonomous driving control parameters, and the differences between the user feedback information and the second driving behavior explanation.

[0107] In this embodiment, it should be noted that training the entire cognitive decision-making model takes a long time and consumes a lot of resources. Therefore, based on user feedback, some sub-models with lower accuracy can be identified and optimized. This can significantly improve model performance, increase model training efficiency, and save computing resources.

[0108] As an example, steps D41-D42 include: determining at least one sub-model to be optimized based on the difference between the user feedback information and the second driving behavior explanation, and the difference between the user feedback information and the second autonomous driving control parameters; then aggregating the difference between the user feedback information and the second autonomous driving control parameters, and the difference between the user feedback information and the second driving behavior explanation, to obtain the total loss of the second cognitive decision model; and optimizing the sub-model to be optimized using gradient descent based on the total loss of the second cognitive decision model.

[0109] Step D50: Send the first optimized model parameters of the optimized second cognitive decision model to the first device, so that the first device can optimize its own deployed first cognitive decision model based on the first optimized model parameters.

[0110] In this embodiment, it should be noted that the first optimized model parameters refer to the model parameters that have been optimized during the model training process. That is, if all the model parameters of the second cognitive decision model have been optimized, then all the model parameters of the optimized second cognitive decision model can be determined as the first optimized model parameters; if some of the target model parameters in the model parameters of the second cognitive decision model have been optimized, then the target model parameters of the optimized second cognitive decision model can be determined as the first optimized model parameters.

[0111] As an example, step D50 includes: determining optimized first optimized model parameters from the model parameters of the optimized second cognitive decision model, and sending the first optimized model parameters to the first device so that the first device can optimize its deployed first cognitive decision model based on the first optimized model parameters.

[0112] In this embodiment, a second cognitive decision-making model with the same model parameters as the first cognitive decision-making model is deployed on a second device. The second cognitive decision-making model is trained using the computing power of the second device. After training, the first device can directly update the first cognitive decision-making model based on the model parameters of the trained second cognitive decision-making model. This eliminates the need for model training on the first device, allowing its resources to be more focused on controlling the driving behavior of the autonomous vehicle, ensuring the accuracy and safety of autonomous driving. Simultaneously, both model training and autonomous driving have sufficient resources available, improving the efficiency of both and enhancing the overall efficiency of autonomous driving optimization.

[0113] Example 3

[0114] Furthermore, embodiments of this application also provide an autonomous driving optimization device, referring to... Figure 4 The autonomous driving optimization device is equipped with a first cognitive decision-making model, and the autonomous driving optimization device includes:

[0115] The acquisition module 10 is used to acquire in-vehicle and out-of-vehicle monitoring data, input the in-vehicle and out-of-vehicle monitoring data into the first cognitive decision model, and obtain at least one first autonomous driving control parameter;

[0116] Output module 20 is used to determine a target first autonomous driving control parameter from each of the first autonomous driving control parameters based on the user question information when the user question information is detected, obtain a first driving behavior explanation corresponding to the target first autonomous driving control parameter generated by the first cognitive decision model, and output the first driving behavior explanation.

[0117] The first optimization module 30 is used to optimize the first cognitive decision model based on the user feedback information when user feedback information generated in response to the explanation of the first driving behavior is detected.

[0118] Optionally, the autonomous driving optimization method involves a second device, on which a second cognitive decision model is deployed, and the model parameters of the second cognitive decision model and the first cognitive decision model are the same;

[0119] The first optimization module 30 is further configured to:

[0120] Upon detecting user feedback information generated in response to the explanation of the first driving behavior, the user feedback information and the in-vehicle and out-of-vehicle monitoring data are sent to the second device, so that the second device can optimize the second cognitive decision model based on the user feedback information and the in-vehicle and out-of-vehicle monitoring data, and send the first optimized model parameters of the optimized second cognitive decision model to the first device;

[0121] The system receives the first optimized model parameters of the optimized second cognitive decision model sent by the second device, and optimizes the first cognitive decision model based on the first optimized model parameters.

[0122] Optionally, the first optimization module 30 is further configured to:

[0123] Obtain the phased adjustment parameters and the initial model parameters of the first cognitive decision-making model;

[0124] The first optimized model parameters and the initial model parameters are aggregated according to the phased adjustment parameters to obtain the second optimized model parameters;

[0125] The model parameters of the first cognitive decision-making model are adjusted based on the parameters of the second optimized model.

[0126] Optionally, the first optimization module 30 is further configured to:

[0127] The staged adjustment parameters, the first optimized model parameters, and the initial model parameters are input into a model parameter aggregation algorithm to obtain the second optimized model parameters, wherein the obtained second optimized model parameters are:

[0128] θ′ t =αθ′ t-1 +(1-α)θ t

[0129] Where, θ′ t For the second optimized model parameters, θ′ t-1 Here are the initial model parameters, α is the stage adjustment parameter, and θ is the parameter for each stage. t These are the parameters for the first optimized model.

[0130] Optionally, the autonomous driving optimization device further includes an offline training module, which is used before the operation of acquiring in-vehicle and out-of-vehicle monitoring data to:

[0131] Acquire training sample vehicle interior and exterior monitoring data, first autonomous driving control parameter annotation values, and first driving behavior interpretation annotation values;

[0132] The training sample vehicle interior and exterior monitoring data are input into the first cognitive decision model to be trained to obtain at least one training sample first autonomous driving control parameter and the training sample first driving behavior explanation corresponding to each training sample first autonomous driving control parameter.

[0133] Based on the differences between the first autonomous driving control parameters and the labeled values ​​of the first autonomous driving control parameters in the training samples, and the differences between the first driving behavior explanations and the labeled values ​​of the first driving behavior explanations in the training samples, the first cognitive decision model to be trained is iteratively updated.

[0134] Optionally, the vehicle interior and exterior monitoring data includes current vehicle interior and exterior monitoring data at the current moment and historical vehicle interior and exterior monitoring data before the current moment; the first cognitive decision model includes a first cognitive layer, a first decision layer and a first interpretation layer;

[0135] The offline training module is also used for:

[0136] The first cognitive layer extracts the first current cognitive layer features from the current vehicle interior and exterior monitoring data, and extracts the first historical cognitive layer features from the historical vehicle interior and exterior monitoring data;

[0137] The first current cognitive layer features and the first historical cognitive layer features are input into the first decision layer to determine the first autonomous driving control parameters;

[0138] The step of outputting the first driving behavior explanation corresponding to the target first autonomous driving control parameter generated by the first cognitive decision model includes:

[0139] By employing a spatiotemporal attention mechanism in the first interpretation layer, target decision features corresponding to the first autonomous driving control parameters are extracted based on the features of the first current cognitive layer, the features of the first historical cognitive layer, and the first autonomous driving control parameters.

[0140] A first driving behavior explanation is generated based on the first autonomous driving control parameters and the target decision characteristics.

[0141] The autonomous driving optimization device provided by this invention employs the autonomous driving optimization method in the above embodiments, solving the technical problem of poor interpretability of end-to-end perception decision-making schemes in related technologies. Compared with related technologies, the autonomous driving optimization device provided by this invention has the same advantages as the autonomous driving optimization method provided in the above embodiments, and other technical features in this autonomous driving optimization device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0142] Example 4

[0143] Furthermore, embodiments of this application also provide an autonomous driving optimization device, on which a second cognitive decision-making model is deployed, the autonomous driving optimization device comprising:

[0144] An initialization module is used to initialize the second cognitive decision model so that the model parameters of the second cognitive decision model are the same as the model parameters of the first cognitive decision model deployed on the first device.

[0145] The receiving module is used to receive user feedback information and vehicle interior and exterior monitoring data sent by the first device;

[0146] The cognitive decision module is used to input the in-vehicle and out-of-vehicle monitoring data into the second cognitive decision model to obtain at least one second autonomous driving control parameter and the second driving behavior explanation corresponding to each of the second autonomous driving control parameters.

[0147] The second optimization module is used to optimize the second cognitive decision model based on the difference between the user feedback information and the second autonomous driving control parameters, and the difference between the user feedback information and the second driving behavior explanation.

[0148] The sending module sends the first optimized model parameters of the optimized second cognitive decision model to the first device, so that the first device can optimize its own deployed first cognitive decision model based on the first optimized model parameters.

[0149] Optionally, the second optimization module is further configured to:

[0150] Based on the user feedback information, the second autonomous driving control parameters, and the second driving behavior explanation, at least one sub-model to be optimized is determined from the second cognitive decision model;

[0151] Based on the differences between the user feedback information and the second autonomous driving control parameters, and the differences between the user feedback information and the second driving behavior explanation, each of the sub-models to be optimized is optimized.

[0152] The autonomous driving optimization device provided by this invention employs the autonomous driving optimization method in the above embodiments, solving the technical problem of poor interpretability of end-to-end perception decision-making schemes in related technologies. Compared with related technologies, the autonomous driving optimization device provided by this invention has the same advantages as the autonomous driving optimization method provided in the above embodiments, and other technical features in this autonomous driving optimization device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0153] Example 5

[0154] Furthermore, embodiments of the present invention provide an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the autonomous driving optimization method in the above embodiments.

[0155] The following is for reference. Figure 5 The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as Bluetooth headsets, mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0156] like Figure 5 As shown, an electronic device may include a processing unit (such as a central processing unit, graphics processing unit, etc.) that can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) or a program loaded from a storage device into random access memory (RAM). The RAM also stores various programs and arrays required for the operation of the electronic device. The processing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0157] Typically, the following systems can be connected to the I / O interface: input devices including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication devices. Communication devices allow electronic devices to communicate wirelessly or wiredly with other devices to exchange arrays. Although electronic devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0158] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the functions defined above in the methods of embodiments of this disclosure.

[0159] The electronic device provided by this invention employs the autonomous driving optimization method in the above embodiments, solving the technical problem of poor interpretability of end-to-end perception decision-making schemes in related technologies. Compared with related technologies, the electronic device provided by this invention has the same advantages as the autonomous driving optimization method provided in the above embodiments, and other technical features in this electronic device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0160] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0161] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0162] Example 6

[0163] Furthermore, this embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, which are used to execute the autonomous driving optimization method in the above embodiment.

[0164] The computer-readable storage medium provided in this embodiment of the invention may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0165] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0166] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device causes the electronic device to: acquire in-vehicle and out-of-vehicle monitoring data and scene labels, and acquire the target human driving experience scene cognitive features corresponding to the scene labels; extract risk features from the in-vehicle and out-of-vehicle monitoring data through a cognitive layer; and iteratively optimize the cognitive layer based on the risk features and the target human driving experience scene cognitive features.

[0167] Alternatively, the aforementioned computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to: acquire in-vehicle and out-of-vehicle monitoring data to be predicted; extract in-vehicle and out-of-vehicle risk features from the monitoring data through a cognitive layer, wherein the in-vehicle and out-of-vehicle risk features include in-vehicle scene cognitive features; and input the in-vehicle and out-of-vehicle risk features into a decision layer to determine first autonomous driving control parameters.

[0168] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0170] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0171] The computer-readable storage medium provided by this invention stores computer-readable program instructions for executing the above-described autonomous driving optimization method, thus solving the technical problem of poor interpretability of end-to-end perception decision-making schemes in related technologies. Compared with related technologies, the advantages of the computer-readable storage medium provided in this invention are the same as those of the autonomous driving optimization method provided in the above embodiments, and will not be repeated here.

[0172] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. An autonomous driving optimization method, characterized in that, The autonomous driving optimization method is applied to a first device, on which a first cognitive decision-making model is deployed, and the autonomous driving optimization method includes the following steps: Acquire in-vehicle and out-of-vehicle monitoring data, input the in-vehicle and out-of-vehicle monitoring data into the first cognitive decision model, and obtain at least one first autonomous driving control parameter; Upon detecting user question information, a target first autonomous driving control parameter is determined from each of the first autonomous driving control parameters based on the user question information. A first driving behavior explanation corresponding to the target first autonomous driving control parameter generated by the first cognitive decision model is obtained, and the first driving behavior explanation is output. Upon detecting user feedback information generated in response to the interpretation of the first driving behavior, the user feedback information and the in-vehicle and out-of-vehicle monitoring data are sent to a second device, so that the second device can optimize the second cognitive decision model based on the user feedback information and the in-vehicle and out-of-vehicle monitoring data, and send the first optimized model parameters of the optimized second cognitive decision model to the first device; wherein, the second device is equipped with a second cognitive decision model, and the model parameters of the second cognitive decision model and the first cognitive decision model are the same; The system receives the first optimized model parameters of the optimized second cognitive decision model sent by the second device, and optimizes the first cognitive decision model based on the first optimized model parameters.

2. The autonomous driving optimization method as described in claim 1, characterized in that, The step of optimizing the first cognitive decision-making model based on the first optimization model parameters includes: Obtain the phased adjustment parameters and the initial model parameters of the first cognitive decision-making model; The first optimized model parameters and the initial model parameters are aggregated according to the phased adjustment parameters to obtain the second optimized model parameters; The model parameters of the first cognitive decision-making model are adjusted based on the parameters of the second optimized model.

3. The autonomous driving optimization method as described in claim 2, characterized in that, The step of aggregating the first optimized model parameters and the initial model parameters according to the stage adjustment parameters to obtain the second optimized model parameters includes: The staged adjustment parameters, the first optimized model parameters, and the initial model parameters are input into a model parameter aggregation algorithm to obtain the second optimized model parameters, wherein the obtained second optimized model parameters are: in, For the second optimized model parameters, Here are the initial model parameters, and α is the stage-wise adjustment parameter. These are the parameters for the first optimized model.

4. The autonomous driving optimization method as described in claim 1, characterized in that, Before the step of acquiring in-vehicle and out-of-vehicle monitoring data, the following steps are also included: Acquire training sample vehicle interior and exterior monitoring data, first autonomous driving control parameter annotation values, and first driving behavior interpretation annotation values; The training sample vehicle interior and exterior monitoring data are input into the first cognitive decision model to be trained to obtain at least one training sample first autonomous driving control parameter and the training sample first driving behavior explanation corresponding to each training sample first autonomous driving control parameter. Based on the differences between the first autonomous driving control parameters and the labeled values ​​of the first autonomous driving control parameters in the training samples, and the differences between the first driving behavior explanations and the labeled values ​​of the first driving behavior explanations in the training samples, the first cognitive decision model to be trained is iteratively updated.

5. The autonomous driving optimization method as described in claim 1, characterized in that, The vehicle interior and exterior monitoring data includes the current vehicle interior and exterior monitoring data at the current moment and the historical vehicle interior and exterior monitoring data before the current moment; the first cognitive decision model includes a first cognitive layer, a first decision layer and a first interpretation layer; The step of inputting the in-vehicle and out-of-vehicle monitoring data into the first cognitive decision model to obtain at least one first autonomous driving control parameter includes: The first cognitive layer extracts the first current cognitive layer features from the current vehicle interior and exterior monitoring data, and extracts the first historical cognitive layer features from the historical vehicle interior and exterior monitoring data; The first current cognitive layer features and the first historical cognitive layer features are input into the first decision layer to determine the first autonomous driving control parameters; The step of outputting the first driving behavior explanation corresponding to the target first autonomous driving control parameter generated by the first cognitive decision model includes: By employing a spatiotemporal attention mechanism in the first interpretation layer, target decision features corresponding to the first autonomous driving control parameters are extracted based on the features of the first current cognitive layer, the features of the first historical cognitive layer, and the first autonomous driving control parameters. A first driving behavior explanation is generated based on the first autonomous driving control parameters and the target decision characteristics.

6. An autonomous driving optimization method, characterized in that, The autonomous driving optimization method is applied to a second device, on which a second cognitive decision-making model is deployed. The autonomous driving optimization method includes the following steps: Initialize the second cognitive decision model so that the model parameters of the second cognitive decision model are the same as the model parameters of the first cognitive decision model deployed on the first device; Receive user feedback information and vehicle interior / exterior monitoring data sent by the first device; The in-vehicle and out-of-vehicle monitoring data are input into the second cognitive decision model to obtain at least one second autonomous driving control parameter and the second driving behavior explanation corresponding to each second autonomous driving control parameter. Based on the differences between the user feedback information and the second autonomous driving control parameters, and the differences between the user feedback information and the second driving behavior explanation, the second cognitive decision model is optimized. The first optimized model parameters of the optimized second cognitive decision model are sent to the first device so that the first device can optimize its own deployed first cognitive decision model based on the first optimized model parameters.

7. The autonomous driving optimization method as described in claim 6, characterized in that, The step of optimizing the second cognitive decision-making model based on the difference between the user feedback information and the second autonomous driving control parameters, and the difference between the user feedback information and the second driving behavior explanation, includes: Based on the user feedback information, the second autonomous driving control parameters, and the second driving behavior explanation, at least one sub-model to be optimized is determined from the second cognitive decision model; Based on the differences between the user feedback information and the second autonomous driving control parameters, and the differences between the user feedback information and the second driving behavior explanation, each of the sub-models to be optimized is optimized.

8. An autonomous driving optimization device, characterized in that, The autonomous driving optimization device is equipped with a first cognitive decision-making model, and the autonomous driving optimization device includes: The acquisition module is used to acquire in-vehicle and out-of-vehicle monitoring data, input the in-vehicle and out-of-vehicle monitoring data into the first cognitive decision model, and obtain at least one first autonomous driving control parameter; The output module is used to determine the target first autonomous driving control parameter from each of the first autonomous driving control parameters based on the user question information when the user question information is detected, obtain the first driving behavior explanation corresponding to the target first autonomous driving control parameter generated by the first cognitive decision model, and output the first driving behavior explanation. A first optimization module is configured to, upon detecting user feedback information generated in response to the explanation of the first driving behavior, send the user feedback information and the in-vehicle and out-of-vehicle monitoring data to a second device, so that the second device can optimize the second cognitive decision model based on the user feedback information and the in-vehicle and out-of-vehicle monitoring data, and send the first optimized model parameters of the optimized second cognitive decision model to the first device; wherein, the second device is equipped with a second cognitive decision model, and the model parameters of the second cognitive decision model and the first cognitive decision model are the same; The system receives the first optimized model parameters of the optimized second cognitive decision model sent by the second device, and optimizes the first cognitive decision model based on the first optimized model parameters.

9. An autonomous driving optimization device, characterized in that, The autonomous driving optimization device is equipped with a second cognitive decision-making model, and the autonomous driving optimization device includes: An initialization module is used to initialize the second cognitive decision model so that the model parameters of the second cognitive decision model are the same as the model parameters of the first cognitive decision model deployed on the first device. The receiving module is used to receive user feedback information and vehicle interior and exterior monitoring data sent by the first device; The cognitive decision module is used to input the in-vehicle and out-of-vehicle monitoring data into the second cognitive decision model to obtain at least one second autonomous driving control parameter and the second driving behavior explanation corresponding to each of the second autonomous driving control parameters. The second optimization module is used to optimize the second cognitive decision model based on the difference between the user feedback information and the second autonomous driving control parameters, and the difference between the user feedback information and the second driving behavior explanation. The sending module sends the first optimized model parameters of the optimized second cognitive decision model to the first device, so that the first device can optimize its own deployed first cognitive decision model based on the first optimized model parameters.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by at least one of the processors to enable at least one of the processors to perform the steps of the autonomous driving optimization method as described in any one of claims 1 to 5 or 6 to 7.

11. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and the computer-readable storage medium stores a program for implementing the autonomous driving optimization method. The program for implementing the autonomous driving optimization method is executed by a processor to implement the steps of the autonomous driving optimization method as described in any one of claims 1 to 5 or 6 to 7.

Images