Automatic driving method and device, storage medium and vehicle-mounted terminal

Abstract

The embodiment of the invention provides an automatic driving method and device, a storage medium and a vehicle-mounted terminal. The method comprises the steps of obtaining perception data of a vehicle; performing feature extraction on the perception data to obtain target features corresponding to the perception data; performing control signal prediction on the target feature to obtain a control signal; determining a causal effect difference value corresponding to each influence factor, wherein the causal effect difference value represents the influence degree of the influence factor on the control signal; determining a control decision reason corresponding to the control signal according to the causal effect difference value; when the vehicle is controlled to automatically drive according to the control signal, the control decision reason is played; the control decision reason corresponding to the control signal is determined by determining the causal effect difference value corresponding to each influence factor, a user can be helped to understand the decision process of the model, and the decision reason of the control strategy is fed back to the user by analyzing the consequences of different decisions, so that the model has interpretability, and the user experience is improved. And the trust of the user to the automatic driving system is improved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle autonomous driving technology, and in particular to an autonomous driving method, device, storage medium and vehicle-mounted terminal. Background Art

[0002] At present, the level of intelligence has become a core factor for consumers to buy cars, and consumers are increasingly interested in autonomous driving functions. Intelligence has become the main area of ​​competition for new energy vehicles. In order to meet consumer needs and improve user experience and trust, autonomous driving systems need to further enhance their explainability and transparency.

[0003] However, in related technologies, solutions that improve the accuracy of trajectory prediction and the accuracy of prediction results by combining information from multiple visual sensors or by combining a large visual model with a large natural language model are all black box models. Users can only observe the input and output of the model and cannot directly understand the decision-making process and logic within the model. The model is not interpretable, which reduces users' trust in the autonomous driving system and reduces the user experience. Summary of the invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide an automatic driving method, device, storage medium and vehicle-mounted terminal that overcome the above problems or at least partially solve the above problems.

[0005] According to a first aspect of the present invention, there is provided an automatic driving method, the method comprising: Acquire the vehicle’s perception data; Extracting features from the perception data to obtain target features corresponding to the perception data; Performing control signal prediction on the target feature to obtain a control signal, wherein the control signal includes at least one influencing factor; Determine a causal effect difference corresponding to each influencing factor, wherein the causal effect difference represents the degree of influence of the influencing factor on the control signal; Determining a control decision cause corresponding to the control signal according to the causal effect difference; When the vehicle is controlled to automatically drive according to the control signal, the reason for the control decision is played.

[0006] Optionally, the performing feature extraction on the perception data to obtain target features corresponding to the perception data includes: Performing feature extraction on the perception data to obtain image features corresponding to the perception data; The image features are screened according to importance weights to obtain target features.

[0007] Optionally, the step of screening the image features according to importance weights to obtain target features includes: Inputting the image features into a target learner, and determining the importance weights of the image features through a spatial attention mechanism; Performing weighted aggregation on the input image features according to the importance weights to obtain a first image feature; Perform feature conversion on the first image feature to obtain the target feature.

[0008] Optionally, performing feature conversion on the first image feature to obtain the target feature includes: Determining a first feature conversion relationship according to the perception data of the vehicle, where the first feature conversion relationship represents a mapping relationship of converting the first image feature into a first overhead perspective feature; Converting the first image feature into the first top-view feature according to the first feature conversion relationship; When the perception data includes the first image features of multiple viewing angles, performing feature fusion on first overhead viewing angle features corresponding to the first image features of each viewing angle; The fused first top-down perspective feature is used as the target feature.

[0009] Optionally, the determining of the causal effect difference corresponding to each influencing factor, wherein the causal effect difference represents the degree of influence of the influencing factor on the control signal, includes: Determine a set of influencing factors of the control signal according to the sensing data, wherein the set of influencing factors includes at least one influencing factor; Selecting an influencing factor from the influencing factor set for intervention and fixation; In the case where an influencing factor is selected for intervention and fixation, the causal effect difference corresponding to the influencing factor is determined.

[0010] Optionally, in the case of selecting an influencing factor for intervention and fixation, determining the causal effect difference corresponding to the influencing factor includes: Selecting an influencing factor from the influencing factor set to intervene in a fixed condition and setting the conditional state as a first conditional state; Setting the conditional state under the original conditions of each influencing factor to the second conditional state; Calculating a first conditional probability of predicting a control signal under the first condition and a second conditional probability of predicting a control signal under the second condition; The difference between the second conditional probability and the first conditional probability is determined as the causal effect difference.

[0011] Optionally, determining the control decision reason corresponding to the control signal according to the causal effect difference includes: Compare the differences in causal effects corresponding to different influencing factors when different influencing factors are fixed for intervention; The influencing factors whose causal effect difference is greater than a preset causal effect difference threshold are used as the control decision reasons.

[0012] According to a second aspect of the present invention, there is provided an automatic driving device, the device comprising: A data acquisition module, used to acquire the perception data of the vehicle; A target feature determination module, used to extract features from the perception data to obtain target features corresponding to the perception data; A prediction module, configured to predict a control signal for the target feature to obtain a control signal, wherein the control signal includes at least one influencing factor; An influence degree determination module, used to determine a causal effect difference corresponding to each influencing factor, wherein the causal effect difference represents the influence degree of the influencing factor on the control signal; A decision reason determination module, used to determine the control decision reason corresponding to the control signal according to the causal effect difference; The output module is used to play the reason for the control decision when controlling the automatic driving of the vehicle according to the control signal.

[0013] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned automatic driving method are implemented.

[0014] According to the fourth aspect of the present invention, a vehicle-mounted terminal is provided, which includes: a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute the computer program stored in the memory to implement the steps of the above-mentioned automatic driving method.

[0015] The embodiments of the present invention include the following advantages: In an embodiment of the present invention, perception data of a vehicle is acquired; feature extraction is performed on the perception data to obtain target features corresponding to the perception data; control signal prediction is performed on the target features to obtain a control signal, wherein the control signal includes at least one influencing factor; the causal effect difference corresponding to each influencing factor is determined, wherein the causal effect difference represents the degree of influence of the influencing factor on the control signal; the control decision reason corresponding to the control signal is determined based on the causal effect difference; and the control decision reason is played when the vehicle is controlled to drive automatically according to the control signal; determining the control decision reason corresponding to the control signal by determining the causal effect difference corresponding to each influencing factor can help users understand the decision-making process of the model, and by analyzing the consequences of different decisions, the decision reasons of the control strategy are fed back to the user, so that the model has explainability and the user's trust in the automatic driving system is enhanced.

[0016] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 is a flowchart of steps of an embodiment of an automatic driving method of the present invention; Figure 2 is a schematic diagram of a scenario of an automatic driving method provided by an embodiment of the present invention; Figure 3 is a flowchart of steps for outputting control decision reasons provided by an embodiment of the present invention; Figure 4 is a flowchart of the steps of feature conversion provided by an embodiment of the present invention; Figure 5 is a schematic diagram of a scenario of counterfactual intervention provided by an embodiment of the present invention; Figure 6 It is a structural block diagram of an embodiment of an automatic driving device of the present invention. DETAILED DESCRIPTION

[0019] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0020] The terms "first", "second", etc. in the specification and claims of the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0021] In the following, in conjunction with the accompanying drawings, an automatic driving method, device, storage medium and vehicle-mounted terminal provided by an embodiment of the present invention are described in detail through specific embodiments and their application scenarios.

[0022] With the development of intelligent technology, the level of intelligence has become a core factor for consumers when purchasing cars. Consumers are also increasingly interested in autonomous driving functions, and intelligence has become the main area of ​​competition for new energy vehicles.

[0023] Traditional modular autonomous driving has problems such as error accumulation and information loss, and the introduction of end-to-end autonomous driving solutions provides a new path to solve these problems. In related technologies, the accuracy of prediction results is improved by combining information from multiple visual sensors; or the accuracy of trajectory prediction is improved by combining a large visual model with a large natural language model. However, these solutions are all black box models that are not interpretable, which will make users distrust the autonomous driving system. Therefore, in order to meet consumer needs and improve user experience and trust, the autonomous driving system needs to further enhance its interpretability and transparency.

[0024] Reference Figure 1 , shows a flowchart of a step of an embodiment of an automatic driving method of the present invention, which may specifically include the following steps: Step 101, obtaining the perception data of the vehicle; During the driving process of the vehicle, sensors such as cameras, radars or lidars installed on the vehicle can be used to monitor and record the vehicle's surrounding environment, the vehicle's own status and driving behavior in real time as perception data. These perception data are crucial for realizing autonomous driving, improving driving safety and optimizing driving experience.

[0025] Step 102, extracting features from the perception data to obtain target features corresponding to the perception data; After the perception data is acquired, features of the video information in the perception data may be extracted to obtain video features corresponding to the video information.

[0026] Specifically, the video information collected by the sensor can be processed by Video Swin Transformer, and video features corresponding to the video information can be output, and the video features can be used as target features.

[0027] Video Swin Transformer is a video processing model based on the Transformer architecture, which is specifically used to extract high-dimensional features from video data. It combines the spatial attention mechanism of Swin Transformer and the temporal attention mechanism in video processing, and can capture the complex features of videos in both temporal and spatial dimensions.

[0028] The video features extracted by Video Swin Transformer can enable the autonomous driving system to perceive the surrounding environment more comprehensively and accurately, make more intelligent and safe driving decisions, and improve the perception ability and stability of operation in various complex traffic environments.

[0029] Optionally, the video features obtained by Video Swin Transformer processing can be transformed and the converted features can be used as target features corresponding to the perception data. Through feature transformation, the video features obtained by Video Swin Transformer processing can better adapt to the needs of autonomous driving tasks, improve the perception accuracy, timing prediction ability, path planning ability and robustness of autonomous driving, thereby achieving more intelligent and safe autonomous driving.

[0030] Step 103, performing control signal prediction on the target feature to obtain a control signal, wherein the control signal includes at least one influencing factor; In practical applications, target features may include information such as the target's position, speed, direction, and shape. After obtaining the target features, the target features can be preprocessed, including normalization, dimensionality reduction, filtering, and other operations to ensure the stability and consistency of the features. The features are then input into a trained deep learning model, and the model predicts and obtains a control signal through forward propagation calculations and other processing. The obtained control signal includes at least one influencing factor to ensure the safe driving of the vehicle in a complex traffic environment.

[0031] Step 104, determining a causal effect difference corresponding to each influencing factor, wherein the causal effect difference represents the degree of influence of the influencing factor on the control signal; The causal effect difference represents the degree of influence of a certain influencing factor on the control signal. Therefore, the causal effect difference corresponding to each influencing factor can be estimated using the causal inference method. Specifically, the counterfactual result of the control signal can be calculated by intervening in a certain factor in the model (such as changing the road type) and observing the change in the control signal, or assuming that a certain factor changes (such as assuming that the weather changes from sunny to rainy).

[0032] When the control signal is output to the vehicle's actuator to execute the corresponding output decision for autonomous driving, different influencing factors will have different effects on the output decision corresponding to the control signal. For example, when a vehicle is driving in a straight line and a STOP sign and a dead end road appear in front of it, when the vehicle is in autonomous driving mode, it will execute the output decision of parking. At this time, it is impossible to determine whether the vehicle stops because of the STOP sign or the dead end road. Therefore, it is necessary to analyze the causal effect difference between the STOP sign and the dead end road on the output decision of parking to determine the degree of influence on the output decision. This not only helps to improve the safety and reliability of the output decision of the car's autonomous driving, but also optimizes the driving experience.

[0033] Step 105, determining the control decision cause corresponding to the control signal according to the causal effect difference; By determining the causal effect difference corresponding to each influencing factor, the degree of influence of the influencing factor on the control signal can be judged. In practical applications, the causal effect difference can be calculated based on the conditional probability, and the causal effect difference can be compared with the preset causal effect difference threshold. The influencing factor with a causal effect difference greater than the preset causal effect difference threshold is used as the control decision cause, so that the control decision cause corresponding to the output decision is more accurate, the autonomous driving model is optimized, the safety of autonomous driving is improved, and the model is interpretable.

[0034] In practical applications, the influencing factor with the largest causal effect difference can also be used as the reason for the control decision.

[0035] Step 106, when controlling the automatic driving of the vehicle according to the control signal, playing the reason for the control decision.

[0036] After the vehicle controller obtains the control signal, it will output the corresponding output decision based on the control signal and the control decision reason corresponding to the output decision, helping the driver to better understand the vehicle's environment and the operating status of the autonomous driving task.

[0037] Optionally, the reasons for control decisions can be visualized and fed back to users through display devices such as the vehicle's central control display screen through data visualization tools such as heat maps combined with voice, helping drivers to better understand the vehicle's environment and the operating status of the autonomous driving task, so as to enhance users' trust in the autonomous driving system.

[0038] Reference Figure 2 , shows a schematic diagram of a scenario of an automatic driving method provided by an embodiment of the present invention. Taking a three-lane scenario as an example, the vehicle is driving straight in the middle lane, and there are vehicles in the left and right lanes. At this time, the target vehicle on the right in front suddenly cuts into the lane from the left, and the vehicle slows down to maintain a safe distance. When the vehicle outputs the deceleration decision, the large language model will simultaneously describe the scenario and the reason for the vehicle's decision.

[0039] The following are examples of scenario descriptions and reason descriptions: Description: The vehicle is about to slow down.

[0040] Reason: The vehicle ahead cuts into this lane, so you need to slow down to maintain a safe distance and avoid a collision.

[0041] Reference Figure 3 , shows a flowchart of the steps of output control decision-making reasons provided by an embodiment of the present invention, which may specifically include the following steps: Step 301, inputting control signals, perception data, historical visual features and causal effect differences corresponding to various influencing factors; In practical applications, the control signals generated by the vehicle, the environmental information in the perception data, the historical visiontoken visual feature information (a highly abstract and condensed version of complex visual information, containing important information such as the shape, color, position, and motion trend of objects), and the causal effect differences corresponding to each influencing factor can be collected and sorted.

[0042] Step 302, extracting key visual features; Based on the collected and sorted control signals, environmental information in the perception data, historical vision token visual feature information and the causal effect differences corresponding to each influencing factor, the data is processed by causal effect analysis, and the key Vision Token visual features are extracted and visualized to show the degree of influence of different visual features on the control signal in the form of a heat map.

[0043] Step 303, inputting key visual features into large language model reasoning; The extracted key Vision Token visual features are fed into the large language model. Through the natural language understanding and generation capabilities of the large language model, the autonomous driving system can enhance its ability to parse and make decisions on complex environmental information, improve the system's perception accuracy, understanding depth, and response speed, and ultimately achieve a safer and smarter autonomous driving experience.

[0044] Step 304, language model inference and language response generation; The large language model deeply processes the key Vision Token visual features, understands the actual meaning represented by the key Vision Token visual features, and converts them into natural language descriptions.

[0045] Step 305: output a voice explanation of the reason for the control decision.

[0046] The voice explanation converted into natural language description is used as the voice explanation of the control decision reason corresponding to the control signal, and the voice explanation of the control decision reason is output, so that the model has interpretability, improving the safety of the autonomous driving system and the user experience.

[0047] An automatic driving method provided by an embodiment of the present invention obtains perception data of a vehicle; obtains target features corresponding to the perception data after feature extraction of the perception data; obtains a control signal by predicting the control signal on the target feature, wherein the control signal includes at least one influencing factor; determines the causal effect difference corresponding to each influencing factor, the causal effect difference characterizing the degree of influence of the influencing factor on the control signal; determines the control decision reason corresponding to the control signal based on the causal effect difference; and plays the control decision reason when controlling the automatic driving of the vehicle according to the control signal; determining the control decision reason corresponding to the control signal by determining the causal effect difference corresponding to each influencing factor can help users understand the decision-making process of the model, and by analyzing the consequences of different decisions, the decision reasons of the control strategy are fed back to the user, so that the model has explainability and enhances the user's trust in the automatic driving system.

[0048] In an embodiment of the present invention, extracting features from the perception data to obtain target features corresponding to the perception data includes: Performing feature extraction on the perception data to obtain image features corresponding to the perception data; The image features are screened according to importance weights to obtain target features.

[0049] After obtaining the video information collected by sensors such as cameras, radars or lidars installed on the vehicle, the video information is input into the Video Swin Transformer module to extract video features.

[0050] In practical applications, the Video Swin Transformer architecture includes: Input layer: used to process video frames and generate 3D patches.

[0051] Patch Partitioning layer: divides the video frame into 3D patches.

[0052] Linear Embedding layer: Converts 3D Patch into Token sequence.

[0053] Swin Transformer Blocks layer: Processes the token sequence through multiple layers of Transformer blocks to extract spatiotemporal features.

[0054] After obtaining the video information, the video is passed to the input layer to process the video frames, and then the video is input to the Patch Partitioning layer to divide the video frames into 3D patches. The size of the input video can be defined as T×H×W×3, consisting of T frames, each frame contains H×W×3 pixels, where H and W represent the height and width information respectively, and T represents the time dimension information. After being processed by the Patch Partitioning layer, 2×4×4×3 tokens can be obtained, and each patch / token consists of a 96-dimensional feature.

[0055] After that, the data is input into the Linear embedding layer, the 3D Patch is converted into a Token sequence, the high-dimensional data is mapped into a low-dimensional space, and then the data is input into the Swin Transformer Blocks layer to extract spatiotemporal features. In this embodiment, the Linear Embedding layer is applied to project the features of each Token to any dimension represented by C, and then the dimensions of the features are changed through multiple Swin Transformer Blocks layers and Patch Partitioning layers until the video tokens image features of the required dimensions are obtained, and then high-level spatiotemporal features are extracted to achieve effective processing and understanding of video information.

[0056] In order to ensure that the extracted features are effective, after obtaining the image features of the target dimension, the image features can be screened according to the importance weights based on the spatial attention mechanism to obtain the target features.

[0057] In one embodiment of the present invention, the step of screening the image features according to importance weights to obtain target features includes: Inputting the image features into a target learner, and determining the importance weights of the image features through a spatial attention mechanism; Performing weighted aggregation on the input image features according to the importance weights to obtain a first image feature; Perform feature conversion on the first image feature to obtain the target feature.

[0058] In an embodiment of the present invention, a Token learner is used as a target learner, image features are input into the Token learner, the importance weights of the image features are determined through a spatial attention mechanism, and then the input image features are weighted and aggregated according to the importance weights to obtain a first image feature. After obtaining the first image feature, feature transformation is performed on the first image feature to obtain the target feature. By using the Token learner to learn effective information in the image features, the model calculation cost can be reduced and the model generalization ability can be improved.

[0059] In practical applications, let X be the spatiotemporal input tensor, and the dimensions of X are T×H×W×C, where H×W corresponds to the spatial dimension (height and width) of the input, T is the temporal dimension (i.e., frame), and C is the number of channels. is the image frame at a certain moment, that is The dimension is H×W×C. For image input T=1, then .

[0060] Specifically, define the tokenizer function: , that is, a tokenizer function is learned through model training, which is adaptively Select the important information to combine.

[0061] For each frame of the image, different tokens will be learned, and S (number of channels) is set to be much smaller than H×W to significantly reduce the number of tokens and improve computational efficiency.

[0062] Among them, the tokenizer function The design is based on the spatial attention mechanism, and the model is based on Learn a spatial weight matrix , the size is H×W, and then multiply by , the final tokenizer function is obtained through global pooling ,Right now:

[0063] in, is the Hadamard product, which is element-wise multiplication. The dimensions are H×W×C, and the function and the broadcast function The intermediate weight tensor for the computation. Represents global average pooling, the purpose is to reduce the dimension to ; The computation is a series of convolution operations and sigmoid layers, and finally all the obtained tokens are combined to form the output tensor: .

[0064] In an embodiment of the present invention, performing feature conversion on the first image feature to obtain the target feature includes: Determining a first feature conversion relationship according to the perception data of the vehicle, the first feature conversion relationship representing a mapping relationship of converting the first image feature into a first overhead perspective feature; Converting the first image feature into the first top-view feature according to the first feature conversion relationship; When the perception data includes the first image features of multiple viewing angles, performing feature fusion on first overhead viewing angle features corresponding to the first image features of each viewing angle; The fused first top-down perspective feature is used as the target feature.

[0065] BEV features refer to environmental features expressed from a top-down perspective. In autonomous driving systems, BEV features are usually used to describe environmental information around the vehicle, such as roads, obstacles, traffic signs, etc.

[0066] In this embodiment, the first image feature may be converted into a BEV (Bird's Eye View) feature, that is, the BEV feature is used as the first top-view feature.

[0067] By determining the mapping relationship of BEV features, the first image features are converted into BEV features. If the perception data contains first image features from multiple perspectives, the first image features from each perspective can be converted and fused according to the mapping relationship of BEV features to finally obtain the target features.

[0068] Reference Figure 4 , shows a flowchart of the steps of feature conversion provided by an embodiment of the present invention, which may specifically include the following steps: Step 401, inputting the perception data of the vehicle surrounding environment acquired by the visual device; In practical applications, the perception data of the vehicle's surrounding environment acquired by the visual device can be input into the BEV module, and the BEV module processes the perception data.

[0069] Step 402, feature extraction; After the BEV module acquires the perception data, features in the perception data may be extracted based on geometric feature extraction, semantic feature extraction, or dynamic feature extraction. In this embodiment, the first image feature learned by the token learner may be directly used as the feature to be extracted.

[0070] Step 403, coordinate conversion; After obtaining the first image feature, the coordinates of the image feature are mapped to the BEV space according to the mapping relationship. For example, the intrinsic parameters (such as focal length, principal point coordinates) and extrinsic parameters (such as camera posture, rotation matrix, translation vector) of the camera can be obtained, and the pixel coordinates of the image feature are converted into world coordinates using the camera parameters. The specific steps are as follows: converting the image pixel coordinates into three-dimensional coordinates in the camera coordinate system, or converting the three-dimensional coordinates in the camera coordinate system into three-dimensional coordinates in the world coordinate system.

[0071] Step 404, feature fusion and optimization; After mapping the coordinates of the image features to the BEV space and obtaining the BEV coordinates, the non-integer coordinates can be interpolated, and then feature interpolation and aggregation can be performed in the BEV grid. If there are first image features from multiple perspectives, the first image features from multiple perspectives can be fused, and the converted BEV features can be further optimized using a deep learning model.

[0072] Step 405: output the bird's-eye view feature.

[0073] After the BEV module processes the first image feature to obtain the corresponding BEV feature, the characteristics of the BEV feature, such as having a global perspective, multi-modal fusion, and being easy to process, can be utilized to help the model make better predictions and decisions. At the same time, it is easier to fuse with other sensor data to provide more comprehensive and accurate environmental perception, thus adapting to different autonomous driving tasks.

[0074] In one embodiment of the present invention, determining the causal effect difference corresponding to each influencing factor, where the causal effect difference characterizes the influence degree of the influencing factor on the control signal, includes: Determine the set of influencing factors of the control signal according to the perception data, and the set of influencing factors includes at least one influencing factor; Select one influencing factor from the set of influencing factors for intervention and fixation; In the case of selecting one influencing factor for intervention and fixation, determine the causal effect difference corresponding to this influencing factor.

[0075] In the process of predicting the control signal based on the obtained target feature to obtain the control signal, there will be different influencing factors affecting the prediction result. Therefore, the control signal will include at least one influencing factor. To ensure the interpretability of the autonomous driving model and more effectively optimize the autonomous driving model based on the output decision of the control signal, it is necessary to determine the influence degree of the influencing factor on the output decision obtained by predicting the control signal.

[0076] In this embodiment, the influencing factors of the control signal are determined based on counterfactual intervention to obtain the causal effect difference corresponding to each influencing factor.

[0077] For example, assume there are , , endogenous variables, where assume and represent the influencing factors affecting the output decision, and the result of the output decision is expressed as , , , are the natural values of the variables , , . represents the intervention to fix the value of the variable to be . At this time, it represents selecting the variable for intervention and fixation, so that the value of the variable is fixed to . is the conditional probability of when , and When the intervention is fixed hour probability.

[0078] Specifically, selecting an influencing factor for intervention and fixation includes: selecting an influencing factor and fixing it for intervention, so that the influencing factor does not serve as input information affecting the output decision, and calculating the conditional probability at this time. Then, the causal effect difference corresponding to the influencing factor can be determined according to the corresponding conditional probability when selecting an influencing factor for intervention and fixation.

[0079] In practical applications, the relevant technology performs causal analysis based on a simple graphical model, and the obtained analysis results are often not accurate enough. Through the method of this embodiment, an influencing factor is selected from the influencing factor set for intervention and fixation; when an influencing factor is selected for intervention and fixation, the causal effect difference corresponding to the influencing factor is determined, which can help the system understand the causal effect of the control signal on the final output decision, determine the influencing factors of the output decision, and feed back the influencing factors to the user, thereby enhancing the user's trust in the autonomous driving system.

[0080] In an embodiment of the present invention, in the case where an influencing factor is selected for intervention and fixation, determining the causal effect difference corresponding to the influencing factor includes: Selecting an influencing factor from the influencing factor set to intervene in a fixed condition and setting the conditional state as a first conditional state; Setting the conditional state under the original conditions of each influencing factor to the second conditional state; Calculating a first conditional probability of predicting a control signal under the first condition and a second conditional probability of predicting a control signal under the second condition; The difference between the second conditional probability and the first conditional probability is determined as the causal effect difference.

[0081] In this embodiment, an influencing factor is selected for intervention and fixing, so that the influencing factor does not serve as input information that affects the output decision, and the influencing factor is set as the first conditional state of the influencing factor as input information that affects the output decision, and the influencing factor is set as the second conditional state of the influencing factor as input information that participates in affecting the output decision.

[0082] The first conditional probability corresponding to the influencing factor under the first condition state and the second conditional probability corresponding to the second condition state are calculated respectively, and the causal effect difference is determined based on the difference between the second conditional probability and the first conditional probability. Causal inference helps to understand the decision-making process of the model, and by analyzing the consequences of different decisions, the final decision-making reasons that affect the decision are found. The final decision-making reasons can then be fed back to the user, making the model explainable and improving the user's trust in the autonomous driving system.

[0083] For example, suppose there is , , The endogenous variable is assumed to be and represents the factors that affect the output decision, and the result of the output decision is expressed as , , , For variables , , The natural value of . Represents an intervention fixed variable The value of . When hour, The conditional probability of When intervention hour probability. Reflects the variable for In individuals The distribution of Reflects each are fixed as hour, The overall distribution of .

[0084] make and represents the factors that affect the output decision, and the result of the output decision is expressed as , and Variables act on superior, Indicates that the influencing factor is used as input information affecting the output decision, and is set to the second condition state; Indicates that variable X is intervened and fixed, so that the influencing factor does not serve as input information that affects the output decision, and it is set to the first condition state; Corresponding Set as the second conditional probability, Corresponding Set it as the first conditional probability, then the difference between the second conditional probability and the first conditional probability is the causal effect difference: Causal effect difference = .

[0085] In an embodiment of the present invention, determining the control decision reason corresponding to the control signal according to the causal effect difference includes: Compare the differences in causal effects of different influencing factors when different influencing factors are fixed for intervention; The influencing factors whose causal effect difference is greater than a preset causal effect difference threshold are used as the control decision reasons.

[0086] In practical applications, when intervening fixed variables When The causal effect difference when the intervention is fixed is compared with the preset causal effect difference threshold to determine the variable As for the degree of influence on the output decision of the control signal, that is, the difference between the second conditional probability and the first conditional probability is compared with the preset probability threshold. If the difference between the second conditional probability and the first conditional probability is greater than the preset probability threshold, it is considered that the causal effect difference is large, indicating that the factor affects the output decision of the control signal; if the difference between the second conditional probability and the first conditional probability is less than the preset probability threshold, the causal effect difference is not large, indicating that the factor does not affect the output decision of the control signal.

[0087] Among them, the preset causal effect difference threshold and the preset probability threshold can be set according to the needs of autonomous driving accuracy, and this application does not impose any restrictions on this.

[0088] Reference Figure 5 , shows a schematic diagram of a scenario of counterfactual intervention provided by an embodiment of the present invention, which may specifically include the following contents: by Figure 5 Take this scenario as an example. The scenario includes an orange school bus, pedestrians on the motor lane, traffic lights, and pedestrians on the zebra crossing. By decomposing the traffic participants, the corresponding relationships between the influencing factors of the orange school bus-traffic participant P, the pedestrian on the motor lane-traffic participant Q, the traffic light-traffic participant M, and the pedestrian on the zebra crossing-traffic participant N are obtained. Then, traffic participant P, traffic participant Q, traffic participant M, and traffic participant N are used as influencing factors P, Q, M, and N for counterfactual intervention to establish the causal diagram on the right side of the scenario diagram. In the diagram, influencing factors P and influencing factors Q directly affect the output decision Y of the control signal, and influencing factors M and influencing factors N interact with each other, thereby affecting the output decision Y of the control signal.

[0089] To determine whether the influencing factor P will affect the decision, first, delete the influencing factor P from the scene, that is, set the influencing factor P to the first conditional state that does not participate in the input information that affects the output decision, and calculate the first conditional probability; then set the influencing factor P to the second conditional state that participates in the input information that affects the output decision, and calculate the second conditional probability. If the second conditional probability is the same as the first conditional probability or the difference between the second conditional probability and the first conditional probability is within the preset probability threshold, it means that the influencing factor P is not the decision-making reason that affects the model; if the second conditional probability is not the same as the first conditional probability and the difference between the second conditional probability and the first conditional probability is outside the preset probability threshold, it means that the influencing factor P is the decision-making reason that affects the model. And so on, until all the influencing factors that affect the model output decision are found, and then the control decision reason is determined, such as Figure 5 In the lower right corner of the picture, the traffic participants are marked with images. The blue area is the control decision reason that affects the model output decision, and the green area is the influencing factor that will not affect the model output decision. The method of this embodiment can not only handle extremely complex scenarios, but also explain the reasons for the decision, which can enhance the user's intelligent driving experience.

[0090] In an embodiment of the present invention, perception data of a vehicle is acquired; feature extraction is performed on the perception data to obtain target features corresponding to the perception data; control signal prediction is performed on the target features to obtain a control signal, wherein the control signal includes at least one influencing factor; the causal effect difference corresponding to each influencing factor is determined, wherein the causal effect difference represents the degree of influence of the influencing factor on the control signal; the control decision reason corresponding to the control signal is determined based on the causal effect difference; and the control decision reason is played when the vehicle is controlled to drive automatically according to the control signal; determining the control decision reason corresponding to the control signal by determining the causal effect difference corresponding to each influencing factor can help users understand the decision-making process of the model, and by analyzing the consequences of different decisions, find the optimal control strategy, make the model interpretable, and improve the safety of the automatic driving system and the user experience.

[0091] Reference Figure 6 , shows a structural block diagram of an embodiment of an automatic driving device of the present application, which may specifically include the following modules: A data acquisition module 601 is used to acquire the perception data of the vehicle; A target feature determination module 602 is used to extract features from the perception data to obtain target features corresponding to the perception data; A prediction module 603 is used to predict a control signal for the target feature to obtain a control signal, wherein the control signal includes at least one influencing factor; An influence degree determination module 604 is used to determine a causal effect difference corresponding to each influencing factor, wherein the causal effect difference represents the influence degree of the influencing factor on the control signal; A decision reason determination module 605, configured to determine a control decision reason corresponding to the control signal according to the causal effect difference; The output module 606 is used to play the reason for the control decision when controlling the automatic driving of the vehicle according to the control signal.

[0092] In an embodiment of the present application, the autonomous driving device provided by the embodiment of the present application obtains perception data of the vehicle; extracts features from the perception data to obtain target features corresponding to the perception data; predicts control signals on the target features to obtain control signals, wherein the control signals include at least one influencing factor; determines causal effect differences corresponding to each influencing factor, the causal effect differences characterizing the degree of influence of the influencing factor on the control signal; determines the control decision reason corresponding to the control signal based on the causal effect differences; and plays the control decision reason when controlling the autonomous driving of the vehicle according to the control signal; determining the control decision reason corresponding to the control signal by determining the causal effect differences corresponding to each influencing factor can help users understand the decision-making process of the model, and by analyzing the consequences of different decisions, the decision reasons of the control strategy are fed back to the user, so that the model is interpretable and the user's trust in the autonomous driving system is enhanced.

[0093] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0094] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned automatic driving method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0095] An embodiment of the present application also provides a vehicle-mounted terminal, including: a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory, and when the computer program is executed by the processor, the various processes of the above-mentioned automatic driving method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0096] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0097] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the present application may be provided in the form of a complete hardware embodiment, a complete software embodiment, or a combination of software and hardware embodiments. Moreover, the present application may be provided in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0098] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0101] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0102] Finally, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0103] It should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the element.

[0104] The above has introduced in detail a self-driving method, device, storage medium, and vehicle-mounted terminal provided by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An automatic driving method, characterized in that: The method comprises: Acquire the vehicle’s perception data; Extracting features from the perception data to obtain target features corresponding to the perception data; Performing control signal prediction on the target feature to obtain a control signal, wherein the control signal includes at least one influencing factor; Determine a causal effect difference corresponding to each influencing factor, wherein the causal effect difference represents the degree of influence of the influencing factor on the control signal; Determining a control decision cause corresponding to the control signal according to the causal effect difference; When the vehicle is controlled to automatically drive according to the control signal, the reason for the control decision is played.

2. The method according to claim 1, characterized in that The extracting features of the sensed data to obtain target features corresponding to the sensed data includes: Performing feature extraction on the perception data to obtain image features corresponding to the perception data; The image features are screened according to importance weights to obtain target features.

3. The method according to claim 2, characterized in that The step of screening the image features according to importance weights to obtain target features includes: Inputting the image features into a target learner, and determining the importance weights of the image features through a spatial attention mechanism; Performing weighted aggregation on the input image features according to the importance weights to obtain a first image feature; Perform feature conversion on the first image feature to obtain the target feature.

4. The method according to claim 3, characterized in that The step of performing feature conversion on the first image feature to obtain the target feature includes: Determining a first feature conversion relationship according to the perception data of the vehicle, where the first feature conversion relationship represents a mapping relationship of converting the first image feature into a first overhead perspective feature; Converting the first image feature into the first top-view feature according to the first feature conversion relationship; When the perception data includes the first image features of multiple viewing angles, performing feature fusion on first overhead viewing angle features corresponding to the first image features of each viewing angle; The fused first top-down perspective feature is used as the target feature.

5. The method according to claim 1, characterized in that: The determining of the causal effect difference corresponding to each influencing factor, wherein the causal effect difference represents the degree of influence of the influencing factor on the control signal, includes: Determine a set of influencing factors of the control signal according to the sensing data, wherein the set of influencing factors includes at least one influencing factor; Selecting an influencing factor from the influencing factor set for intervention and fixation; In the case where an influencing factor is selected for intervention and fixation, the causal effect difference corresponding to the influencing factor is determined.

6. The method according to claim 5, characterized in that In the case where an influencing factor is selected for intervention and fixation, determining the causal effect difference corresponding to the influencing factor includes: Selecting an influencing factor from the influencing factor set to intervene in a fixed condition and setting the conditional state as a first conditional state; Setting the conditional state under the original conditions of each influencing factor to the second conditional state; Calculating a first conditional probability of predicting a control signal under the first condition and a second conditional probability of predicting a control signal under the second condition; The difference between the second conditional probability and the first conditional probability is determined as the causal effect difference.

7. The method according to claim 1, characterized in that The determining the control decision reason corresponding to the control signal according to the causal effect difference includes: Compare the differences in causal effects of different influencing factors when different influencing factors are fixed for intervention; The influencing factors whose causal effect difference is greater than a preset causal effect difference threshold are used as the control decision reasons.

8. An automatic driving device, characterized in that: The device comprises: A data acquisition module, used to acquire the perception data of the vehicle; A target feature determination module, used to extract features from the perception data to obtain target features corresponding to the perception data; A prediction module, configured to predict a control signal for the target feature to obtain a control signal, wherein the control signal includes at least one influencing factor; An influence degree determination module, used to determine a causal effect difference corresponding to each influencing factor, wherein the causal effect difference represents the influence degree of the influencing factor on the control signal; A decision reason determination module, used to determine the control decision reason corresponding to the control signal according to the causal effect difference; The output module is used to play the reason for the control decision when controlling the automatic driving of the vehicle according to the control signal.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the automatic driving method according to any one of claims 1 to 7 is implemented.

10. A vehicle-mounted terminal, characterized in that: It includes a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the automatic driving method as described in any one of claims 1 to 7.

Images