Automatic driving system reliability self-diagnosis method and platform based on multi-modal sensor fusion

Abstract

The invention provides an automatic driving system reliability self-diagnosis method and platform based on multi-modal sensor fusion. The method comprises the following steps: acquiring image modal data, performing feature extraction by using a visual Transform model, generating an environment scene label, and representing the type of a current environment; carrying out statistics on key modal characteristics in actual automatic driving operation data to obtain a modal score vector corresponding to each environment scene label to reflect the prior credibility of each sensor modal in the environment, and calculating a modal trust weight through softmax to obtain a confidence weight vector; performing weighted fusion on the key modal features based on the confidence weight vector to obtain fusion features; generating a modal reconstruction feature according to the fusion feature, and generating a modal consistency error vector based on the modal reconstruction feature; and in combination with the confidence weight vector and the modal consistency error vector, determining an early warning signal of the current system for judging whether the system has a fusion degradation or modal failure risk.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving, and in particular relates to a reliability self-diagnosis method and platform for autonomous driving systems based on multimodal sensor fusion. Background Art

[0002] The development of autonomous driving systems places extremely high demands on environmental perception capabilities. To improve perception accuracy and robustness, modern autonomous driving platforms generally employ multimodal sensor systems, including cameras, millimeter-wave radar, lidar, and electromagnetic sensors. By fusing perception data from different modalities, they comprehensively model the objects, roads, and obstacles surrounding the vehicle. The redundant information introduced by multimodal fusion theoretically enhances the fault tolerance of perception systems, but in practice, the reliability of perception systems still faces significant challenges. This is particularly true in variable environmental conditions, such as rain and fog, nighttime driving, and tunnels, where the signal-to-noise ratio of different modalities fluctuates dramatically. Traditional fusion strategies typically employ fixed weights or simple weighting based on single-moment features, lacking the ability to adapt to the current environment, resulting in ineffective fusion results in certain scenarios.

[0003] More critically, existing systems generally lack the ability to proactively monitor and model trends in the health of sensors. They are unable to promptly identify risks when soft sensor failures (such as image color cast, radar delay, and electromagnetic signal interference) occur, often waiting until system perception results have exhibited serious deviations before issuing passive alarms. Furthermore, most current autonomous driving perception platforms focus on "perception accuracy" rather than "perception credibility." Therefore, even if the fusion results appear normal, it is difficult to determine whether they are based on reliable data sources. Existing technologies have yet to provide a mechanism that can comprehensively consider scene changes, modal degradation, and the degree of information redundancy between modalities to dynamically assess the current reliability level of the system, nor have they achieved real-time interpretation of perception quality and prediction of future trends. This gap directly impacts the safety and maintainability of the system in actual road environments, and has become a critical issue that urgently needs to be addressed, especially in the context of long-term operation, aging of multi-sensor systems, or frequent complex environments. Summary of the Invention

[0004] The purpose of this invention is to propose a reliability self-diagnosis method and platform for autonomous driving systems based on multimodal sensor fusion, which systematically addresses the shortcomings of existing autonomous driving perception systems in soft fault identification, perception credibility assessment, and health trend modeling.

[0005] To achieve the above objectives, a first aspect of the present invention provides a reliability self-diagnosis method for an autonomous driving system based on multimodal sensor fusion, the method comprising:

[0006] S1. Obtain image modal data, use the visual Transformer model to extract features, and generate environmental scene labels to indicate the type of the current environment;

[0007] S2. Obtain a modal score vector corresponding to each environmental scenario label by statistically analyzing key modal features in actual autonomous driving operation data. This score vector reflects the prior credibility of each sensor modality in that environment. Calculate the modal trust weight using softmax to obtain a confidence weight vector. The key modal features include image modal data, millimeter-wave radar target detection rate, and electromagnetic sensor short-range obstacle detection rate.

[0008] S3. Performing weighted fusion on the key modal features based on the confidence weight vector to obtain a fusion feature;

[0009] S4. Generating a modal reconstruction feature according to the fusion feature, and generating a modal consistency error vector based on the modal reconstruction feature;

[0010] S5. Determine a warning signal for the current system based on the confidence weight vector and the modal consistency error vector, so as to judge whether the system has a risk of fusion degradation or modal failure.

[0011] Furthermore, the environmental scene labels include sunny environments with sufficient lighting during the day, foggy environments with severe visibility attenuation, rainy environments with water droplets and mirror reflection problems, low-light or no-light environments at night, and entering tunnels or dimly lit enclosed spaces, with the characteristics of rapid changes between strong lighting and darkness.

[0012] Furthermore, S2 also includes:

[0013] Obtain current key modal features and historical modal features;

[0014] According to the current key modal features and historical modal features, a feedback correction value of each modal feature is calculated by a semantic difference measurement function, and a feedback correction term is obtained based on the feedback correction value of each modal feature, which is used to correct the modal score vector.

[0015] Furthermore, the calculation of the modal trust weight by softmax to obtain the confidence weight vector includes:

[0016] The modal score vector of each modality and the feedback correction term are combined, and a risk suppression term is introduced to calculate the modal trust weight through softmax, where the softmax maps the modal score vector of each modality to a weight in the range of [0,1] and ensures that the sum is 1.

[0017] Furthermore, the risk suppression item is used to pre-constrain the potentially unreliable modalities to prevent their trust weights from being too high in extreme cases.

[0018] Furthermore, the S3 includes:

[0019] Using the confidence weight vector to perform weighted fusion on each modality feature in the spatial dimension to obtain a preliminary fusion feature;

[0020] On the basis of the preliminary fusion features, based on the preliminary fusion features, each modal feature is cross-modally self-aligned through a spatial semantic guidance module based on the dot product attention mechanism and a scene structure maintenance regularization term to obtain an aligned feature vector, and the aligned feature vectors are fused to obtain a fusion feature.

[0021] Furthermore, the spatial semantic guidance module based on the dot product attention mechanism uses the preliminary fusion feature as a query and the corresponding modal feature vector as a key and value to learn that the modal feature vector is aligned to the global fusion representation;

[0022] The scene structure preserving regularization term introduces a bias to the structure of the current modal feature vector according to the current environment scene label, so as to maintain dynamic consistency with the current environment scene label.

[0023] Furthermore, the generating of a modal reconstruction feature according to the fusion feature, and generating a modal consistency error vector based on the modal reconstruction feature, specifically includes:

[0024] A small reverse mapping module is designed for each modality to generate corresponding modality reconstruction features from the fusion features;

[0025] After the reconstruction is completed, a combined error function is designed to measure the consistency between the reconstructed features of each modality and the aligned feature vector;

[0026] A modal consistency error vector is obtained according to the combined error function of each modality, which represents the errors of the three modal characteristic vectors in the image modal data, the millimeter wave radar target detection rate and the electromagnetic sensor's detection rate of short-range obstacles.

[0027] Furthermore, the S5 specifically includes:

[0028] Obtain the confidence weight vector of the key feature vector to form the modal trust weight vector;

[0029] Calculating a current system-level fusion consistency error based on the modal trust weight vector and the modal consistency error vector to represent the overall deviation degree between the fusion result and the original modal features;

[0030] When the consistency error is greater than a preset global fixed threshold, it is considered that the current system is distorted and an early warning is triggered.

[0031] In a second aspect of the present invention, an autonomous driving system reliability self-diagnosis platform based on multimodal sensor fusion is provided, the platform comprising:

[0032] The modal perception unit is used to obtain image modal data, extract features using the visual Transformer model, and generate environmental scene labels to indicate the type of the current environment;

[0033] A modal analysis unit is configured to obtain a modal score vector corresponding to each environmental scenario label by statistically analyzing key modal features in actual autonomous driving operation data. The score vector reflects the prior credibility of each sensor modality in that environment and calculates the modal trust weight through softmax to obtain a confidence weight vector. The key modal features include image modal data, millimeter-wave radar target detection rate, and electromagnetic sensor short-range obstacle detection rate.

[0034] A modality fusion unit, configured to perform weighted fusion on the key modality features based on the confidence weight vector to obtain a fusion feature;

[0035] a modal correction unit, configured to generate a modal reconstruction feature according to the fusion feature, and generate a modal consistency error vector based on the modal reconstruction feature;

[0036] An early warning unit is used to combine the confidence weight vector and the modal consistency error vector to determine an early warning signal of the current system, so as to judge whether the system has the risk of fusion degradation or modal failure.

[0037] The beneficial technical effects of the present invention are at least as follows:

[0038] This paper establishes a dynamic fusion control mechanism that automatically adjusts the trust weights of different sensor modalities during the fusion process by identifying the current driving environment, thereby avoiding reliance on unreliable data in certain modality degradation scenarios. Furthermore, this paper designs an inter-modal information consistency analysis method that uses semantic reconstruction errors between multiple modalities to identify potential soft faults, revealing internal performance degradation even when the sensor's surface output is normal.

[0039] Furthermore, the present invention introduces a lightweight trend modeling module that leverages the temporal evolution of modal diagnostic indicators to predict future health trends for each modality, enabling early warning and adaptive policy adjustments. By integrating these multi-layered diagnostic and prediction mechanisms, the present invention builds a "perception credibility-driven" intelligent perception platform that maintains a stable, interpretable, and fault-tolerant perception system in complex environments, significantly improving the perception safety and self-healing capabilities of autonomous driving systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention is further described with reference to the accompanying drawings. However, the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative effort.

[0041] Figure 1 This is a flow chart of the reliability self-diagnosis method of the autonomous driving system based on multimodal sensor fusion of the present invention.

[0042] Figure 2 This is a framework diagram of the autonomous driving system reliability self-diagnosis platform based on multimodal sensor fusion of the present invention. DETAILED DESCRIPTION

[0043] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0044] like Figure 1 As shown, an embodiment of the present invention provides a reliability self-diagnosis method for an autonomous driving system based on multimodal sensor fusion, the method comprising:

[0045] S1. Obtain image modal data, use the visual Transformer model to extract features, and generate environmental scene labels to indicate the type of the current environment.

[0046] Specifically, get the input data:

[0047] Image modality data This is image data captured by a camera. Its dimensions are H × W × 3, representing the height, width, and three RGB channels of each frame. The camera image is normalized and denoised to ensure data quality.

[0048] Furthermore, the goal of this step is to input image data through the camera Use the deep learning model to process and analyze the image, identify the current scene type, and output the scene label S t This process extracts image features based on the visual Transformer model and then classifies them into specific scene types (such as "sunny", "rainy", "tunnel", etc.) through a classifier. Scene label S t Provide key information for subsequent modal trust adjustment.

[0049] First, the image data Preprocessing, including denoising, color normalization, and resizing, ensures that the image meets subsequent processing requirements. For autonomous driving, cameras may capture diverse environmental scenes (such as sunny days, nighttime, and foggy conditions). Therefore, data augmentation technology is used to process image data in these diverse environments.

[0050] Next, the image data is passed through a Visual Transformer (ViT) model for feature extraction. The model divides the input image into multiple fixed-size patches and processes these patches through a self-attention mechanism to extract spatial and contextual information from the image. The output of the Transformer is a deep feature representation of the image. This representation can effectively capture the semantic information in the image.

[0051] These image features are then input into a fully connected layer (classifier), which outputs a probability distribution for each scene category. By using the softmax activation function, the probability distribution is normalized and the scene label S with the maximum probability is output. t For example, if the model infers that the probability of "sunny" is 0.7, the probability of "rainy" is 0.2, and the probability of "tunnel" is 0.1, the system will choose label S t ="clear".

[0052] The specific classification process can be expressed by the following formula:

[0053]

[0054] in, Representing an image The semantic feature vector extracted by the visual Transformer (such as BEiT) model, W cls and b cls are the weights and biases of the classifier, respectively. Softmax is a function that normalizes multi-class outputs. Argmax indicates that the class with the highest probability is selected as the scene label S. t .

[0055] Output variable: St , scene label, indicating the type of the current environment. This label is selected from the following fixed set:

[0056] S t ∈"clear","fog","rain","night","tunnel"

[0057] "clear": a sunny environment with plenty of light during the day;

[0058] "fog": A foggy environment with severe visibility degradation;

[0059] "rain": A rainy environment with problems such as water droplets and mirror reflections;

[0060] "night": low-light or no-light environment at night;

[0061] "tunnel": Entering a tunnel or dark, enclosed space characterized by rapid changes in illumination between strong light and darkness.

[0062] As you can understand, this step uses the visual Transformer model to extract deep features from the input image and combines it with a classifier to identify the current scene type. This scene labeling provides an important basis for subsequent modality trust adjustment, enabling the system to dynamically adjust the trust weight of each modality based on different environments.

[0063] S2. By statistically analyzing the key modal features in the actual autonomous driving operation data, a modal score vector corresponding to each environmental scene label is obtained to reflect the prior credibility of each sensor modality in the environment. The modal trust weight is calculated by softmax to obtain a confidence weight vector; wherein the key modal features include image modal data, millimeter wave radar target detection rate, and electromagnetic sensor detection rate at short distances.

[0064] Specifically, the input of this step is the environment scene label S output in step 1 t . In addition, it should be noted that in this system, "millimeter wave radar" specifically refers to high-frequency electromagnetic sensors used for medium and long-range target detection. The operating frequency band is 24GHz or 77GHz. It is deployed at the front and rear of the vehicle, etc., and is mainly used for forward obstacle detection and speed estimation. "Electromagnetic sensor" refers to short-range, non-imaging electromagnetic auxiliary sensing equipment, such as capacitive / inductive short-range detectors, ultra-wideband (UWB) small radars around the vehicle body, near-field sensing modules, etc., which are mainly deployed on wheel covers, side skirts or the edge of the bottom of the vehicle, and are often used for the perception of blind spots around the vehicle or extremely close objects. These two types of sensors have significant differences in operating range, signal characteristics and deployment scenarios, so they are regarded as two independent modes in this step.

[0065] In the autonomous driving system, the reliability of multiple sensor modalities is significantly affected by the environment. In order to achieve a dynamic and scene-adaptive perception fusion strategy, this step is based on the current scene label S t , generating a set of modal trust weights This is used for weighted feature fusion in the next step. This paper proposes a three-part joint scoring mechanism: a static environmental score, a modality consistency feedback correction term, and a risk mitigation regularization term. This mechanism not only reflects the system's understanding of the environment but also considers the current performance of the modality and fault-tolerant control in risky scenarios.

[0066] The system is used for each fixed scene S t Set a non-normalized modality rating vector This vector reflects the prior confidence level of each sensor modality in the environment. The score is obtained by analyzing performance indicators from actual autonomous driving operation data. For example, in the "fog" scenario, the system extracts the following key performance indicators by analyzing over 120 hours of vehicle data:

[0067] The camera image clarity is seriously degraded, and the Canny edge recognition rate is less than 30%. cam =0.2;

[0068] The target detection rate of millimeter wave radar is maintained above 88%, assuming s radar =0.9;

[0069] Electromagnetic sensors (such as UWB radar) are still stable in short-range obstacle detection. em =0.85.

[0070] Based on this, the corresponding scores for each scene are as follows:

[0071] m clear =[0.8,0.7,0.6]

[0072] m fog =[0.2,0.9,0.85]

[0073] m rain =[0.4,0.8,0.75]

[0074] m night =[0.1,0.7,0.65]

[0075] m tunnel =[0.3,0.6,0.7]

[0076] These scores preserve the performance differences between modalities and serve as the basis for confidence trends before softmax normalization.

[0077] To enhance adaptability, the system also calculates the semantic consistency between the modal output and the historical consistency as a feedback correction term. It is defined as follows:

[0078]

[0079] in, is the feature vector output by the perception network of modality m in the i-th frame, is the historical mean feature of the mode, which is obtained by performing a sliding average calculation on the features output by mode m in the past 64 frames. Specifically, the system averages the feature values ​​of the past 64 frames at each time t to obtain a stable historical feature value This avoids the impact of anomalies in a single frame. Divergence(·) is a semantic difference measurement function (such as cosine distance or KL divergence), and λ is an adjustment coefficient. The system uses a sliding window to average historical consistency to avoid fluctuations in trust weight caused by occasional anomalies. For example, if the camera modality is unstable in the last 5 frames, its correction value will be negative, thus reducing its trust.

[0080] In addition, in high-risk scenarios, even if the current output of some modes is stable, the system risk may increase due to physical defects. Therefore, the system introduces risk suppression items This item is set for each scenario, such as:

[0081] In the "night" and "fog" scenes,

[0082] In the "tunnel" scenario,

[0083] The electromagnetic mode is stable in most scenarios and is usually assumed to be

[0084] These items add penalty factors in the calculation to pre-constrain the potentially unreliable modalities and prevent their trust weight from being too high in extreme cases.

[0085] Finally, the system combines the above three scores and calculates the modal trust weight through softmax:

[0086]

[0087] Here, α is the adjustment coefficient of the risk regularization term, controlling its impact range. The softmax operation maps the three modal scores to weights in the range [0, 1], ensuring that the sum is 1 to meet the requirements of subsequent weighted fusion.

[0088] It can be understood that the output is the modal trust weight vector Each element is a normalized weighted value, representing the system's current level of confidence in that modality. This output serves as direct input to the next step, "modal feature fusion," guiding the multimodal perception system to dynamically adjust its fusion strategy in different scenarios.

[0089] S3. Perform weighted fusion on the key modal features based on the confidence weight vector to obtain fused features.

[0090] Specifically, first, the present invention uses the confidence weight vector W t The three modal features are weightedly fused in the spatial dimension. To this end, all modal features are first mapped to the same dimension d through a shared 1×1 convolution to obtain standardized features. Then perform weighted summation to construct the initial fusion feature F t :

[0091]

[0092] in, represents the modal confidence weight output from step 2, is the modality-aware feature tensor aligned with feature dimensions. This formula ensures that the contribution of each modality in the fusion is directly related to its credibility in the current scenario, thus preventing weak modalities from dominating the fusion output at critical moments.

[0093] However, relying solely on weighted summation cannot solve two key problems: 1) spatial semantic misalignment between modalities; 2) imbalance of uneven distribution of perception within the main modality in extreme scenarios. Therefore, this step proposes a structural alignment enhancement term to fuse the features F t To guide, cross-modal self-alignment of each modal feature is performed:

[0094]

[0095] Attn align (·) is a spatial semantic guidance module based on the point-wise attention mechanism, where F t As the query, the features of modality m are key and value, which are used to learn how the modality features should be "pulled closer" to the global fusion representation;

[0096] It is an innovative scene structure preservation regularization term, which is based on the current scene S t , introducing a bias towards preserving modal structure. For example, it strengthens longitudinal continuous edge information in the "tunnel" scene and preserves dynamic consistency in temporally stable regions in the "rain" scene.

[0097] Specifically, for the regularization term, the present invention is implemented as a set of scene-specific regularization weight templates Indicates the importance of each pixel in preserving the structure of the current scene. This term essentially penalizes the difference between the local gradient of the modality feature map and the structural gradient of the global fusion feature map, ensuring that the modality is not diluted by the fusion mechanism in its "dominant area" during the fusion process.

[0098] This mechanism effectively integrates three types of objectives:

[0099] Use confidence weighting to improve overall robustness;

[0100] Using attention alignment to achieve modality semantic consistency;

[0101] Structural regularization is used to maintain physical continuity and scene adaptability in key structural areas.

[0102] After the above processing, the final fusion feature F is output t , as input to the downstream diagnosis module.

[0103] It can be understood that the output of this step is the fusion perception feature tensor It integrates the three modes in the environment S at the current moment t The perceptual information under the scene is preserved, the spatial semantic structure is retained, and through modal alignment and scene regularization processing, it has high semantic consistency and structural fidelity. t It will serve as the input signal for the next step, “anomaly detection and fault prediction,” to support further diagnostic analysis.

[0104] S4. Generate a modal reconstruction feature according to the fusion feature, and generate a modal consistency error vector based on the modal reconstruction feature.

[0105] Specifically, this step aims to construct a set of evaluation index systems specifically for diagnosing modal health and fusion consistency, by t Each modal signature is then reversely "reconstructed" and compared with its true modal signature to calculate the semantic consistency error. This mechanism is essentially a fusion-based reverse mapping modeling process that does not involve behavioral decision-making but instead generates diagnostic metrics to support subsequent fault prediction and credibility assessment. Its innovation lies in elevating modal consistency evaluation from a "modal-to-modal" comparison to a "fusion-to-modal" reverse interpretation path, making the behavior of the fusion model interpretable.

[0106] First, a small reverse mapping module is designed for each modality m(m∈{cam,radar,em}) For the fusion feature F t Generate modal reconstruction features The structure is as follows:

[0107] Two layers of 1×1 convolution (the number of channels is d→d / 2→d respectively);

[0108] Each layer is followed by ReLU activation;

[0109] Local self-attention (such as local window attention or simple convolutional attention mechanism) is added after the last layer to enhance the sensitivity of local structure.

[0110] Furthermore, the reconstruction process formula is as follows:

[0111]

[0112] The structure is lightweight, has few shared parameters, is easy to deploy and supports parallel reconstruction of different modalities.

[0113] After the reconstruction is completed, the present invention uses a combined error function Measure the reconstruction features of each modality With the original features Consistency between:

[0114]

[0115] Among them, item 1 is the content error, which represents the Euclidean distance in the feature space;

[0116] Item 2 For structural error, the Sobel operator is used to extract horizontal and vertical gradient maps to reflect the consistency of the modal in the spatial structure; s is the weight of the structural item (usually set to It can be adjusted adaptively according to the scene, such as increasing the weight of this item on foggy days).

[0117] It should be noted that the entire process does not require labels and does not involve external models. All modules can be trained end-to-end and can also be used independently in the inference phase.

[0118] For example, in the "night" scene, if the camera mode is degraded due to visibility, its and reality If the difference increases and the structural error term also rises abnormally, the system will record Significantly higher than normal. If a historical sliding average or modal health baseline is deployed, this value is used to indicate that the modality may have a temporary failure or fusion bias.

[0119] S5. Determine a warning signal for the current system based on the confidence weight vector and the modal consistency error vector, so as to judge whether the system has a risk of fusion degradation or modal failure.

[0120] Specifically, the input for this step is:

[0121] The modal consistency error vector output from step 4 is

[0122] The modal trust weight vector output from step 2

[0123] in represents the structural consistency error of mode m, Indicates the system's current level of trust in the modality.

[0124] The goal of this step is to directly calculate the overall system fusion consistency state using existing modal error information and trust weights, and output a clear warning signal to determine whether the system is at risk of fusion degradation or modal failure. To avoid excessive modeling and complex trend processing, this step uses a weighted error judgment mechanism to ensure the system is executable, lightweight, and logically self-consistent.

[0125] The system is characterized by three modal structural consistency errors. Its trust weight Calculate the current system-level fusion consistency error

[0126]

[0127] This value indicates the overall deviation between the fusion result and the original features of the modality. The fusion weight reflects the importance of each modality to the system in the current scenario.

[0128] The system sets a global fixed threshold τ (e.g., τ = 0.35) to determine whether there is an abnormal deviation in the current fusion state:

[0129]

[0130] in is the indicator function, if If the threshold is exceeded, the system will consider that the current perception fusion has been significantly distorted and trigger an early warning.

[0131] For example:

[0132] If the current trimodal errors are {0.25, 0.12, 0.30} and the trust weights are {0.5, 0.3, 0.2}, then calculate:

[0133]

[0134] Since 0.225<τ, the system does not trigger an early warning.

[0135] If the error is {0.48, 0.36, 0.42} and the weight is {0.6, 0.2, 0.2}, then:

[0136]

[0137] Because 0.456>τ, the system triggers an early warning.

[0138] Final output: System fusion error score:

[0139]

[0140] Indicates the degree of consistency deviation of the current system fusion;

[0141] System warning signal:

[0142]

[0143] Indicates whether the system triggers a perception fusion abnormality warning.

[0144] To better implement the above method, Figure 2 As shown, the present invention also proposes an autonomous driving system reliability self-diagnosis platform based on multimodal sensor fusion, based on any of the above-mentioned multimodal perception methods for road monitoring adapted to dynamic environments, the platform comprising:

[0145] The modality perception unit 101 is used to obtain image modality data, perform feature extraction using a visual Transformer model, and generate an environment scene label to indicate the type of the current environment;

[0146] The modal analysis unit 102 is configured to obtain a modal score vector corresponding to each environmental scene label by statistically analyzing key modal features in actual autonomous driving operation data, reflecting the prior credibility of each sensor modality in the environment, and calculating the modal trust weight through softmax to obtain a confidence weight vector. The key modal features include image modal data, millimeter-wave radar target detection rate, and electromagnetic sensor short-range obstacle detection rate.

[0147] A modality fusion unit 103 is configured to perform weighted fusion on the key modality features based on the confidence weight vector to obtain a fusion feature;

[0148] a modal correction unit 104, configured to generate a modal reconstruction feature according to the fusion feature, and generate a modal consistency error vector based on the modal reconstruction feature;

[0149] The early warning unit 105 is used to combine the confidence weight vector and the modal consistency error vector to determine an early warning signal of the current system, so as to judge whether the system has the risk of fusion degradation or modal failure.

[0150] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0151] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0152] In the description of this application, it should be noted that the terms "upper" and "lower" etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, or the orientations or positional relationships in which the invented product is usually placed when in use. These are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they should not be understood as limitations on this application.

[0153] It should also be noted that, in the description of this application, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A reliability self-diagnosis method for an autonomous driving system based on multimodal sensor fusion, characterized in that: The method comprises: S1. Obtain image modal data, use the visual Transformer model to extract features, and generate environmental scene labels to indicate the type of the current environment; S2. Obtain a modal score vector corresponding to each environmental scenario label by statistically analyzing key modal features in actual autonomous driving operation data. This score vector reflects the prior credibility of each sensor modality in that environment. Calculate the modal trust weight using softmax to obtain a confidence weight vector. The key modal features include image modal data, millimeter-wave radar target detection rate, and electromagnetic sensor short-range obstacle detection rate. S3. Performing weighted fusion on the key modal features based on the confidence weight vector to obtain a fusion feature; S4. Generating a modal reconstruction feature according to the fusion feature, and generating a modal consistency error vector based on the modal reconstruction feature; S5. Determine a warning signal for the current system based on the confidence weight vector and the modal consistency error vector, so as to judge whether the system has a risk of fusion degradation or modal failure.

2. The reliability self-diagnosis method of an autonomous driving system based on multimodal sensor fusion according to claim 1, characterized in that: The environmental scene labels include sunny environments with sufficient light during the day, foggy environments with severe visibility attenuation, rainy environments with water droplets and mirror reflection problems, low-light or no light environments at night, and entering tunnels or dimly lit enclosed spaces, with the characteristics of rapid changes between strong light and darkness.

3. The reliability self-diagnosis method of an autonomous driving system based on multimodal sensor fusion according to claim 1, characterized in that: S2 also includes: Obtain current key modal features and historical modal features; According to the current key modal features and historical modal features, a feedback correction value of each modal feature is calculated by a semantic difference measurement function, and a feedback correction term is obtained based on the feedback correction value of each modal feature, which is used to correct the modal score vector.

4. The reliability self-diagnosis method of an autonomous driving system based on multimodal sensor fusion according to claim 3, characterized in that: The modal trust weight is calculated by softmax to obtain the confidence weight vector, including: The modal score vector of each modality and the feedback correction term are combined, and a risk suppression term is introduced to calculate the modal trust weight through softmax, where the softmax maps the modal score vector of each modality to a weight in the range of [0,1] and ensures that the sum is 1.

5. The reliability self-diagnosis method of an autonomous driving system based on multimodal sensor fusion according to claim 4, characterized in that: The risk suppression item is used to pre-constrain the potentially unreliable modalities to prevent their trust weight from being too high in extreme cases.

6. The reliability self-diagnosis method of an autonomous driving system based on multimodal sensor fusion according to claim 1, characterized in that: Said S3 comprises: Using the confidence weight vector to perform weighted fusion on each modality feature in the spatial dimension to obtain a preliminary fusion feature; On the basis of the preliminary fusion features, based on the preliminary fusion features, each modal feature is cross-modally self-aligned through a spatial semantic guidance module based on the dot product attention mechanism and a scene structure maintenance regularization term to obtain an aligned feature vector, and the aligned feature vectors are fused to obtain a fusion feature.

7. The reliability self-diagnosis method of an autonomous driving system based on multimodal sensor fusion according to claim 6, characterized in that: The spatial semantic guidance module based on the dot product attention mechanism uses the preliminary fusion feature as the query and the corresponding modal feature vector as the key and value, and is used to learn that the modal feature vector is aligned to the global fusion representation; The scene structure preserving regularization term introduces a bias to the structure of the current modal feature vector according to the current environment scene label, so as to maintain dynamic consistency with the current environment scene label.

8. The reliability self-diagnosis method of an autonomous driving system based on multimodal sensor fusion according to claim 6, characterized in that: Generating a modal reconstruction feature according to the fusion feature, and generating a modal consistency error vector based on the modal reconstruction feature, specifically includes: A small reverse mapping module is designed for each modality to generate corresponding modality reconstruction features from the fusion features; After the reconstruction is completed, a combined error function is designed to measure the consistency between the reconstructed features of each modality and the aligned feature vector; A modal consistency error vector is obtained according to the combined error function of each modality, which represents the errors of the three modal characteristic vectors in the image modal data, the millimeter wave radar target detection rate and the electromagnetic sensor's detection rate of short-range obstacles.

9. The reliability self-diagnosis method of an autonomous driving system based on multimodal sensor fusion according to claim 8, characterized in that: Said S5 specifically includes: Obtain the confidence weight vector of the key feature vector to form the modal trust weight vector; Calculating a current system-level fusion consistency error based on the modal trust weight vector and the modal consistency error vector to represent the overall deviation degree between the fusion result and the original modal features; When the consistency error is greater than a preset global fixed threshold, it is considered that the current system is distorted and an early warning is triggered.

10. The reliability self-diagnosis platform of the autonomous driving system based on multimodal sensor fusion is characterized by: The system comprises: The modal perception unit is used to obtain image modal data, extract features using the visual Transformer model, and generate environmental scene labels to indicate the type of the current environment; A modal analysis unit is configured to obtain a modal score vector corresponding to each environmental scenario label by statistically analyzing key modal features in actual autonomous driving operation data. The score vector reflects the prior credibility of each sensor modality in that environment and calculates the modal trust weight through softmax to obtain a confidence weight vector. The key modal features include image modal data, millimeter-wave radar target detection rate, and electromagnetic sensor short-range obstacle detection rate. A modality fusion unit, configured to perform weighted fusion on the key modality features based on the confidence weight vector to obtain a fusion feature; a modal correction unit, configured to generate a modal reconstruction feature according to the fusion feature, and generate a modal consistency error vector based on the modal reconstruction feature; An early warning unit is used to combine the confidence weight vector and the modal consistency error vector to determine an early warning signal of the current system, so as to judge whether the system has the risk of fusion degradation or modal failure.

Images