Self-diagnosis method and platform for reliability of autonomous driving system based on multi-modal sensor fusion

Abstract

The application provides an automatic driving system reliability self-diagnosis method and platform based on multi-modal sensor fusion, which comprises the following steps: acquiring image modal data, using a visual Transformer model for feature extraction, generating an environment scene label to represent the type of the current environment; by statistically analyzing key modal features in actual automatic driving operation data, a modal score vector corresponding to each environment scene label is obtained, which is used to reflect the prior confidence of each sensor modal in the environment, a modal trust weight is calculated by using a softmax function, and a confidence weight vector is obtained; the key modal features are weighted and fused based on the confidence weight vector to obtain a fusion feature; a modal reconstruction feature is generated according to the fusion feature, and a modal consistency error vector is generated based on the modal reconstruction feature; and the confidence weight vector and the modal consistency error vector are combined to determine a warning signal of the current system, which is used to judge whether the system has a fusion degradation or modal failure risk.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving, and in particular relates to a self-diagnosis method and platform for the reliability of autonomous driving systems based on multimodal sensor fusion. Background Technology

[0002] The development of autonomous driving systems places extremely high demands on environmental perception capabilities. To improve the accuracy and robustness of perception, modern autonomous driving platforms generally employ multimodal sensor systems, including cameras, millimeter-wave radar, lidar, and electromagnetic sensors. By fusing perception data from different modalities, they comprehensively model objects, roads, and obstacles around the vehicle. The redundancy introduced by multimodal fusion theoretically enhances the fault tolerance of the perception system; however, in practical applications, the reliability of the perception system still faces significant challenges. Especially in variable environmental conditions, such as rainy / foggy weather, nighttime driving, and tunnel environments, the signal-to-noise ratio of different modalities changes drastically. Traditional fusion strategies typically use fixed weights or simple weighting based on single-moment features, lacking the ability to adapt to the current environment, leading to fusion failures in specific scenarios.

[0003] More critically, existing systems generally lack the ability to proactively monitor and model the health status of sensors, failing to promptly identify risks when sensors experience soft faults (such as image color distortion, radar delay, and electromagnetic interference). They often only issue reactive warnings when the system's perception results have already deviated significantly. Furthermore, most current autonomous driving perception platforms focus on "perception accuracy" rather than "perception reliability," making it difficult to determine whether the fusion results are based on reliable data sources even if they appear normal. Current technologies have not yet provided a mechanism to comprehensively consider scene changes, modal degradation, and the degree of information redundancy between modalities to dynamically assess the system's current reliability level, nor have they achieved real-time interpretation of perception quality and prediction of future trends. This gap directly impacts the safety and maintainability of the system in real-world road environments, especially under conditions of long-term operation, aging multi-sensor systems, or frequent complex environments, making it a critical issue that urgently needs to be addressed. Summary of the Invention

[0004] The purpose of this invention is to propose a self-diagnosis method and platform for the reliability of autonomous driving systems based on multimodal sensor fusion, which systematically solves the shortcomings of existing autonomous driving perception systems in soft fault identification, perception reliability assessment and health trend modeling.

[0005] To achieve the above objectives, a method for self-diagnostic reliability of an autonomous driving system based on multimodal sensor fusion is provided in a first aspect of the present invention, the method comprising:

[0006] S1. Obtain image modal data, extract features using the visual Transformer model, and generate environmental scene labels to represent the type of the current environment;

[0007] S2. By statistically analyzing the key modal features in the actual autonomous driving operation data, a modal score vector corresponding to each environmental scene label is obtained, which is used to reflect the prior confidence of each sensor mode in this environment. The modal trust weight is calculated by softmax to obtain the confidence weight vector. The key modal features include image modal data, millimeter-wave radar target detection rate and electromagnetic sensor detection rate at short distance obstacles.

[0008] S3. Based on the confidence weight vector, the key modal features are weighted and fused to obtain fused features;

[0009] S4. Generate modal reconstruction features based on the fusion features, and generate a modal consistency error vector based on the modal reconstruction features;

[0010] S5. Combining the confidence weight vector and the modal consistency error vector, determine the early warning signal of the current system to determine whether there is a risk of fusion degradation or modal failure in the system.

[0011] Furthermore, the environmental scene labels include sunny environments with sufficient daylight, hazy environments with severe visibility reduction, rainy environments with water droplets and specular reflections, low-light or no-light environments at night, and entering tunnels or dimly lit enclosed spaces, characterized by rapid changes between strong lighting and darkness.

[0012] Furthermore, S2 also includes:

[0013] Obtain current key modal features and historical modal features;

[0014] Based on the current key modal features and historical modal features, a feedback correction value for each modal feature is calculated using a semantic difference metric function. A feedback correction term is obtained based on the feedback correction value of each modal feature, which is used to correct the modal scoring vector.

[0015] Further, the step of calculating the modal confidence weights using softmax to obtain the confidence weight vector includes:

[0016] Combining the modality score vector and feedback correction term for each modality, and introducing a risk suppression term, modality trust weights are calculated using softmax, where softmax maps the modality score vector for each modality to a weight in the range [0,1], and ensures that the sum is 1.

[0017] Furthermore, the risk suppression term is used to pre-constrain potentially unreliable modes to prevent their trust weight from being too high in extreme cases.

[0018] Further, S3 includes:

[0019] The confidence weight vector is used to perform spatial dimensional weighted fusion of each modality feature to obtain preliminary fused features;

[0020] Based on the preliminary fusion features, a spatial semantic guidance module based on the dot product attention mechanism and a scene structure preservation regularization term are used to perform cross-modal self-alignment on each modal feature to obtain an aligned feature vector. The aligned feature vectors are then fused to obtain the fused features.

[0021] Furthermore, the spatial semantic guidance module based on the dot product attention mechanism uses the preliminary fusion features as the query, and the corresponding modal feature vectors as the key and value, to learn that the modal feature vectors are aligned to the global fusion representation;

[0022] The scene structure regularization term introduces a bias in the structure of the current modality feature vector based on the current environment scene label, in order to maintain dynamic consistency with the current environment scene label.

[0023] Further, the step of generating modal reconstruction features based on the fusion features and generating a modal consistency error vector based on the modal reconstruction features specifically includes:

[0024] A small inverse mapping module is designed for each modality to generate corresponding modality reconstruction features from the fused features;

[0025] After reconstruction is completed, a combined error function is designed to measure the consistency between each modal reconstruction feature and the aligned feature vector.

[0026] The modal consistency error vector is obtained based on the combined error function of each mode, representing the error of the three modal feature vectors in the image modal data, the target detection rate of the millimeter-wave radar, and the detection rate of the electromagnetic sensor in short-range obstacles.

[0027] Furthermore, S5 specifically includes:

[0028] Obtain the confidence weight vector of the key feature vectors to form the modal confidence weight vector;

[0029] The current system-level fusion consistency error is calculated based on the modal trust weight vector and the modal consistency error vector, which is used to represent the overall degree of deviation between the fusion result and the original modal features.

[0030] If the consistency error exceeds a preset global fixed threshold, the current system is considered to be distorted, and an early warning is triggered.

[0031] In a second aspect, the present invention provides a self-diagnostic platform for the reliability of an autonomous driving system based on multimodal sensor fusion, the platform comprising:

[0032] The modality sensing unit is used to acquire image modality data, extract features using a visual Transformer model, and generate environmental scene labels to represent the type of the current environment.

[0033] The modal analysis unit is used to obtain a modal score vector corresponding to each environmental scene label by statistically analyzing key modal features in actual autonomous driving operation data. This vector reflects the prior confidence of each sensor modality in that environment. The modal trust weight is calculated using softmax to obtain a confidence weight vector. The key modal features include image modal data, millimeter-wave radar target detection rate, and electromagnetic sensor detection rate at short-range obstacles.

[0034] A modality fusion unit is used to perform weighted fusion of the key modal features based on the confidence weight vector to obtain fused features;

[0035] A modality correction unit is used to generate modality reconstruction features based on the fusion features, and to generate a modality consistency error vector based on the modality reconstruction features;

[0036] The early warning unit is used to combine the confidence weight vector and the modal consistency error vector to determine the early warning signal of the current system, which is used to determine whether there is a risk of fusion degradation or modal failure in the system.

[0037] The beneficial technical effects of the present invention are at least as follows:

[0038] This invention establishes a dynamic fusion control mechanism that can automatically adjust the trust weights of different sensor modalities during the fusion process by identifying the current driving environment, thereby avoiding reliance on unreliable data in certain modal degradation scenarios. At the same time, this invention designs an intermodal information consistency analysis method that uses semantic reconstruction errors between multimodalities to identify potential soft faults, revealing internal performance degradation even when the sensor surface output is normal.

[0039] Furthermore, this invention introduces a lightweight trend modeling module, which utilizes the temporal evolution characteristics of modal diagnostic indicators to predict the future health trends of each modality, enabling early warning and adaptive strategy adjustment. Through the integration of the above multi-level diagnostic and prediction mechanisms, this invention constructs a "perception credibility-driven" intelligent perception platform that can maintain the stability of the perception system in complex environments, possess strong interpretability, and have fault tolerance and prevention capabilities, significantly improving the perception safety and self-healing capabilities of autonomous driving systems. Attached Figure Description

[0040] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0041] Figure 1 This is a flowchart of the self-diagnosis method for the reliability of autonomous driving systems based on multimodal sensor fusion, as described in this invention.

[0042] Figure 2 This is a framework diagram of the self-diagnosis platform for the reliability of autonomous driving systems based on multimodal sensor fusion, as described in this invention. Detailed Implementation

[0043] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0044] like Figure 1 As shown, this embodiment of the invention provides a self-diagnosis method for the reliability of an autonomous driving system based on multimodal sensor fusion. The method includes:

[0045] S1. Obtain image modal data, extract features using the visual Transformer model, and generate environmental scene labels to represent the type of the current environment.

[0046] Specifically, obtain the input data:

[0047] Image modal data This is image data acquired through a camera, with dimensions of H×W×3, representing the height, width, and information of the three RGB channels of each frame. The camera images have undergone normalization and noise reduction processing to ensure data quality.

[0048] Furthermore, the goal of this step is to input image data via a camera. Deep learning models are used to process and analyze images, identify the current scene type, and output scene labels S. t This process extracts image features based on a visual Transformer model, and then classifies them into specific scene types (such as "sunny day," "rainy day," "tunnel," etc.) using a classifier. Scene label S t This provides crucial information for subsequent modal trust adjustment.

[0049] First, image data The images undergo preprocessing, including noise reduction, color normalization, and resizing. This ensures the images meet the requirements for subsequent processing. For image data in autonomous driving, cameras may capture diverse environmental scenes (such as sunny days, nights, and foggy weather), therefore data augmentation techniques are used to process image data from these different environments.

[0050] Next, the image data undergoes feature extraction using a Visual Transformer (ViT) model. This model segments the input image into multiple fixed-size patches and processes these patches using a self-attention mechanism to extract spatial and contextual information from the image. The output of the Transformer is a deep feature representation of the image. This representation can effectively capture semantic information in an image.

[0051] These image features are then fed into a fully connected layer (classifier), which outputs a probability distribution for each scene category. The probability distribution is normalized using a softmax activation function, ultimately outputting the scene label S with the highest probability. t For example, if the model infers a probability of 0.7 for "sunny day," 0.2 for "rainy day," and 0.1 for "tunnel," the system will select label S. t ="clear".

[0052] The specific classification process can be represented by the following formula:

[0053]

[0054] in, Representing an image The semantic feature vector W extracted by the visual Transformer model (such as BEiT) cls and b cls These represent the weights and biases of the classifier, respectively. Softmax is a function that normalizes the outputs of multiple classes, and argmax represents selecting the class with the highest probability as the scene label S. t .

[0055] Output variable: St The scene label indicates the type of the current environment. This label is selected from the following fixed set:

[0056] S t ∈"clear","fog","rain","night","tunnel"

[0057] "clear": a sunny environment with plenty of daylight;

[0058] "fog": A smoggy environment with significant visibility reduction;

[0059] "rain": A rainy environment with problems such as water droplets and specular reflection;

[0060] "night": a low-light or dark environment at night;

[0061] "tunnel": Entering a tunnel or a dark, enclosed space, characterized by rapid changes between bright light and darkness.

[0062] Understandably, this step extracts depth features from the input image using a visual Transformer model and combines this with a classifier to identify the current scene type. This scene label provides crucial information for subsequent modality trust adjustment, enabling the system to dynamically adjust the trust weights of each modality based on different environments.

[0063] S2. By statistically analyzing the key modal features in the actual autonomous driving operation data, a modal score vector corresponding to each environmental scene label is obtained, which is used to reflect the prior confidence of each sensor mode in this environment. The modal trust weight is calculated by softmax to obtain the confidence weight vector. The key modal features include image modal data, millimeter-wave radar target detection rate and electromagnetic sensor detection rate of short-range obstacles.

[0064] Specifically, the input for this step is the environment scene label S output from step one. t Additionally, it should be noted that in this system, "millimeter-wave radar" specifically refers to a high-frequency electromagnetic sensor used for mid-to-long-range target detection, operating in the 24GHz or 77GHz frequency band. These sensors are deployed at the front and rear of the vehicle, primarily for forward obstacle detection and speed estimation. "Electromagnetic sensors," on the other hand, refer to short-range, non-imaging electromagnetic-assisted sensing devices, such as capacitive / inductive near-field detectors, ultra-wideband (UWB) miniature radars around the vehicle, and near-field sensing modules. These are mainly deployed on wheel arches, side skirts, or the edges of the vehicle's undercarriage, and are commonly used for detecting objects in blind spots or at extremely close range. These two types of sensors differ significantly in their operating range, signal characteristics, and deployment scenarios; therefore, they are considered two independent modes in this step.

[0065] In autonomous driving systems, the reliability of multiple sensor modalities is significantly affected by the environment. To achieve a dynamic, scene-adaptive perception fusion strategy, this step involves using the current scene label S... t Generate a set of modal trust weights This invention proposes a joint scoring mechanism consisting of three parts: static environmental scoring, modal consistency feedback correction term, and risk suppression regularization term. This mechanism not only reflects the system's understanding of the environment but also considers the current modal performance and fault-tolerant control under risky scenarios. This is used for feature weighting fusion in the next step.

[0066] The system provides each fixed scenario S t Define a nonnormalized modal score vector This vector reflects the prior confidence level of each sensor mode in this environment. The score is obtained through statistical analysis of performance metrics from actual autonomous driving operation data. For example, in the "fog" scenario, the system extracts the following key performance metrics through analysis of over 120 hours of onboard data:

[0067] The camera image clarity is severely degraded, and the Canny edge detection rate is below 30%. cam =0.2;

[0068] The millimeter-wave radar target detection rate remains above 88%, assuming s radar =0.9;

[0069] Electromagnetic sensors (such as UWB radar) remain stable in short-range obstacle detection, assuming s em =0.85.

[0070] Based on this, the corresponding scores for each scenario are as follows:

[0071] m clear =[0.8,0.7,0.6]

[0072] m fog =[0.2,0.9,0.85]

[0073] m rain =[0.4,0.8,0.75]

[0074] m night =[0.1,0.7,0.65]

[0075] m tunnel =[0.3,0.6,0.7]

[0076] These scores preserve the performance differences between modalities and will serve as the basis for trust trends before softmax normalization.

[0077] To enhance adaptability, the system also calculates the semantic consistency between modal outputs and historical consistency as a feedback correction term. Its definition is as follows:

[0078]

[0079] in, Let m be the feature vector of modality m output by the perceptual network in the i-th frame. The historical mean feature of this mode is calculated by performing a moving average of the features output by mode m over the past 64 frames. Specifically, at each time t, the system averages the feature values ​​from the past 64 frames to obtain a stable historical feature value. This avoids the impact of anomalies in a single frame. Divergence(·) is a semantic difference metric (such as cosine distance or KL divergence), and λ is the adjustment coefficient. The system uses a sliding window to average historical consistency, thereby avoiding fluctuations in trust weights caused by occasional anomalies. For example, if the camera modality is unstable in the last 5 frames, its correction value... The value will be negative, thus reducing their trust.

[0080] Furthermore, in high-risk scenarios, even if some modes currently have stable outputs, physical defects may still increase the system risk. Therefore, a risk mitigation term is introduced into the system. This setting is configured for each type of scenario, such as:

[0081] In the "night" and "fog" scenarios,

[0082] In the "tunnel" scenario,

[0083] Electromagnetic modes are stable in most scenarios and are typically assumed to be stable.

[0084] These terms incorporate a penalty factor into the calculation to pre-constrain potentially unreliable modes and prevent their trust weights from becoming too high in extreme cases.

[0085] Finally, the system combines the above three scores and calculates the modal trust weight using softmax:

[0086]

[0087] Here, α is the adjustment coefficient of the risk regularization term, which controls its influence range. The softmax operation maps the three modality scores to weights in the range [0,1] and ensures that the sum is 1, which meets the requirements of subsequent weighted fusion.

[0088] Understandably, the output is a modal trust weight vector. Each element is a normalized weight value, representing the system's current level of trust in that modality. This output will serve as the direct input for the next step, "modal feature fusion," guiding the multimodal perception system to dynamically adjust its fusion strategy in different scenarios.

[0089] S3. The key modal features are weighted and fused based on the confidence weight vector to obtain fused features.

[0090] Specifically, firstly, this invention uses a confidence weight vector W t We perform spatial weighted fusion of the three modal features. To do this, we first map all modal features to the same dimension d through a shared 1×1 convolution to obtain standardized features. Then, a weighted summation is performed to construct the initial fusion feature F. t :

[0091]

[0092] in, This represents the modal confidence weights output from step two. It is a modality-aware feature tensor aligned with feature dimensions. This formula ensures that the contribution of each modality in the fusion is directly related to its credibility in the current scene, thereby avoiding weak modalities from dominating the fusion output at critical moments.

[0093] However, relying solely on weighted summation cannot solve two key problems: 1) spatial semantic misalignment between modalities; and 2) imbalance due to uneven perceptual distribution within the main modality in extreme scenarios. Therefore, this step proposes a structural alignment enhancement term to fuse features F. t To guide this process, cross-modal self-alignment is performed on features across different modalities:

[0094]

[0095] Attn align (·) is a spatial semantic guidance module based on the dot product attention mechanism, where F t As a query, modality m has key and value features, which are used to learn how modality features should be "pulled" into the global fusion representation;

[0096] It is an innovative scenario structure that maintains regularity, based on the current scenario S. t This introduces biases to preserve modal structure. For example, it enhances vertically continuous edge information in the "tunnel" scene and preserves dynamic consistency of time-stable regions in the "rain" scene.

[0097] Specifically, for regularization terms, this invention implements a set of scenario-specific regularization weight templates. This indicates the structural importance of each pixel in the current scene. Essentially, this term penalizes the difference between the local gradient on the modality feature map and the structural gradient of the global fused feature map, ensuring that the modality is not diluted by the fusion mechanism in its "dominant region" during the fusion process.

[0098] This mechanism effectively integrates three types of objectives:

[0099] Use confidence weighting to improve overall robustness;

[0100] Achieving modal semantic consistency using attention alignment;

[0101] Use structural regularization to maintain the physical continuity and scene adaptability of critical structural regions.

[0102] After the above processing, the final fused feature F is output. t This serves as the input for the downstream diagnostic module.

[0103] Understandably, the output of this step is a fused sensing feature tensor. It integrates the three modes of the current moment in environment S t It preserves spatial semantic structure by processing perceived information, and achieves high semantic consistency and structural fidelity through modal alignment and scene regularization. t This will serve as the input signal for the next step, "anomaly detection and fault prediction," supporting further diagnostic analysis.

[0104] S4. Generate modal reconstruction features based on the fusion features, and generate a modal consistency error vector based on the modal reconstruction features.

[0105] Specifically, this step aims to construct a set of evaluation indexes specifically for diagnosing modal health and fusion consistency, by analyzing fusion characteristics F... t The mechanism reversely "reconstructs" the features of each modality and then compares them with the true modality features to calculate the semantic consistency error. Essentially, this is a fusion-based reverse mapping modeling process that does not involve behavioral decisions but generates diagnostic indicators to support subsequent fault prediction and credibility assessment. Its innovation lies in elevating modality consistency evaluation from a "modal-to-modal" comparison to a "fusion-to-modal" reverse interpretation path, making the behavior of the fusion model interpretable.

[0106] First, design a small inverse mapping module for each mode m (m∈{cam,radar,em}). Used from fusion feature F t Generated modal reconstruction features The structure is as follows:

[0107] Two 1×1 convolutional layers (with channels d→d / 2→d respectively);

[0108] Each layer is followed by ReLU activation;

[0109] Add local self-attention (such as local window attention or simple convolutional attention mechanism) after the last layer to enhance local structure sensitivity.

[0110] Furthermore, the formula for the reconstruction process is as follows:

[0111]

[0112] The structure is lightweight, has few shared parameters, is easy to deploy, and supports parallel reconfiguration of different modalities.

[0113] After reconstruction, this invention uses a combined error function. Measure each modal reconstruction feature With original features Consistency between them:

[0114]

[0115] Among them, item 1 Content error, representing the Euclidean distance in feature space;

[0116] Item 2 To address structural errors, the Sobel operator is used to extract horizontal and vertical gradient maps, reflecting the consistency of modes in spatial structure; λ s Weights of structural items (generally set to) (It can be adaptively adjusted according to the scene, such as increasing the weight of this item on foggy days).

[0117] It is important to note that the entire process requires no labels and does not involve external models. All modules can be trained end-to-end and can also be used independently during the inference phase.

[0118] For example, in a "night" scenario, if the camera modality experiences reduced visibility, its With reality If the difference increases and the structural error term also rises abnormally, the system will record... Significantly higher than normal values. If a historical moving average or modal health baseline is deployed in conjunction with this value, it is used to indicate that the modality may have temporary failures or fusion biases.

[0119] S5. Combining the confidence weight vector and the modal consistency error vector, determine the early warning signal of the current system to determine whether there is a risk of fusion degradation or modal failure in the system.

[0120] Specifically, the input for this step is:

[0121] The modal consistency error vector output in step four

[0122] The modal trust weight vector output in step two

[0123] in This represents the structural consistency error of mode m. This indicates the level of trust the system currently has in this modality.

[0124] The goal of this step is to directly calculate the overall fusion consistency state of the system using existing modal error information and trust weights, and output a clear early warning signal to determine whether the system has the risk of fusion degradation or modal failure. To avoid over-modeling and complex trend processing, this step adopts a weighted error determination mechanism to ensure that the system is executable, lightweight in deployment, and logically consistent.

[0125] The system is based on the structural consistency error of the three modes. Its trust weight Calculate the current system-level fusion consistency error

[0126]

[0127] This value represents the overall deviation between the fusion result and the original modal features. The fusion weight reflects the importance of each modality to the system in the current scenario.

[0128] The system sets a global fixed threshold τ (e.g., τ = 0.35) to determine whether there are any abnormal deviations in the current fusion state:

[0129]

[0130] in For indicator functions, if If the threshold is exceeded, the system considers the current perception fusion to be significantly distorted and triggers an early warning.

[0131] For example:

[0132] If the current trimodal error is {0.25, 0.12, 0.30} and the confidence weight is {0.5, 0.3, 0.2}, then calculate:

[0133]

[0134] Since 0.225 < τ, the system does not trigger an early warning.

[0135] If the error is {0.48, 0.36, 0.42} and the weight is {0.6, 0.2, 0.2}, then:

[0136]

[0137] Because 0.456 > τ, the system triggers an early warning.

[0138] Final output: System fusion error score:

[0139]

[0140] This indicates the degree of consistency deviation in the current system integration;

[0141] System warning signal:

[0142]

[0143] This indicates whether the system has triggered an abnormal warning for perception fusion.

[0144] To better implement the above methods, such as Figure 2 As shown, this invention also proposes a self-diagnostic platform for the reliability of autonomous driving systems based on multimodal sensor fusion, based on any of the above-described multimodal perception methods for road monitoring adapted to dynamic environments, the platform comprising:

[0145] The modality perception unit 101 is used to acquire image modality data, perform feature extraction using a visual Transformer model, and generate environmental scene labels to represent the type of the current environment.

[0146] The modal analysis unit 102 is used to obtain a modal score vector corresponding to each environmental scene label by statistically analyzing key modal features in actual autonomous driving operation data. This vector reflects the prior confidence of each sensor modality in the environment. The modal trust weight is calculated by softmax to obtain a confidence weight vector. The key modal features include image modal data, millimeter-wave radar target detection rate, and electromagnetic sensor detection rate at short-range obstacles.

[0147] Modality fusion unit 103 is used to perform weighted fusion of the key modality features based on the confidence weight vector to obtain fused features;

[0148] The modality correction unit 104 is used to generate modality reconstruction features based on the fusion features, and to generate a modality consistency error vector based on the modality reconstruction features;

[0149] The early warning unit 105 is used to combine the confidence weight vector and the modal consistency error vector to determine the early warning signal of the current system, and to determine whether there is a risk of fusion degradation or modal failure in the system.

[0150] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application.

[0151] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0152] In the description of this application, it should be noted that the terms "upper" and "lower" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use. They are only used to facilitate the description of this application and to simplify the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0153] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A self-diagnosis method for the reliability of an autonomous driving system based on multimodal sensor fusion, characterized in that, The method includes: S1. Obtain image modal data, extract features using the visual Transformer model, and generate environmental scene labels to represent the type of the current environment; S2. By statistically analyzing the key modal features in the actual autonomous driving operation data, a modal score vector corresponding to each environmental scene label is obtained, which is used to reflect the prior confidence of each sensor mode in this environment. The modal trust weight is calculated by softmax to obtain the confidence weight vector. The key modal features include image modal data, millimeter-wave radar target detection rate and electromagnetic sensor detection rate at short distance obstacles. S3. Based on the confidence weight vector, the key modal features are weighted and fused to obtain fused features; S4. Generate modal reconstruction features based on the fusion features, and generate a modal consistency error vector based on the modal reconstruction features; S5. Combining the confidence weight vector and the modal consistency error vector, determine the early warning signal of the current system to determine whether there is a risk of fusion degradation or modal failure in the system; The step of calculating modal confidence weights using softmax to obtain the confidence weight vector includes: Combining the modality score vector and feedback correction term for each modality, and introducing a risk suppression term, modality trust weights are calculated using softmax, where the softmax maps the modality score vector for each modality to... The weights are set within the range, and the sum is guaranteed to be 1; the risk suppression term is used to pre-constrain potentially unreliable modes to prevent their trust weights from being too high in extreme cases. The S3 includes: The confidence weight vector is used to perform spatial dimensional weighted fusion of each modality feature to obtain preliminary fused features; Based on the preliminary fusion features, a spatial semantic guidance module based on the dot product attention mechanism and a scene structure preservation regularization term are used to perform cross-modal self-alignment on each modal feature to obtain aligned feature vectors. The aligned feature vectors are then fused to obtain fused features. The spatial semantic guidance module based on the dot product attention mechanism uses the preliminary fusion features as the query, and the corresponding modal feature vectors as the key and value, to learn how the modal feature vectors are aligned to the global fusion representation; The scene structure regularization term introduces a bias in the structure of the current modality feature vector based on the current environment scene label, in order to maintain dynamic consistency with the current environment scene label.

2. The self-diagnosis method for reliability of autonomous driving systems based on multimodal sensor fusion according to claim 1, characterized in that, The environmental scene labels include sunny environments with sufficient daylight, smoggy environments with severe visibility reduction, rainy environments with water droplets and specular reflections, low-light or no-light environments at night, and entering tunnels or dimly lit enclosed spaces, characterized by rapid changes between strong lighting and darkness.

3. The self-diagnosis method for reliability of autonomous driving systems based on multimodal sensor fusion according to claim 1, characterized in that, S2 also includes: Obtain current key modal features and historical modal features; Based on the current key modal features and historical modal features, a feedback correction value for each modal feature is calculated using a semantic difference metric function. A feedback correction term is obtained based on the feedback correction value of each modal feature, which is used to correct the modal scoring vector.

4. The self-diagnosis method for reliability of autonomous driving systems based on multimodal sensor fusion according to claim 1, characterized in that, The step of generating modal reconstruction features based on the fusion features, and generating a modal consistency error vector based on the modal reconstruction features, specifically includes: A small inverse mapping module is designed for each modality to generate corresponding modality reconstruction features from the fused features; After reconstruction is completed, a combined error function is designed to measure the consistency between each modal reconstruction feature and the aligned feature vector. The modal consistency error vector is obtained based on the combined error function of each mode, representing the error of the three modal feature vectors in the image modal data, the target detection rate of the millimeter-wave radar, and the detection rate of the electromagnetic sensor in short-range obstacles.

5. The self-diagnosis method for reliability of autonomous driving system based on multimodal sensor fusion according to claim 4, characterized in that, S5 specifically includes: Obtain the confidence weight vector of the key feature vectors to form the modal confidence weight vector; The current system-level fusion consistency error is calculated based on the modal trust weight vector and the modal consistency error vector, which is used to represent the overall degree of deviation between the fusion result and the original modal features. If the consistency error exceeds a preset global fixed threshold, the current system is considered to be distorted, and an early warning is triggered.

6. A platform for implementing the self-diagnosis method for reliability of an autonomous driving system based on multimodal sensor fusion as described in claim 1, characterized in that, The platform includes: The modality sensing unit is used to acquire image modality data, extract features using a visual Transformer model, and generate environmental scene labels to represent the type of the current environment. The modal analysis unit is used to obtain a modal score vector corresponding to each environmental scene label by statistically analyzing key modal features in actual autonomous driving operation data. This vector reflects the prior confidence of each sensor modality in that environment. The modal trust weight is calculated using softmax to obtain a confidence weight vector. The key modal features include image modal data, millimeter-wave radar target detection rate, and electromagnetic sensor detection rate at short-range obstacles. A modality fusion unit is used to perform weighted fusion of the key modal features based on the confidence weight vector to obtain fused features; A modality correction unit is used to generate modality reconstruction features based on the fusion features, and to generate a modality consistency error vector based on the modality reconstruction features; The early warning unit is used to combine the confidence weight vector and the modal consistency error vector to determine the early warning signal of the current system, which is used to determine whether there is a risk of fusion degradation or modal failure in the system.

Images