A head-up display multi-modal anomaly testing method, system, device, and medium

Abstract

The application provides a head-up display multi-modal anomaly test method, system, device and medium, and relates to the technical field of automobile electronics. The method comprises the following steps: determining a prototype vector matched with a current multi-modal fusion vector in a prototype library; performing abnormal score processing on each mode of the multi-modal fusion vector and the matched prototype vector respectively to obtain abnormal score values corresponding to each mode; determining a corresponding target distribution result according to an abnormal score value sequence of a target mode in a first time length, wherein the target mode is any one of the multi-modal modes; performing verification on the target distribution result and a baseline distribution of the target mode by using a statistical test method of distribution difference maximization to determine a mode drift amount of the target mode; updating an alarm threshold value according to the mode drift amount of the target mode; and determining a test result of the target mode according to the abnormal score value of the target mode and the updated alarm threshold value. The application aims to improve the abnormal test accuracy of a head-up display system.

Description

Technical Field

[0001] This application relates to the field of automotive electronics technology, specifically to a multimodal anomaly testing method, system, device, and medium for head-up displays. Background Technology

[0002] With the rapid development of intelligent connected vehicles, the Head-Up Display (HUD), as the core interactive unit of the intelligent cockpit, has seen a significant increase in functional complexity and reliability requirements. Traditional HUD testing methods are based on a distributed electronic and electrical architecture, primarily relying on CAN bus communication for signal interaction. Test systems typically consist of independent CAN devices, a manual operating interface, and semi-automated scripts. Specifically, the testing process requires manual configuration of test cases using a CAN signal simulator, and verification of the HUD projected content (such as vehicle speed, navigation icons, and warning lights) relies on manual visual inspection or basic image processing algorithms. These traditional solutions include template comparison, fixed thresholds, single-modality testing, and pure post-testing backtracking. Because these traditional solutions mostly test the HUD in a single modality, their testing accuracy and robustness are insufficient. Summary of the Invention

[0003] In view of this, this application provides a multimodal anomaly testing method, system, device, and medium for head-up displays. It aims to solve or partially solve the problems existing in the prior art.

[0004] The first aspect of this application provides a multimodal anomaly testing method for heads-up displays, the method comprising:

[0005] Determine the prototype vector in the prototype library that matches the current multimodal fusion vector. The multimodal fusion vector is obtained by embedding vector fusion of multimodal data related to head-up display. The prototype vector in the prototype library is a standard multimodal fusion vector corresponding to the desired display state.

[0006] For the multimodal fusion vector and the matching prototype vector, anomaly scoring processing is performed on each modality to obtain the anomaly score value corresponding to each modality.

[0007] Based on the abnormal score value sequence of the target modality within the first time period, the corresponding target distribution result is determined, wherein the target modality is any one of the multimodal modalities;

[0008] The target distribution result and baseline distribution of the target modality are verified by the statistical test method of maximizing the distribution difference, and the modal drift of the target modality is determined. The baseline distribution is the distribution of the target modality when it is normal in the corresponding expected display state.

[0009] Update the alarm threshold based on the modal drift of the target mode;

[0010] determine a test result of the target modality according to the abnormal score value of the target modality and the current updated alarm threshold.

[0011] The second aspect of the present application provides a head-up display multi-modal anomaly test system, the system comprising:

[0012] a prototype vector determination module, configured to determine a prototype vector in a prototype library that matches a current multi-modal fusion vector, the multi-modal fusion vector being obtained by fusing embedding vectors of multi-modal data related to the head-up display, the prototype vector in the prototype library being a standard multi-modal fusion vector corresponding to an expected display state;

[0013] an abnormal score module, configured to perform abnormal score processing of each modality on the multi-modal fusion vector and the matched prototype vector respectively, to obtain abnormal score values corresponding to each modality;

[0014] a target distribution result determination module, configured to determine a corresponding target distribution result according to a sequence of abnormal score values of a target modality within a first time length, the target modality being any one of the multi-modal;

[0015] a modality drift amount determination module, configured to perform a check on a target distribution result of the target modality and a baseline distribution by a distribution difference maximization statistical test method, to determine a modality drift amount of the target modality, the baseline distribution being a distribution of the target modality when normal under a corresponding expected display state;

[0016] an alarm threshold updating module, configured to update an alarm threshold according to the modality drift amount of the target modality;

[0017] an evaluation module, configured to determine a test result of the target modality according to the abnormal score value of the target modality and the current updated alarm threshold.

[0018] The third aspect of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored on the memory and running on the processor, the computer program being executed by the processor to implement the steps of the head-up display multi-modal anomaly test method according to the first aspect of the present application.

[0019] The fourth aspect of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of the head-up display multi-modal anomaly test method according to the first aspect of the present application.

[0020] The head-up display multi-modal anomaly test method provided by the present application has the following advantages:

[0021] The head-up display multi-modal anomaly test method provided by the application first determines a prototype vector in a prototype library that matches a current multi-modal fusion vector, the multi-modal fusion vector is obtained by fusing embedding vectors of multi-modal data related to the head-up display, and the prototype vector in the prototype library is a standard multi-modal fusion vector corresponding to an expected display state. The multi-modal fusion vector and the matched prototype vector are subjected to abnormal score processing of each modality respectively to obtain abnormal score values corresponding to each modality. A target distribution result corresponding to a target modality is determined according to a sequence of abnormal score values of the target modality within a first time length, the target modality being any one of the multi-modal. The target distribution result of the target modality and a baseline distribution are checked by a statistical test method of distribution difference maximization to determine a modality drift of the target modality, the baseline distribution being a distribution of the target modality when normal under the corresponding expected display state. An alarm threshold is updated according to the modality drift of the target modality. A test result of the target modality is determined according to the abnormal score value of the target modality and the currently updated alarm threshold. Thus, the method provided by the application improves the accuracy and robustness of anomaly detection by fusing multi-modal data related to the head-up display for anomaly detection of the head-up display, and further improves the accuracy of anomaly detection of the head-up display by filtering out inherent and normal random fluctuations in the anomaly detection process through drift monitoring of each modality data. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the application. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 A flowchart of a head-up display multi-modal anomaly test method according to an embodiment of the application is shown.

[0024] Figure 2 A schematic diagram of a head-up display multi-modal anomaly test system according to an embodiment of the application is shown. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the application will be described clearly and completely in the following with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0026] REFERENCE Figure 1 ,Figure 1 A flow chart of a head-up display multi-modal anomaly test method according to an embodiment of the present application is shown in FIG. 1, which includes the following steps: Figure 1

[0027] Step S1: Determine the prototype vector in the prototype library that matches the current multi-modal fusion vector, the multi-modal fusion vector being obtained by fusing the embedding vectors of multi-modal data related to the head-up display, and the prototype vector in the prototype library being a standard multi-modal fusion vector corresponding to the expected display state.

[0028] In this embodiment, the present application pre-establishes a prototype library, which contains a large number of prototype samples, one prototype sample corresponding to one expected display state, and the prototype samples in the prototype library at least including the standard multi-modal fusion vector of the head-up display system in the expected display state, which is referred to as the prototype vector in the present application. The expected display state is for the head-up display system, i.e. the expected display state is a display state of the head-up display system, such as a navigation display state, a virtual character display state, etc.

[0029] In this embodiment, during the anomaly test process, the head-up display system is first controlled to work and display. During the display process, multi-modal data related to the head-up display is collected in real time, and then the current collected data of each modality is processed into the respective corresponding embedding vectors and fused to obtain the current multi-modal fusion vector. The prototype vector in the prototype library is also a multi-modal fusion vector obtained by fusing the embedding vectors of multiple modalities corresponding to the multi-modal data, except that the prototype vector in the prototype library is a standard multi-modal fusion vector corresponding to the expected display state, i.e. the multi-modal fusion vector corresponding to the normal display of the head-up display system in the expected display state. The time window of one piece of multi-modal data in the present application is preferably 1 second, and the step is 100 ms.

[0030] ​In this embodiment, the multi-modal data related to the heads-up display includes at least image data, text sequence data, signal data, and voice data. The image data refers to the HUD projection image projected by the heads-up display system to the front windshield, and the HUD projection image (such as a 1920x720 RGB image) is collected in real time by an industrial camera. The text sequence data is a sequence composed of characters extracted from the projection image. The signal data includes CAN signal data, LIN signal data, DDS (Data Distribution Service) service logs, etc. The CAN signal data and the LIN signal data are collected by a signal collection card built in a HIL test bench (Hardware-in-the-Loop Test Bench), with a sampling period of 10 ms, supporting signal timestamp binding, and the DDS service logs are collected by an embedded log service to collect service response time, timeout alarm, etc. The voice data is the voice output by the user, which is collected by a microphone array in a soundproof box and converted into a text sequence by an ASR module (Automatic Speech Recognition).

[0031] In this embodiment, after obtaining the current multi-modal fusion vector through multi-modal fusion, the multi-modal fusion vector is matched with the prototype vectors in the prototype library to determine the prototype vector in the prototype library that is closest to the current multi-modal fusion vector. The closest prototype vector determined at this time belongs to the standard quantity of the current multi-modal fusion vector.

[0032] Step S2: Perform abnormal scoring processing on each modality for the multi-modal fusion vector and the matched prototype vector, respectively, to obtain abnormal scoring values corresponding to each modality.

[0033] In this embodiment, after determining the prototype vector matched with the current multi-modal fusion vector through step S1, abnormal scoring processing is performed on the two vectors. The specific processing process is as follows: the embedding vectors of each modality in the current multi-modal fusion vector are respectively subjected to abnormal scoring processing with the embedding vectors of the corresponding modality in the matched prototype vector, so as to obtain the abnormal scores under each modality. That is, when performing abnormal scoring processing of the modalities, the embedding vectors of the same modality in the two multi-modal fusion vectors (i.e., the current multi-modal fusion vector and the matched prototype vector) are subjected to abnormal scoring data.

[0034] In this embodiment, the multi-modal fusion vector is used to determine the matched prototype vector, and the embedding vectors of the single modality in the unified semantic space in the multi-modal fusion vector are used to determine the abnormal scores.

[0035] Step S3: determining a target distribution result according to the abnormal score value sequence of the target modality in the first time length, the target modality being any one of the multiple modalities.

[0036] In this embodiment, after obtaining the abnormal score value of each modality at the corresponding time through step S2, the abnormal score value of each modality at multiple time is determined through the same implementation. After obtaining multiple abnormal score values of each modality in the first time length, the distribution of the abnormal score value of each modality is determined, wherein the first time length can be set according to the actual scene, which is not limited here, such as 5 seconds, 10 seconds, etc. Since the distribution of the abnormal score value of each modality is determined, and the subsequent implementation of determining the modal drift of each modality and the subsequent abnormal test of each modality is the same, for the convenience of subsequent description and understanding of this application, the subsequent description is taken as an example of one modality, which is called target modality for the convenience of description. Specifically: after obtaining the abnormal score value sequence of the target modality in the first time length (the abnormal score value sequence is composed of all abnormal score values of the target modality in the first time length in time sequence), the distribution of the abnormal score sequence is obtained to obtain the corresponding target distribution result. The target distribution result of each modality is obtained through the same implementation.

[0037] Step S4: verifying the target distribution result of the target modality and the baseline distribution through the statistical test method of distribution difference maximization, and determining the modal drift of the target modality, the baseline distribution being the distribution of the target modality in the corresponding expected display state.

[0038] In this embodiment, after obtaining the target distribution result of each modality through step S3, the target distribution result of the target modality and the baseline distribution are verified through the statistical test method of distribution difference maximization, and the modal drift of the target modality is determined. Wherein, the statistical test method of distribution difference maximization is preferably KS test. Wherein, the baseline distribution is the theoretical distribution of the target modality in the corresponding expected display state, i.e. the baseline distribution comes from a period of reference data confirming "all normal", and the standard verification before release is preferred, if there is no such data, the abnormal score sequence of the first stable period (such as no alarm for a certain number of minutes, KS does not trigger drift) after online is taken to determine the baseline distribution. The same method is used for the statistics of the target distribution result and the baseline distribution, and is saved with the version number for subsequent comparison. The modal drift of each modality is obtained through the same implementation.

[0039] Step S5: updating the alarm threshold according to the modal drift of the target modality.

[0040] In the embodiment, after the modal drift of each modality is obtained through step S4, the alarm threshold for the target modality anomaly detection is updated based on the modal drift of the target modality, so as to filter out the normal random fluctuations inherent in the anomaly detection process. An optional implementation is that a drift threshold corresponding to the target modality is set, and when it is determined that the modal drift of the target modality is greater than the drift threshold for a continuous preset number of times, it is determined that the target modality has indeed drifted, and the alarm threshold of the target modality is updated at this time; and when the modal drift of the target modality does not satisfy the condition that it is greater than the drift threshold for a continuous preset number of times, it is determined that the target modality has not drifted, and the alarm threshold of the target modality is not updated at this time, and the original alarm threshold is still used for anomaly detection of the target modality. The preset number of times can be set according to the actual scene, which is not limited here, such as 3 times, 5 times, etc. Through the same implementation, the alarm threshold corresponding to each modality can be updated, judged and updated.

[0041] Step S6: determining the test result of the target modality according to the anomaly score value of the target modality and the current updated alarm threshold.

[0042] In the embodiment, step S5 is to determine whether to update the alarm threshold based on the modal drift of the target modality, and to update the alarm threshold when the update of the alarm threshold is needed, and to determine whether the target modality has an anomaly in the display process of the head-up display system with the updated alarm threshold and the anomaly score value of the target modality; and when the update of the alarm threshold is not needed, the original alarm threshold is still used for anomaly determination, that is, the original alarm threshold and the anomaly score value of the target modality are used to determine whether the target modality has an anomaly in the display process of the head-up display system. The way to determine whether there is an anomaly is to determine whether the current anomaly score value of the target modality is greater than the current compared alarm threshold. If the current anomaly score value of the target modality is greater than the current compared alarm threshold, it is determined that the target modality has an anomaly in the display process of the head-up display system; and if the current anomaly score value of the target modality is less than or equal to the current compared alarm threshold, it is determined that the target modality has no anomaly in the display process of the head-up display system. Through the same implementation, whether each modality has an anomaly in the working process of the head-up display can be determined.

[0043] The head-up display multi-modal anomaly test method provided by the application first determines a prototype vector in a prototype library that matches a current multi-modal fusion vector, the multi-modal fusion vector being obtained by fusing embedding vectors of multi-modal data related to the head-up display, and the prototype vector in the prototype library being a standard multi-modal fusion vector corresponding to an expected display state; the multi-modal fusion vector and the matching prototype vector are subjected to abnormal score processing of each modality respectively to obtain abnormal score values corresponding to each modality; a target distribution result corresponding to a target modality is determined according to a sequence of abnormal score values of the target modality within a first time length, the target modality being any one of the multi-modalities; the target distribution result of the target modality and a baseline distribution are verified by a statistical test method of distribution difference maximization to determine a modality drift of the target modality, the baseline distribution being a distribution of the target modality when normal under the corresponding expected display state; an alarm threshold is updated according to the modality drift of the target modality; and a test result of the target modality is determined according to the abnormal score value of the target modality and the currently updated alarm threshold. Thus, the method provided by the application improves the accuracy and robustness of abnormal detection by fusing multi-modal data related to the head-up display for abnormal detection of the head-up display, and further improves the accuracy of abnormal detection of the head-up display by filtering inherent and normal random fluctuations in the abnormal detection process through drift monitoring of each modality data.

[0044] In combination with the above embodiments, in an implementation, the embodiments of the application further provide a head-up display multi-modal anomaly test method. In the head-up display multi-modal anomaly test method, step S5 can include: in a case where the modality drift of the target modality meets a self-drift determination condition, updating an alarm threshold of the target modality by a preset threshold updating algorithm to obtain an updated alarm threshold; and an expression of the preset threshold updating algorithm is: wherein, is an update step; and are the alarm thresholds of the mth modality at time t and time t updated, respectively; and are the target distribution result and the baseline distribution of the mth modality at time t, respectively; is an effective modality drift for updating, represents the product of the drift direction and intensity, is a distribution mean value, and gives the direction of upward and downward movement; is the KS distance of the target distribution result and the baseline distribution.

[0045] In this embodiment, a preset threshold update algorithm is pre-established for updating the alarm threshold of the target mode. When the modal drift of the target mode meets its own drift determination condition, the alarm threshold of the target mode is updated by the preset threshold update algorithm. The meaning of the expression of the preset threshold update algorithm is that when the overall distribution shifts upward (i.e., the score is larger), the alarm threshold is increased, and when it shifts downward, the alarm threshold is decreased. Preferably, the drift determination condition of the target mode is that the modal drift of the target mode is determined to meet its own drift determination condition when the modal drift of the target mode is greater than the drift threshold for a consecutive preset number of times. The drift threshold is preferably set to 0.2. The update step size ranges from [0.1, 0.3].

[0046] In this embodiment, to facilitate location and playback, the confirmed drift modes are tagged according to simple rules: if the image mode drifts and is accompanied by changes in brightness-related indicators, it is tagged as ambient light change; if the voice mode drifts and the effective frame ratio of SNR / VAD in that window decreases significantly, it is tagged as voice change; if the signal mode drifts and there are phenomena such as message arrival interval jitter, increased frame drop rate, increased interface timeout rate, or increased error codes, it is tagged as signal interference; if at least two or more modes drift simultaneously within a similar time period and continue for a period of time, it is tagged as configuration update; if none of the above conditions are met, it is temporarily judged as unclassified.

[0047] In conjunction with the above embodiments, in one implementation, this application also provides a method for testing multimodal anomalies in head-up displays. In this method, step S5 may include: when the modal drift of the target modality meets its own drift determination condition, updating the alarm threshold of the target modality using a preset threshold update algorithm to obtain an updated initial alarm threshold; and clamping the updated initial alarm threshold within a preset range to obtain an updated alarm threshold.

[0048] In this embodiment, to prevent excessive jitter during the alarm threshold update process, this application performs a clamping process on the update result to avoid this problem. Specifically: Similar to the previous embodiment, when the modal drift of the target mode meets its own drift determination condition, the alarm threshold of the target mode is first updated using a pre-established preset threshold update algorithm, resulting in the updated initial alarm threshold. Further clamping processing is then applied to this initial alarm threshold to prevent excessive jitter during the update process. The expression for the clamping processing is as follows: , For the first The initial alarm threshold after each modal update is the result of the calculation by the preset threshold update algorithm. The clamping function is defined as follows: forcing x to be within the interval [a, b], and are the upper and lower alarm threshold limits allowed for the mth modality, respectively.

[0049] In combination with the above embodiments, in an implementation, the embodiments of the present application further provide a head-up display multi-modality anomaly test method. In the head-up display multi-modality anomaly test method, the method further includes: in the case where the modality drift amount of the target modality returns to the normal value and lasts for a preset number of times, gradually restoring the updated alarm threshold to the initial alarm threshold at a preset step size.

[0050] In the present embodiment, in the subsequent monitoring process, if the modality drift amount of the target modality returns to the normal value and lasts for a second time duration, the updated alarm threshold of the target modality is gradually restored to the initial alarm threshold at a pre-set smaller step size. The purpose of this is to pull back the possibly temporary changed judgment standard after confirming stability, so as to restore stability. The small step size return is to avoid frequent up and down adjustment, return to the vicinity of the basic baseline, and also prevent the alarm threshold from being stuck at the adjusted value for a long time, causing the subsequent real anomaly to be missed. The preset number of times is preferably 5 times, that is, the modality drift amount of the target modality is within the normal value for 5 consecutive times, and the updated alarm threshold is gradually restored to the initial alarm threshold at a preset step size.

[0051] In combination with the above embodiments, in an implementation, the embodiments of the present application further provide a head-up display multi-modality anomaly test method. In the head-up display multi-modality anomaly test method, the method further includes: in the case where the modality drift amounts of the plurality of modalities all satisfy the respective drift judgment conditions within the second time duration, determining the duration of the drift of all the modalities, the plurality of modalities including at least two modalities; in the case where the duration satisfies the set condition, determining that the current is in the version change state, enlarging the update step size, and freezing the alarm threshold of other non-drift modalities for a preset time duration.

[0052] In the embodiment, based on the obtained modal drift amount of each modality, it is determined whether the modal drift amount of the plurality of modalities satisfies the respective drift determination condition in the second time length. If so, it is further determined the duration of time when the plurality of modalities satisfies the respective drift determination condition. If the duration of time when the plurality of modalities satisfies the respective drift determination condition satisfies the set condition, it is indicated that the current situation is caused by the version change, at this time, the update step of the alarm threshold of the plurality of modalities (i.e., the plurality of modalities satisfying the respective drift determination condition) is enlarged, and the alarm threshold of the modalities other than the plurality of modalities is frozen for a preset time length (i.e., a preset time length). In an embodiment of the set condition, a duration threshold is set, and when the duration of time when the plurality of modalities satisfies the respective drift determination condition is greater than or equal to the duration threshold, it is determined that the set condition is satisfied. The preset time length of the freezing can be set according to the actual scene, which is not limited here, such as 5 seconds, 10 seconds, etc. The enlargement of the update step here is to make the alarm threshold quickly catch up with the new distribution and shorten the noise and false alarm period. Because of the version change, the overall modal distribution will change, and if the alarm threshold is updated with a small step, the system will lag for a long time and report false exceptions.

[0053] In combination with the above embodiments, in an embodiment, the application further provides a head-up display multi-modal anomaly test method. In the head-up display multi-modal anomaly test method, when the multi-modal data includes image data, text sequence data, signal data, and voice data, a current multi-modal fusion vector is determined, including:

[0054] Step S01: Collect multi-modal data related to head-up display.

[0055] In the embodiment, the embodiment of the application for generating a multi-modal fusion vector is: first, collect multi-modal data related to head-up display, and the multi-modal data at least includes image data, text sequence data, signal data, and voice data. The image data refers to a HUD projection image projected to the front windshield by the head-up display system, and the HUD projection image (such as a 1920x720 RGB image) is collected in real time by an industrial camera. The text sequence data is a sequence composed of characters extracted from the projection image. The signal data includes CAN signal data, LIN signal data, and DDS service log, etc. The CAN signal data and the LIN signal data are collected by a signal collection card built in a HIL test bench (Hardware-in-the-Loop Test Bench), and the sampling period is preferably 10 ms, supports signal timestamp binding, and the DDS service log collects service response time, timeout alarm, etc. through an embedded log service. The voice data is the voice output by the user, and the audio is collected by using a microphone array in a soundproof box, and is converted into text sequence by an ASR module.

[0056] Step S02: timestamp alignment of the multi-modal data by unified timestamp, and encapsulation into synchronous data packets by sliding window.

[0057] In this embodiment, after the multi-modal data related to the head-up display is collected by step S01, the collected multi-modal data is timestamped with a unified timestamp (ms level) and encapsulated into synchronous data packets (such as JSON / multi-channel tensor) using a sliding window. Preferably, the width of the data window is set to 1 second and the step is 100 ms. The time synchronization at this time is to align the data of each modality in the multi-modal data in time, so as to ensure the time synchronization of the multi-modal data.

[0058] Step S03: resolution unification of image data in the synchronous data packet, and brightness normalization and gamma normalization processing to obtain target image data.

[0059] In this embodiment, for the synchronous data packet obtained in step S02, in order to ensure the structural consistency of the model input tensor, the present application will further preprocess each modality data and unify it to the same sampling window after preprocessing. First, the image data in the synchronous data packet is resolution-unified (such as scaled to 640x240), and then brightness normalization and gamma normalization processing are performed to obtain target image data.

[0060] Step S04: uniformly converting text sequence data in the synchronous data packet into Token and splicing into fixed-length sequences to obtain target text sequence data.

[0061] In this embodiment, for the text sequence data in the synchronous data packet, after uniformly converting it into Token, it is spliced into target text sequence data of fixed-length sequence (maximum 128 characters), and the insufficient part is padded.

[0062] Step S05: time axis padding of signal data in the synchronous data packet into equal-interval data stream to obtain target signal data.

[0063] In this embodiment, for the signal data in the synchronous data packet, it is padded into an equal-interval data stream (window width=1s, step=100ms) along the time axis, thereby obtaining target signal data.

[0064] Step S06: endpoint detection and noise suppression preprocessing of the speech data in the synchronous data packet, and embedding representation extraction of the preprocessed speech data to obtain target speech data.

[0065] In this embodiment, for the voice data in the synchronous data packet, VAD endpoint detection and NS noise suppression preprocessing are performed thereon, and then the preprocessed result is input into Wav2Vec to extract an embedded representation, so as to obtain target voice data.

[0066] Step S07: The preprocessed multi-modal data is unified to the same sampling window, and multi-modal window data is constructed through a sliding window.

[0067] In this embodiment, the preprocessed multi-modal data in the synchronous data packet is unified to the same sampling window, and then corresponding multi-modal window data is constructed through a sliding window.

[0068] Step S08: For each type of modal data in the multi-modal window data, a corresponding encoder is called to encode each type of modal data respectively to obtain an embedded vector corresponding to each type of modal.

[0069] In this embodiment, after the preprocessed multi-modal window data is obtained through step S07, for each type of modal, a corresponding encoder is called to encode each type of modal to obtain an embedded vector corresponding to each type of modal. For example, for image modal data in the multi-modal window data, a lightweight ViT (Vision Transformer) + GAP pooling is called to encode, and a 512-dimensional embedded vector is output; for text sequence modal data in the multi-modal window data, a BERT-base model is called to encode, and a 768-dimensional embedded vector is output; for signal modal data in the multi-modal window data, a 1D-CNN + time attention module (TA-Layer) is called to encode, and a 512-dimensional embedded vector is output by splicing 128-dimensional embedded vectors of each channel; for voice modal data in the multi-modal window data, a Wav2Vec2.0 is called to encode, and a 256-dimensional embedded vector is output. All encoder outputs are standardized through LayerNorm to ensure that the scales are uniform for subsequent semantic fusion.

[0070] Step S09: The obtained embedded vectors of each type of modal are weighted and fused to obtain a current multi-modal fusion vector.

[0071] In this embodiment, after the embedded vectors of each type of modal in the same multi-modal window data are obtained through step S08, the embedded vectors of all types of modal are weighted and fused to obtain a multi-modal fusion vector corresponding to the multi-modal window data.

[0072] In combination with the above embodiments, in an implementation, the present embodiment further provides a head-up display multi-modal anomaly test method. In the head-up display multi-modal anomaly test method, step S09 can include steps S09_1 to S09_5:

[0073] Step S09_1: Align the embedding vectors corresponding to the image modal and the embedding vectors corresponding to the text modal in the multi-modal window data to a unified semantic space through double-tower comparison, to obtain image unified embedding vectors and text unified embedding vectors.

[0074] In the present embodiment, in the field of anomaly detection technology of head-up display systems, images and texts related to head-up display are relatively fixed and do not appear obvious and particularly large interference, while signals and voices related to head-up display are more prone to problems. Therefore, the present application first aligns the embedding vectors of images and texts to a unified semantic space through double-tower comparison, so that the subsequent gating processing based on the embedding vectors of the two modalities which are pre-aligned to a unified semantic space through double-tower comparison can better resist interference, thereby making the fusion accuracy and robustness of multi-modal data better. Thus, first, the embedding vectors corresponding to the image modal and the embedding vectors corresponding to the text modal in the multi-modal window data are aligned to a unified semantic space through double-tower comparison, i.e., the two modalities are encoded by a visual tower and a text tower respectively, and the image modal and the text modal corresponding to the semantics are mapped to a unified semantic space through a contrastive target, so that they are closer and non-corresponding samples are farther, thereby obtaining image unified embedding vectors corresponding to the embedding vectors of the image modal and text unified embedding vectors corresponding to the embedding vectors of the text modal, which will be used for subsequent multi-modal fusion.

[0075] Step S09_2: Determine the gating value corresponding to each modality according to the quality of each modality data in the multi-modal window data.

[0076] In the present embodiment, the gating value corresponding to each modality is determined according to the quality of each modality data in the multi-modal window data, which can adaptively reduce the gating value of a modality when the modality data of the modality in the multi-modal window data is missing or poor in quality. The better the data quality of a modality, the greater the gating value corresponding to the modality. The indicators for evaluating the quality of modality data include image brightness stability, speech SNR / VAD effective frame proportion, CAN frame loss / jitter, service timeout rate, etc.

[0077] Step S09_3: Determine the initial fusion weight of the corresponding modality based on the gating value and the reference weight of the same modality.

[0078] In the present embodiment, the present application pre-sets the corresponding reference weight for each modality, and after obtaining the gating value corresponding to each modality through step S09_2, the gating value and the reference weight of the same modality are multiplied to obtain the initial fusion weight corresponding to the modality. Through the same implementation, each modality can obtain an initial fusion weight.

[0079] Step S09_4: Normalizing the initial fusion weights of all modalities to obtain the respective fusion weights of each modality.

[0080] In this embodiment, the initial fusion weights of all modalities obtained through step S09_3 are normalized to obtain the respective fusion weights of each modality.

[0081] Step S09_5: Weighted fusion of the target embedding vectors of each modality based on the fusion weights of each modality to obtain a multi-modal fusion vector; wherein the target embedding vector of the image modality is the image unified embedding vector, the target embedding vector of the text modality is the text unified embedding vector, and the target embedding vector of other modalities is the embedding vector obtained by encoding through the corresponding encoder.

[0082] In this embodiment, after obtaining the fusion weights of each modality through step S09_4, the target embedding vectors of each modality corresponding to the multi-modal window data are weighted fused based on the fusion weights of each modality to obtain a multi-modal fusion vector corresponding to the multi-modal window data, and the expression is: wherein, is the multi-modal fusion vector, is the current time slice, is the embedding vector of the i-th modality at time t; is the fusion weight of the i-th modality. Thus, the generation from the multi-modal window data to the multi-modal fusion vector is completed.

[0083] In combination with the above embodiments, in an implementation manner, the present embodiment further provides a head-up display multi-modal anomaly test method. In the head-up display multi-modal anomaly test method, a prototype library is constructed, including:

[0084] Step S001: determining whether an end collection condition is met in the process of continuously collecting multi-modal window data in a desired display state.

[0085] In this embodiment, since the construction manner of the prototype sample corresponding to each desired display state is the same, the construction of the prototype sample corresponding to one desired display state is taken as an example. When the construction of the prototype sample corresponding to all desired display states is completed, all prototype samples constitute the prototype library.

[0086] In this embodiment, the multi-modal window data normally displayed in the desired display state is continuously collected, and then it is constantly judged whether all the multi-modal window data collected at the cut-off judgment time meets the end collection condition.

[0087] Step S002: In the case of meeting the end acquisition condition, each of the acquired multi-modal window data is processed into a respective corresponding multi-modal fusion vector, and a dimension-by-dimension average is performed in the unified semantic space to obtain the prototype vector in the expected display state.

[0088] In the embodiment, when the end acquisition condition is met, no further acquisition of the multi-modal window data normally displayed in the expected display state is performed, and the generation of the prototype vector in the expected display state is performed on all the multi-modal window data acquired at the end of the acquisition. Each of the multi-modal window data acquired at the end of the acquisition is processed into a respective corresponding multi-modal fusion vector according to the same multi-modal fusion vector generation implementation described above. Then, a dimension-by-dimension average is performed on all the multi-modal fusion vectors corresponding to all the acquired multi-modal window data in the unified semantic space, and the prototype vector in the expected display state is obtained in this way. According to the same implementation, the prototype vector corresponding to each expected display state can be constructed and obtained.

[0089] Step S003: In the case where the number of all the acquired multi-modal window data meets the first condition, a covariance matrix of the expected display state in the unified semantic space is calculated according to the multi-modal fusion vectors of all the acquired multi-modal window data.

[0090] In the embodiment, since the covariance matrix is used in determining the prototype vector matched with the multi-modal fusion vector corresponding to the current actually collected multi-modal window data, the covariance matrix corresponding to each expected display state is constructed in constructing the prototype library. An optional implementation of constructing the covariance matrix corresponding to the expected display state is: constructing the covariance matrix corresponding to the expected display state based on the number of multi-modal window data used in constructing the prototype vector corresponding to the expected display state; when the number of multi-modal window data used in constructing the prototype vector corresponding to the expected display state is large enough, the constructed covariance matrix corresponding to the expected display state is the covariance matrix; when the number of multi-modal window data used in constructing the prototype vector corresponding to the expected display state is moderate, the constructed covariance matrix corresponding to the expected display state is the shrinkage covariance matrix; when the number of multi-modal window data used in constructing the prototype vector corresponding to the expected display state is small, the constructed covariance matrix corresponding to the expected display state is the diagonal covariance matrix. An optional implementation of the first condition, the second condition and the third condition included in constructing the covariance matrix of the expected display state in the unified semantic space is that each of the first condition, the second condition and the third condition corresponds to a threshold value, and the relationship among the threshold value a corresponding to the first condition, the threshold value b corresponding to the second condition and the threshold value c corresponding to the third condition is a > b > c. When the number of all collected multi-modal window data is greater than or equal to a, it is determined that the first condition is satisfied, and the covariance matrix corresponding to the expected display state is the covariance matrix; when the number of all collected multi-modal window data is greater than or equal to b and less than a, it is determined that the second condition is satisfied, and the covariance matrix corresponding to the expected display state is the shrinkage covariance matrix; when the number of all collected multi-modal window data is greater than or equal to c and less than b, it is determined that the third condition is satisfied, and the covariance matrix corresponding to the expected display state is the diagonal covariance matrix. Thus, when the number of all collected multi-modal window data of the expected display state satisfies the first condition, the covariance matrix of the expected display state in the unified semantic space is calculated according to the multi-modal fusion vectors of all collected multi-modal window data, and the covariance matrix is determined as the covariance matrix corresponding to the expected display state.

[0091] Step S004: in the case where the number of all collected multi-modal window data satisfies the second condition, the shrinkage covariance matrix of the expected display state in the unified semantic space is calculated according to the multi-modal fusion vectors of all collected multi-modal window data.

[0092] In the embodiment, in a case where the number of all the multi-modal window data in the expected display state collected satisfies the second condition, a shrinkage covariance matrix of the expected display state in the unified semantic space is calculated according to the multi-modal fusion vectors of all the multi-modal window data collected, and the shrinkage covariance matrix is determined as the covariance matrix corresponding to the expected display state.

[0093] Step S005: In a case where the number of all the multi-modal window data collected satisfies the third condition, a diagonal covariance matrix of the expected display state in the unified semantic space is calculated according to the multi-modal fusion vectors of all the multi-modal window data collected.

[0094] In the embodiment, in a case where the number of all the multi-modal window data in the expected display state collected satisfies the third condition, a diagonal covariance matrix of the expected display state in the unified semantic space is calculated according to the multi-modal fusion vectors of all the multi-modal window data collected, and the diagonal covariance matrix is determined as the covariance matrix corresponding to the expected display state. When a version is upgraded or a new page is added, only the normal multi-modal window data of the added expected display state need to be collected, the prototype vector and the covariance matrix of the added expected display state need to be determined and written into the prototype library, and other states do not need to be changed, so that the maintenance cost is low.

[0095] In combination with the above embodiments, in an implementation, the embodiments of the present application further provide a head-up display multi-modal anomaly test method. In the head-up display multi-modal anomaly test method, the ending collection condition is any two of three ending conditions; the three ending conditions are respectively: the mean difference between the multi-modal fusion vectors of the latest two batches of multi-modal window data is lower than a first threshold value, and the number of one batch of multi-modal window data is a preset value; the shrinkage coefficient of the shrinkage covariance between the multi-modal fusion vectors of the latest two batches of multi-modal window data converges to less than or equal to a second threshold value, and the relative change of the diagonal variance between the multi-modal fusion vectors of the latest two batches of multi-modal window data is less than a third threshold value; the KS distance of the multi-modal fusion vector distribution of the latest two batches of multi-modal window data is less than or equal to a fourth threshold value.

[0096] In this embodiment, three termination conditions are defined. During the actual acquisition of multimodal window data in the desired display state, the acquisition is considered complete if the acquired multimodal window data satisfies any two of the three termination conditions. The first termination condition is: determining whether the mean difference between the multimodal fusion vectors of the latest two batches of multimodal window data is lower than a first threshold. If it is, the first termination condition is met. The number of data in a batch of multimodal window data is a preset value, preferably 10, and the first threshold is preferably 0.02. The second termination condition is: determining whether the contraction coefficient of the contraction covariance between the multimodal fusion vectors of the latest two batches of multimodal window data converges to less than or equal to a second threshold, and whether the relative change in the diagonal variance between the multimodal fusion vectors of the latest two batches of multimodal window data is less than a third threshold. If both conditions are met simultaneously, the second termination condition is satisfied. The second threshold is preferably 0.2, and the third threshold is preferably 0.1. The third termination condition is: determine whether the KS distance of the multimodal fusion vector distribution of the latest two batches of multimodal window data is less than or equal to the fourth threshold. If it is less than or equal to the fourth threshold, the third termination condition is satisfied. The fourth threshold is preferably 0.1.

[0097] In conjunction with the above embodiments, in one implementation, this application also provides a method for testing multimodal anomalies in head-up displays. In this method, step S1 may include steps S101 to S102:

[0098] Step S101: Determine the matching degree between each prototype vector in the prototype library and the current multimodal fusion vector using a joint matching algorithm; the expression for the joint matching algorithm is: Where c is the matching degree, The Mahalanobis distance, For cosine similarity, Let be the multimodal fusion vector at time t. The prototype vector is from the prototype library. To correspond to the desired display state in the prototype vector The covariance matrix under the following conditions These are the weighting coefficients.

[0099] In this embodiment, a joint matching algorithm is predefined to determine the matching degree between prototype vectors in the prototype library and the current multimodal fusion vector. By substituting a prototype vector from the prototype library and the current multimodal fusion vector into the joint matching algorithm, the matching degree between the prototype vector and the current multimodal fusion vector can be calculated. Through the same implementation method, the matching degree between each prototype vector in the prototype library and the current multimodal fusion vector can be calculated separately.

[0100] Step S102: According to all the matching degrees obtained, determine the prototype vector in the prototype library that matches the current multi-modal fusion vector.

[0101] In this embodiment, after obtaining the matching degree between each prototype vector in the prototype library and the current multi-modal fusion vector respectively through step S101, it is determined which prototype vector corresponds to the highest matching degree, and the prototype vector with the highest matching degree is determined as the prototype vector that matches the current multi-modal fusion vector.

[0102] In combination with the above embodiments, in an implementation, the embodiments of the present application also provide a head-up display multi-modal anomaly test method. In the head-up display multi-modal anomaly test method, step S2 can include: performing anomaly scoring processing of each modality on the multi-modal fusion vector and the matched prototype vector respectively by an anomaly scoring algorithm to obtain anomaly scoring values corresponding to each modality; and the expression of the anomaly scoring algorithm is: wherein, is the anomaly score of the mth modality; is the weighting coefficient of the two joint evaluations corresponding to the modality; is the Mahalanobis distance, used to measure the difference in distribution scale; is the single-modality embedding vector of the mth modality in the unified semantic space at time t; is the single-modality embedding vector of the mth modality in the unified semantic space in the matched prototype vector; is the covariance matrix of the corresponding expected display state in the single-modality embedding vector .

[0103] In this embodiment, the present application defines an anomaly scoring algorithm in advance for anomaly scoring calculation of each modality on the current multi-modal fusion vector and the matched prototype vector. The embedding vectors in the same modality in the current multi-modal fusion vector and the matched prototype vector are substituted into the anomaly scoring algorithm for calculation to obtain the anomaly score value in the modality. Among them, the type of the covariance matrix of the corresponding expected display state in the single-modality embedding vector in the anomaly scoring algorithm is the same as the matrix type of the covariance matrix of the expected display state in the prototype vector in the prototype library, such as covariance matrix, shrinkage covariance matrix, and diagonal covariance matrix. Through the same implementation, the anomaly score value of each modality can be calculated respectively.

[0104] In combination with the above embodiments, in an implementation, the embodiments of the present application also provide a head-up display multi-modal anomaly test method. In the head-up display multi-modal anomaly test method, the method further includes:

[0105] Step S7: when the test result is abnormal, a multi-modal factor graph is constructed based on the multi-modal data collected in the corresponding time period, the multi-modal factor graph including modal event nodes and edges between the modal event nodes, the edges being oriented based on the calling dependency and time sequence between the modal event nodes.

[0106] In this embodiment, when the test result is abnormal, it indicates that at least one modality is abnormal, and the application further performs root cause analysis. First, when the test result is abnormal, a multi-modal factor graph is constructed based on the collected multi-modal data in a short time window around the abnormal occurrence time, and the expression is , where V represents a modal event node, including a service and interface event node (DDS / API call, return code, timeout), a bus signal node (CAN / LIN ID), a display node (HUD layer / control), a voice node (ASR / TTS response), etc.; E represents a set of edges between nodes, the edges being oriented based on the calling dependency and time sequence between the modal event nodes, and the edge features including time delay , packet loss / jitter, rendering time consumption, historical correlation degree, etc.

[0107] Step S8: the initial attention weight of each edge in the multi-modal factor graph is calculated through typed linear mapping and attention mechanism, and the calculation expression is: , where is the correlation strength value between node i and node j; is the initial attention weight between node i and node j; is the time sequence consistency; is the signal quality, which is the geometric mean of the gating value of the modality to which node i belongs and the gating value of the modality to which node j belongs ; is the historical dependency degree; is the evidence fit degree of the edge; is the drift label prior of the edge; is the weight coefficient; the time sequence consistency, the signal quality, the historical dependency degree, the evidence fit degree, and the drift label prior are all dimensionless values of the respective feature mapping to [0, 1].

[0108] In this embodiment, after the corresponding multi-modal factor graph is obtained by step S7, typed mapping (different linear mapping of service / signal / display / voice) is used for different types of modal event nodes, and the score of each edge is calculated based on the typed mapping, and then softmax is performed to obtain the initial attention weight of the edge . Wherein, (Default ). For the same out-edge set softmax normalization , by default β = 5, so that and .

[0109] In this embodiment, the present application maps each feature constituting the edge association strength to a dimensionless score in [0, 1], and obtains , and then sums the weights to obtain the association strength of the edge , and finally obtains the initial attention weight of the node pointing to each adjacent node through softmax .

[0110] In this embodiment, the calculation process of the time sequence consistency is as follows: let the edge direction be , , be the data acquisition time corresponding to j, be the data acquisition time corresponding to i. Let , by default, the closer to the score of "i before j", the greater the score; if j triggers before i, take when = 1, it is considered as strong consistency.

[0111] In this embodiment, the calculation process of the signal quality is as follows: the gating value of the modal to which the nodes i and j belong , which comes from the quality evaluation in the foregoing, take , when the quality of either end is poor, this item is automatically reduced.

[0112] In this embodiment, the calculation process of the historical dependence is as follows: select a period of stable system data, and count how many times node j appears within a set time window when node i occurs, denoted as , and how many times node i occurs in total, denoted as . The appearance ratio is used to represent the strength of the sequence relationship between the two: , and all form a sequence relationship table, which is saved with the version for subsequent use. If there is no sequence relationship between i and j in the stable period, the ratio is recorded as 0. Therefore, can be directly taken , the greater the value, the stronger the history of "i followed by j". The calculation of the evidence coincidence degree and the calculation of the drift label prior will be described in subsequent embodiments.

[0113] Step S9: According to the initial attention weight, a preset number of rounds of message passing is performed, and the initial attention weight is iteratively updated to obtain the influence strength of each edge on the abnormal output node, which is the modal event node corresponding to the abnormal modal.

[0114] In this embodiment, after obtaining the initial attention weight of each edge through step S8, a preset number of rounds (preferably 2 to 3 rounds) of message passing is performed to iteratively update the initial attention weight of the edge, so as to obtain the influence strength of each edge on the abnormal output node, which is actually the modal event node corresponding to the abnormal modal. The influence strength of the edge on the abnormal output node is actually the attention weight of the edge after iterative update. For the temporal consistency , if i precedes j and , the smaller the , the greater the temporal consistency. For the historical dependence , it refers to whether A will usually be followed by B within the past data. If it is often "event i→event j", the historical dependence from i to j is higher, and almost irrelevant is lower. The strength has a value of 0 to 1. Therefore, the attention weight has a greater value when the causal temporal sequence, good data quality at both ends, historical dependence, current appearance in the hot evidence, and consistency with the determined drift type are met.

[0115] Step S10: Enumerate all candidate paths with a path length not exceeding a preset value, starting from the candidate root cause node and ending at the abnormal output node.

[0116] In this embodiment, a certain amount of candidate root cause nodes are selected based on analysis, and it is analyzed which one or which ones of them are the real root cause nodes. Specifically, all candidate paths with a path length not exceeding a preset value (preferably 4 to 5 hops) are enumerated, starting from the candidate root cause node and ending at the abnormal output node. Through the same implementation, each candidate root cause node can enumerate all candidate paths corresponding to itself.

[0117] Step S11: Determine the path influence of the candidate root cause node according to the influence strength of all edges in all candidate paths corresponding to the candidate root cause node.

[0118] In this embodiment, for all candidate paths corresponding to the candidate root cause node enumerated in step S10, the path influence of the candidate root cause node is calculated based on the influence strength of all edges in all candidate paths corresponding to the candidate root cause node, and the calculation expression is . Through the same implementation, the path influence of each candidate root cause node can be calculated and obtained.

[0119] Step S12: performing virtual ablation of root cause analysis on the candidate root cause node, and calculating a joint abnormal score value after ablation, the joint abnormal score value being a sum of abnormal contribution degrees of each modality under the same data window, the abnormal contribution degree of a modality being determined based on an abnormal score value of the modality and a corresponding abnormal weight.

[0120] In the embodiment, virtual ablation of root cause analysis is performed on each candidate root cause node, and the virtual ablation of each candidate root cause node is the same, which is illustrated by taking one candidate root cause node as an example: the influence intensity of the out-edge of the candidate root cause node is set to zero or the feature of the node is replaced by the baseline statistics, that is, it is assumed that the joint abnormal score of the entire system is in the case of not being affected by the candidate root cause node. After the virtual ablation of root cause analysis on the candidate root cause node, the joint abnormal score value after ablation is calculated. The joint abnormal score value is a sum of abnormal contribution degrees of each modality under the same data window, and the abnormal contribution degree of a modality is determined based on an abnormal score value of the modality and a corresponding abnormal weight. Through the same implementation, each candidate root cause node can obtain a corresponding joint abnormal score value.

[0121] Step S13: determining a drop value corresponding to the candidate root cause node according to the joint abnormal score value after ablation of the candidate root cause node and the joint abnormal score value before ablation.

[0122] In the embodiment, after obtaining the respective corresponding joint abnormal score values of each candidate root cause node through step S12, for the same candidate root cause node, the difference between the joint abnormal score value after ablation and the joint abnormal score value before ablation is determined as the drop value corresponding to the candidate root cause node. Through the same implementation, each candidate root cause node can obtain a corresponding drop value.

[0123] Step S14: determining a comprehensive root cause score of the candidate root cause node according to the drop value and the path influence of the candidate root cause node.

[0124] In the embodiment, the implementation of determining the comprehensive root cause score of the candidate root cause node is the same, which is illustrated by taking one candidate root cause node as an example. Based on the path influence of the candidate root cause node obtained through step S11 and the drop value of the candidate root cause node obtained through step S13, the comprehensive root cause score of the candidate root cause node is calculated. The specific calculation expression is wherein λ is a weight, and the value range is [0.4, 0.7], represents the path influence of the candidate root cause node r, represents the drop value of the candidate root cause node r. Through the same implementation, each candidate root cause node can obtain a corresponding comprehensive root cause score.

[0125] Step S15: determining the final root cause result according to the comprehensive root cause score of each candidate root cause node.

[0126] In this embodiment, after obtaining the comprehensive root cause score of each candidate root cause node through step S14, the comprehensive root cause scores of all candidate root cause nodes are sorted from high to low, and the top K are taken as the root cause result, where K can be configured, and the default is 3, the recommended range is 1 to 5, and if the number of candidate root cause nodes is insufficient, the actual number is output.

[0127] In this embodiment, the root cause chain and the evidence package are output, and for each selected root cause node, how it affects the high-confidence path of the final abnormal output node (such as HUD display abnormality) along the factor graph is listed. For each path, the following information is provided: the node sequence from the root cause node to the abnormal output node in chronological order; the attention weight of each edge on the path, indicating the strength of the association between the two modal event nodes; the time difference between the corresponding modal event nodes. At the same time, the path length is limited to no more than 5 hops to avoid noise caused by long chains. At the same time, verifiable evidence fields are provided: log timestamp / return code, CAN / LIN ID and arrival interval / frame loss count, HUD frame number and pixel offset, ASR keyword and response delay, etc.

[0128] In combination with the above embodiments, in an implementation, the embodiments of the present application also provide a head-up display multi-modal anomaly test method. In the head-up display multi-modal anomaly test method, the evidence fitting degree of the edge is determined, including:

[0129] Step S8_1a: determining the abnormal weight of each modal event node through an abnormal weight determination algorithm; the expression of the abnormal weight determination algorithm is: wherein, is the abnormal weight of the mth modal event node, is the basic weight of the mth modal event node, is the gating value of the mth modal event node, is the sum of the product of the gating value and the basic weight of all modal event nodes.

[0130] In the embodiment, the application predefines an abnormal weight determination algorithm, which is used to determine the respective abnormal weight of each modal event node. Since each modal event node is determined based on the corresponding modal data, there is corresponding modal data for one modal event node. Therefore, in the abnormal weight determination algorithm, the base weight and the gating value of the modal event node are actually determined based on the corresponding modal data. The base weight is a predefined constant value based on the corresponding modal, and the gating value is dynamically set based on the quality of the corresponding modal data. The better the quality, the higher the value. By substituting the base weight and the gating value corresponding to one modal event node into the abnormal weight determination algorithm for calculation, the abnormal weight corresponding to the modal event node can be obtained. Through the same implementation, each modal event node in the multi-modal factor graph can calculate and obtain a corresponding abnormal weight. The base weight setting for each modal includes: image is usually UI change, rendering exception, set to 0.4; signal part is usually state driven, source exception, set to 0.3; text is usually prompt word misplacement, missing word, set to 0.2; voice is mostly interactive receipt, set to 0.1.

[0131] Step S8_1b: determining the abnormal contribution degree of the modal event node according to the abnormal score value and the abnormal weight of the modal event node.

[0132] In the embodiment, the abnormal score value of the abnormal event node is the abnormal score value corresponding to the modal data corresponding to the abnormal event node. For each modal event node, the abnormal score value and the abnormal weight of the same modal event node are directly multiplied to obtain the abnormal contribution degree of the modal event node. Through the same implementation, each modal event node in the multi-modal factor graph can calculate and obtain a corresponding abnormal contribution degree.

[0133] In the embodiment, the abnormal contribution degrees of various modalities at the corresponding time can also be determined through the same implementation, and then the contribution of the largest one or two is given to give an intuitive prompt: if it is an image modality, a simple heat layer is superimposed on the currently collected image frame, and the suspicious area is highlighted; if it is a signal / service modality, the corresponding CAN ID / interface name+timestamp is marked; if it is a voice or text modality, the keywords / response delay in the corresponding time window are marked.

[0134] Step S8_1c: determining the evidence fit degree of the edge according to the abnormal contribution degrees of the two modal event nodes of the edge in the multi-modal factor graph.

[0135] In the embodiment, the evidence fit degree of the edge is determined based on the abnormal contribution degrees of the two modal event nodes of the edge in the multi-modal factor graph. The higher the abnormal contribution degrees of the modal event nodes corresponding to the edge, the higher the evidence fit degree of the edge.

[0136] In this embodiment, the degree of agreement of evidence The specific calculation process is as follows: Within a recent sliding window (default 60s), the abnormal contribution of modal event nodes is statistically analyzed second by second according to the system sampling period (default 1s). To avoid the influence of dimensions, normalization is first performed. For each second, the abnormal contribution of a modal event node is divided by the sum of the abnormal contributions of all modal event nodes to obtain the proportion of modal event nodes between 0 and 1. In the past 60s, if the proportion of a certain modal event node is ≥0.3 in a certain second, it is recorded that it is active in that second. Then, for the two ends of an edge, i.e., node i and node j, the number of seconds that both ends are active simultaneously, and the number of seconds that at least one end is active, are counted. The former is divided by the latter to obtain a number between 0 and 1. This value is the evidence consistency of the edge. An evidence consistency close to 1 indicates that the two ends are more synchronized, the evidence is very consistent, and they are more likely to influence each other, indicating a strong relationship; a consistency close to 0 indicates that they are basically asynchronous, indicating a weak relationship; if neither end is active within 60s, it is recorded as 0.

[0137] In conjunction with the above embodiments, in one implementation, this application also provides a method for testing multimodal anomalies in head-up displays. In this method, determining the drift label prior of the edge includes steps S8_2a to S8_2b:

[0138] Step S8_2a: Determine the modal type of the modal event node in the multimodal factor graph.

[0139] In this embodiment, the modality type of the modal event node in the multimodal factor graph is first determined. The modality type of the modal event node is the same as the modality type of its corresponding modal data.

[0140] Step S8_2b: Determine the drift label prior of the edge based on the modal types of the two modal event nodes of the edge in the multimodal factor graph and the modal type in which drift exists.

[0141] In this embodiment, the modal types exhibiting drift under the current abnormal state have been determined in the aforementioned steps. At this point, this application determines whether the modal type of the modal event node corresponding to the edge in the multimodal factor graph is the same as the modal type exhibiting drift. If they are the same, the modal event node is determined to be a node exhibiting drift, and then the specific edge drift label prior is calculated. The specific calculation process is as follows: when both i and j are judged as drifting and their order is related to the table hour, Only one end was judged to be drifting and Take 0.6; if no stable period relationship is statistically observed, but a time-based "i first, then j" pattern is visible, i.e. Take 0.3; the rest take 0. The default configuration is so, and the specific parameters can be fine-tuned in [0, 1].

[0142] The head-up display multi-modal anomaly test method provided by the application has the following advantages:

[0143] The prototype library driven joint anomaly score (without anomaly samples) is proposed, and no anomaly samples or retraining is required after going online. When a certain modality is temporarily missing or of poor quality, its weight is automatically reduced, and the overall judgment can be maintained stable. This method is more friendly to OTA new pages and frequent version changes, and has lower maintenance cost. The adaptive drift monitoring and dynamic adjustment of the modality level threshold can maintain stable sensitivity in long-term operation. The root cause positioning enhanced by the causal diagram, compared with the scheme of only giving an alarm without explanation, this method can be traced, verified and automatically retested.

[0144] Based on the same inventive concept, an embodiment of the application provides a head-up display multi-modal anomaly test system, as shown in Figure 2 The head-up display multi-modal anomaly test system 200 includes:

[0145] The prototype vector determination module 201 is configured to determine a prototype vector in a prototype library that matches a current multi-modal fusion vector, the multi-modal fusion vector being obtained by fusing embedding vectors of multi-modal data related to the head-up display, and the prototype vector in the prototype library being a standard multi-modal fusion vector corresponding to an expected display state;

[0146] The anomaly score module 202 is configured to perform anomaly score processing of each modality on the multi-modal fusion vector and the matched prototype vector, respectively, to obtain anomaly score values corresponding to each modality.

[0147] The target distribution result determination module 203 is configured to determine a corresponding target distribution result according to a sequence of anomaly score values of a target modality within a first time length, the target modality being any one of the multi-modal.

[0148] The modality drift amount determination module 204 is configured to check the target distribution result of the target modality and a baseline distribution by a distribution difference maximization statistical test method, to determine a modality drift amount of the target modality, the baseline distribution being a distribution of the target modality when normal under the corresponding expected display state.

[0149] The alarm threshold updating module 205 is configured to update an alarm threshold according to the modality drift amount of the target modality.

[0150] The evaluation module 206 is configured to determine a test result of the target modality according to the anomaly score value of the target modality and the currently updated alarm threshold.

[0151] Optionally, the alarm threshold updating module 205 is specifically configured to update the alarm threshold of the target modality by a preset threshold updating algorithm in a case where the modality drift amount of the target modality satisfies a self drift determination condition, to obtain an updated alarm threshold.

[0152] An expression of the preset threshold updating algorithm is as follows: , wherein, is an update step; and are the alarm thresholds of the mth modality at the time t and the time t, respectively, after updating; and are the target distribution result and the baseline distribution of the mth modality at the time t, respectively; is an effective modality drift amount for updating, represents the product of the drift direction and the intensity, is a distribution mean value, and gives the direction of the upward and downward movement; is the KS distance of the target distribution result and the baseline distribution.

[0153] Optionally, the alarm threshold updating module 205 comprises:

[0154] a first updating module configured to update the alarm threshold of the target modality by a preset threshold updating algorithm in a case where the modality drift amount of the target modality satisfies a self drift determination condition, to obtain an updated initial alarm threshold;

[0155] a second updating module configured to clamp the updated initial alarm threshold in a preset interval range, to obtain an updated alarm threshold.

[0156] Optionally, the head-up display multi-modality anomaly test system 200 further comprises:

[0157] a recovery module configured to gradually recover the updated alarm threshold to the initial alarm threshold by a preset step length in a case where the modality drift amount of the target modality is recovered to a normal value and lasts for a preset number of times.

[0158] Optionally, the head-up display multi-modality anomaly test system 200 further comprises:

[0159] a drift determination module configured to determine a continuous time length of drift in a case where the modality drift amounts of a plurality of modalities satisfy respective drift determination conditions within a second time length, the plurality of modalities comprising at least two modalities;

[0160] an update freezing module configured to determine that a current version is in a version changing state, enlarge the update step length, and freeze the alarm thresholds of other non-drift modalities for a preset time length in a case where the continuous time length satisfies a set condition.

[0161] Optionally, in the case that the multi-modal data includes image data, text sequence data, signal data, and speech data, the head-up display multi-modal anomaly test system 200 further comprises a multi-modal fusion vector determination module for determining a current multi-modal fusion vector;

[0162] The multi-modal fusion vector determination module comprises:

[0163] The data acquisition module is configured to acquire multi-modal data related to the head-up display;

[0164] The time alignment module is configured to perform timestamp alignment on the multi-modal data by using a uniform timestamp and encapsulate the multi-modal data into a synchronous data packet by using a sliding window;

[0165] The first preprocessing module is configured to perform resolution unification on image data in the synchronous data packet, and perform brightness normalization and gamma normalization processing to obtain target image data;

[0166] The second preprocessing module is configured to uniformly convert text sequence data in the synchronous data packet into Token and splice the Token into a fixed-length sequence to obtain target text sequence data;

[0167] The third preprocessing module is configured to perform time axis padding on signal data in the synchronous data packet to obtain equal-interval data stream to obtain target signal data;

[0168] The fourth preprocessing module is configured to perform endpoint detection and noise suppression preprocessing on speech data in the synchronous data packet, and extract embedded representation from the preprocessed speech data to obtain target speech data;

[0169] The unification module is configured to unify the preprocessed multi-modal data to the same sampling window, and construct multi-modal window data by using a sliding window;

[0170] The encoding module is configured to call respective encoders to respectively encode each type of modal data in the multi-modal window data to obtain respective embedded vectors of each type of modal data;

[0171] The weighted fusion module is configured to perform weighted fusion on the obtained embedded vectors of each type of modal data to obtain a current multi-modal fusion vector.

[0172] Optionally, the weighted fusion module comprises:

[0173] The unified semantic space alignment module is configured to align the embedded vectors of the image modal and the embedded vectors of the text modal in the multi-modal window data to a unified semantic space by using a double-tower comparison to obtain image unified embedded vectors and text unified embedded vectors;

[0174] A gating value determination module is configured to determine a respective gating value of each modality according to the quality of the modality data in the multi-modal window data.

[0175] An initial fusion weight determination module is configured to determine an initial fusion weight of the corresponding modality based on the gating value and the reference weight of the same modality.

[0176] A normalization processing module is configured to normalize the initial fusion weights of all modalities to obtain a respective fusion weight of each modality.

[0177] A weighted fusion sub-module is configured to perform weighted fusion on the target embedding vectors of each modality based on the fusion weight of each modality to obtain a multi-modal fusion vector; wherein the target embedding vector of the image modality is an image unified embedding vector, the target embedding vector of the text modality is a text unified embedding vector, and the target embedding vector of the other modality is an embedding vector obtained by encoding through a corresponding encoder.

[0178] Optionally, the head-up display multi-modal anomaly test system 200 further comprises a prototype library construction module configured to construct a prototype library; the prototype library construction module comprises:

[0179] A collection determination module is configured to determine whether an end collection condition is met in the process of continuously collecting multi-modal window data in the expected display state.

[0180] A prototype vector determination module is configured to, in the case where the end collection condition is met, process each collected multi-modal window data into a respective multi-modal fusion vector, and perform dimension-by-dimension averaging in a unified semantic space to obtain a prototype vector in the expected display state.

[0181] A first covariance matrix determination module is configured to, in the case where the number of all collected multi-modal window data meets a first condition, calculate a covariance matrix of the expected display state in the unified semantic space according to the multi-modal fusion vectors of all collected multi-modal window data.

[0182] A second covariance matrix determination module is configured to, in the case where the number of all collected multi-modal window data meets a second condition, calculate a shrinkage covariance matrix of the expected display state in the unified semantic space according to the multi-modal fusion vectors of all collected multi-modal window data.

[0183] A third covariance matrix determination module is configured to, in the case where the number of all collected multi-modal window data meets a third condition, calculate a diagonal covariance matrix of the expected display state in the unified semantic space according to the multi-modal fusion vectors of all collected multi-modal window data.

[0184] Optionally, the ending acquisition condition is that any two of three ending conditions are met; the three ending conditions are respectively:

[0185] The mean difference between the multi-modal fusion vectors of the latest two batches of multi-modal window data is lower than a first threshold value, and the number of a batch of multi-modal window data is a preset value;

[0186] The shrinkage coefficient of the shrinkage covariance between the multi-modal fusion vectors of the latest two batches of multi-modal window data converges to less than or equal to a second threshold value, and the relative change of the diagonal covariance between the multi-modal fusion vectors of the latest two batches of multi-modal window data is less than a third threshold value;

[0187] The KS distance of the multi-modal fusion vector distribution of the latest two batches of multi-modal window data is less than or equal to a fourth threshold value.

[0188] Optionally, the prototype vector determination module 201 comprises:

[0189] The matching degree determination module is configured to determine the matching degrees of each prototype vector in the prototype library and the current multi-modal fusion vector by using a joint matching algorithm.

[0190] The matching prototype vector determination module is configured to determine the prototype vector in the prototype library that matches the current multi-modal fusion vector according to all the obtained matching degrees.

[0191] The expression of the joint matching algorithm is:

[0192]

[0193] Wherein, c is the matching degree, is the Mahalanobis distance, is the cosine similarity, is the multi-modal fusion vector at time t, is the prototype vector in the prototype library, is the covariance matrix of the corresponding expected display state under the prototype vector is the weighting coefficient.

[0194] Optionally, the anomaly score module 202 is specifically configured to perform anomaly score processing on the multi-modal fusion vector and the matched prototype vector in each modality respectively by using an anomaly score algorithm, to obtain the anomaly score value corresponding to each modality.

[0195] The expression of the anomaly score algorithm is:

[0196] ;

[0197] Wherein, is the anomaly score of the mth modality; ​a weighting coefficient for the two joint evaluations of the corresponding modalities; is a Mahalanobis distance, used to measure the difference in scale of the distribution; is the single-modal embedding vector of the mth modality in the unified semantic space at time t; is the single-modal embedding vector of the mth modality in the unified semantic space in the matched prototype vector; is the covariance matrix of the single-modal embedding vectors of the corresponding expected display states.

[0198] Optionally, the head-up display multi-modal anomaly test system 200 further comprises:

[0199] a factor graph construction module, configured to, when the test result is abnormal, construct a multi-modal factor graph based on the multi-modal data collected in the corresponding time period, the multi-modal factor graph comprising modal event nodes and edges between the modal event nodes, the edges being oriented based on the calling dependency and time sequence between the modal event nodes;

[0200] an attention weight determination module, configured to calculate initial attention weights of each edge in the multi-modal factor graph through a typed linear mapping and an attention mechanism, the calculation expression being:

[0201] wherein, is the association strength value of the edge between node i and node j; is the initial attention weight between node i and node j; is the time sequence consistency; is the signal quality, being the geometric mean of the gating value of the modality to which node i belongs and the gating value of the modality to which node j belongs ; is the historical dependency degree; is the evidence fit degree of the edge; is the drift label prior of the edge; is a weight coefficient; the time sequence consistency, the signal quality, the historical dependency degree, the evidence fit degree, and the drift label prior are all dimensionless values of respective feature mappings to [0, 1];

[0202] an influence strength determination module, configured to perform a preset number of rounds of message passing according to the initial attention weights, iteratively update the initial attention weights, and obtain an influence strength of each edge on an abnormal output node, the abnormal output node being a modal event node corresponding to an abnormal modality;

[0203] ​​The candidate path determination module is configured to enumerate all candidate paths with a path length not exceeding a preset value, starting from a candidate root cause node and ending at the abnormal output node;

[0204] The path influence determination module is configured to determine a path influence corresponding to the candidate root cause node according to the influence strength of all edges in all candidate paths corresponding to the candidate root cause node;

[0205] The virtual ablation processing module is configured to perform virtual ablation of root cause analysis on the candidate root cause node, and calculate a joint abnormal score value after ablation, the joint abnormal score value being a sum of abnormal contribution degrees of each modality under the same data window, the abnormal contribution degree of a modality being determined based on an abnormal score value of the modality and a corresponding abnormal weight;

[0206] The drop value determination module is configured to determine a drop value corresponding to the candidate root cause node according to the joint abnormal score value after ablation of the candidate root cause node and the joint abnormal score value before ablation;

[0207] The comprehensive root cause score determination module is configured to determine a comprehensive root cause score of the candidate root cause node according to the drop value and the path influence of the candidate root cause node;

[0208] The root cause result determination module is configured to determine a final root cause result according to the comprehensive root cause score of each candidate root cause node.

[0209] Optionally, the head-up display multi-modality abnormality test system 200 further comprises an evidence coincidence degree determination module configured to determine an evidence coincidence degree of an edge;

[0210] The evidence coincidence degree determination module comprises:

[0211] The abnormal weight determination module is configured to determine an abnormal weight corresponding to each modality event node by an abnormal weight determination algorithm;

[0212] The abnormal contribution degree determination module is configured to determine an abnormal contribution degree of a modality event node according to an abnormal score value and an abnormal weight of the modality event node;

[0213] The evidence coincidence degree determination submodule is configured to determine an evidence coincidence degree of an edge according to abnormal contribution degrees of two modality event nodes of the edge in the multi-modality factor graph;

[0214] The abnormal weight determination algorithm has an expression as follows:

[0215] ;

[0216] wherein, is an abnormal weight of an mth modality event node, a base weight for the mth modal event node, a gating value for the mth modal event node, summing products of the gating values and the base weights for all modal event nodes.

[0217] Optionally, the head-up display multi-modal anomaly testing system 200 further comprises a drift label prior determination module configured to determine a drift label prior of an edge.

[0218] The drift label prior determination module comprises:

[0219] a modal type determination module configured to determine a modal type of a modal event node in the multi-modal factor graph;

[0220] a drift label prior determination submodule configured to determine a drift label prior of an edge in the multi-modal factor graph according to the modal types of two modal event nodes of the edge and a modal type in which drift exists.

[0221] Based on the same inventive concept, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and running on the processor, and the computer program, when executed by the processor, implements the steps in the head-up display multi-modal anomaly testing method according to the first aspect of the present application.

[0222] Based on the same inventive concept, an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps in the head-up display multi-modal anomaly testing method according to the first aspect of the present application.

[0223] For the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts are referred to the part of the description of the method embodiment.

[0224] It should be noted that, for the method embodiment, in order to simply describe, the method embodiment is described as a series of action combinations, but those skilled in the art should know that the method embodiment is not limited to the described action sequence, because according to the method embodiment, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily necessary for the method embodiment.

[0225] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same and similar parts between each embodiment are referred to each other.

[0226] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In one embodiment, the application can be implemented in software and / or firmware. In particular, the application forms part of the software and / or firmware of a terminal device. The software and / or firmware can be executed on a processor-based platform.

[0227] Embodiments of the application are described herein with reference to the drawings, in which are shown flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the application. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing device, or other programmable data processing terminal devices to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal devices, create means for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more functions specified by one or more blocks Figure 1 one or more functions specified by one or more blocks.

[0228] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing terminal devices to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more functions specified by one or more blocks Figure 1 one or more functions specified by one or more blocks.

[0229] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal devices to cause a series of operational steps to be performed on the computer or other programmable terminal devices to produce a computer-implemented process such that the instructions which execute on the computer or other programmable terminal devices provide steps for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 one or more functions specified by one or more blocks Figure 1 one or more functions specified by one or more blocks.

[0230] While preferred embodiments of the application have been described, modifications and variations can be apparent to those skilled in the art once aware of the general underlying concepts. Accordingly, the appended claims are intended to encompass all modifications and variations as falling within the scope of the application.

[0231] Finally, it is to be understood that the terms such as first and second, and the like, herein are used only to distinguish one from another entity or action, and do not necessarily require or imply any such actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0232] The method, system, device and medium for head-up display multi-modal anomaly test provided by the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed, and the above description of the present application should not be understood as a limitation.

Claims

1. A multimodal anomaly testing method for head-up displays, characterized in that, The method includes: Determine the prototype vector in the prototype library that matches the current multimodal fusion vector. The multimodal fusion vector is obtained by embedding vector fusion of multimodal data related to head-up display. The prototype vector in the prototype library is a standard multimodal fusion vector corresponding to the desired display state. For the multimodal fusion vector and the matching prototype vector, anomaly scoring processing is performed on each modality to obtain the anomaly score value corresponding to each modality. Based on the abnormal score value sequence of the target mode within the first time period, the corresponding target distribution result is determined. The target mode is any one of the multimodal modes, and the abnormal score value sequence of the target mode within the first time period is a sequence composed of all abnormal score values ​​of the target mode within the first time period in chronological order. The target distribution result and baseline distribution of the target modality are verified by the statistical test method of maximizing the distribution difference, and the modal drift of the target modality is determined. The baseline distribution is the distribution of the target modality when it is normal in the corresponding expected display state. Update the alarm threshold based on the modal drift of the target mode; The test result of the target modality is determined based on the anomaly score of the target modality and the currently updated alarm threshold. The step of updating the alarm threshold based on the modal drift of the target mode includes: If the modal drift of the target mode meets its own drift determination condition, the alarm threshold of the target mode is updated by a preset threshold update algorithm to obtain the updated alarm threshold. The expression for the preset threshold update algorithm is: ,in, To update the step size; and These are the alarm thresholds for the m-th mode at time t and after the update at time t, respectively; and These are the target distribution and baseline distribution of the m-th mode at time t, respectively; For the effective modal drift amount used for updating, This represents the product of the drift direction and the intensity. Given the mean of the distribution, indicate the direction of the upward or downward shift; The KS distance between the target distribution and the baseline distribution.

2. The multimodal anomaly testing method for head-up displays according to claim 1, characterized in that, If the modal drift of the target mode meets its own drift determination condition, the alarm threshold of the target mode is updated using a preset threshold update algorithm to obtain the updated alarm threshold, including: If the modal drift of the target mode meets its own drift determination condition, the alarm threshold of the target mode is updated by a preset threshold update algorithm to obtain the updated initial alarm threshold. The updated initial alarm threshold is clamped within a preset range to obtain the updated alarm threshold.

3. The multimodal anomaly testing method for head-up displays according to claim 1, characterized in that, The method further includes: If the modal drift of the target mode returns to its normal value and continues for a preset number of times, the updated alarm threshold will be gradually restored to the original alarm threshold with a preset step size.

4. The multimodal anomaly testing method for head-up displays according to claim 1, characterized in that, The method further includes: If the modal drift of multiple modes meets their respective drift determination conditions within the second time period, the duration of drift occurrence is determined, wherein the multiple modes include at least two modes; If the duration meets the set conditions, it is determined that the current state is a version change state, the update step size is increased, and the alarm threshold preset duration of other non-drift modes is frozen.

5. The multimodal anomaly testing method for head-up displays according to claim 1, characterized in that, In cases where multimodal data includes image data, text sequence data, signal data, and speech data, the current multimodal fusion vector is determined, including: Collect multimodal data related to head-up displays; The multimodal data is timestamped using a unified timestamp and then encapsulated into a synchronization data packet using a sliding window. The image data in the synchronization data packet is unified in resolution and normalized in brightness and gamma to obtain the target image data. The text sequence data in the synchronization data packet is uniformly converted into Tokens and concatenated into a fixed-length sequence to obtain the target text sequence data; The signal data in the synchronization data packet is padded with time axis to form an equally spaced data stream to obtain the target signal data; Endpoint detection and noise suppression preprocessing are performed on the voice data in the synchronization data packet, and embedding representation extraction is performed on the preprocessed voice data to obtain the target voice data; The preprocessed multimodal data are unified into the same sampling window, and multimodal window data is constructed by a sliding window; For each modality in the multimodal window data, the corresponding encoder is called to encode it separately to obtain the embedding vector corresponding to each modality; The embedding vectors of each modality are weighted and fused to obtain the current multimodal fusion vector.

6. The multimodal anomaly testing method for head-up displays according to claim 5, characterized in that, The obtained embedding vectors for each modality are weighted and fused to obtain the current multimodal fusion vector, including: By using a dual-tower comparison, the embedding vectors corresponding to the image modality and the text modality in the multimodal window data are aligned to a unified semantic space to obtain unified image embedding vectors and unified text embedding vectors. Based on the quality of each modality in the multimodal window data, determine the corresponding gate value for each modality; Based on the gating value and baseline weight of the same modality, the initial fusion weight of the corresponding modality is determined; The initial fusion weights of all modalities are normalized to obtain the fusion weights corresponding to each modality. Based on the fusion weights of each modality, the target embedding vectors of each modality are weighted and fused to obtain a multimodal fusion vector; among them, the target embedding vector of the image modality is the unified image embedding vector, the target embedding vector of the text modality is the unified text embedding vector, and the target embedding vectors of other modalities are the embedding vectors obtained by encoding through the corresponding encoder.

7. The multimodal anomaly testing method for head-up displays according to claim 1, characterized in that, Building a prototype library includes: During the continuous acquisition of multimodal window data under the desired display state, determine whether the conditions for ending the acquisition are met; Under the condition of ending the acquisition, each acquired multimodal window data is processed into its corresponding multimodal fusion vector, and then averaged in a unified semantic space to obtain the prototype vector in the desired display state. If the number of all collected multimodal window data satisfies the first condition, the covariance matrix of the desired display state in the unified semantic space is calculated based on the multimodal fusion vector of all collected multimodal window data. If the number of all collected multimodal window data satisfies the second condition, the shrinkage covariance matrix of the desired display state in the unified semantic space is calculated based on the multimodal fusion vector of all collected multimodal window data. If the number of all collected multimodal window data satisfies the third condition, the diagonal covariance matrix of the desired display state in the unified semantic space is calculated based on the multimodal fusion vector of all collected multimodal window data.

8. The multimodal anomaly testing method for head-up displays according to claim 7, characterized in that, The data collection ends when any two of three conditions are met; these three conditions are: The mean difference between the multimodal fusion vectors of the latest two batches of multimodal window data is lower than the first threshold, and the number of multimodal window data in a batch is a preset value; The shrinkage coefficient of the shrinkage covariance between the multimodal fusion vectors of the latest two batches of multimodal window data converges to less than or equal to the second threshold, and the relative change of the diagonal variance between the multimodal fusion vectors of the latest two batches of multimodal window data is less than the third threshold. The KS distance of the multimodal fusion vector distribution of the latest two batches of multimodal window data is less than or equal to the fourth threshold.

9. The multimodal anomaly testing method for head-up displays according to claim 1, characterized in that, Determine the prototype vectors in the prototype library that match the current multimodal fusion vector, including: The matching degree between each prototype vector in the prototype library and the current multimodal fusion vector is determined by a joint matching algorithm. Based on all the obtained matching degrees, determine the prototype vector in the prototype library that matches the current multimodal fusion vector; The expression for the joint matching algorithm is: Where c is the matching degree. The Mahalanobis distance, For cosine similarity, Let be the multimodal fusion vector at time t. The prototype vector is from the prototype library. To correspond to the desired display state in the prototype vector The covariance matrix under the following conditions These are the weighting coefficients.

10. The multimodal anomaly testing method for head-up displays according to claim 1, characterized in that, For the multimodal fusion vector and the matching prototype vector, anomaly scoring processing is performed on each modality to obtain the anomaly score value corresponding to each modality, including: An anomaly scoring algorithm is used to perform anomaly scoring processing on the multimodal fusion vector and the matching prototype vector for each modality, thereby obtaining the anomaly score value corresponding to each modality. The expression for the anomaly scoring algorithm is: ； in, The anomaly score for the m-th mode; These are the weighting coefficients for the joint evaluation of the two corresponding modalities; Mahalanobis distance is used to measure differences in distribution scale. Let be the single-modal embedding vector of the m-th modality at time t in the unified semantic space; The single-modal embedding vector of the m-th modality in the unified semantic space is the matching prototype vector; To correspond to the desired display state in the single-modal embedding vector The covariance matrix under the given conditions.

11. The multimodal anomaly testing method for head-up displays according to claim 1, characterized in that, The method further includes: When an abnormal test result occurs, a multimodal factor graph is constructed based on the multimodal data collected within the corresponding time period. The multimodal factor graph includes each modal event node and the edges between each modal event node. The edges are oriented based on the call dependencies and time sequence between each modal event node. The initial attention weights of each edge in the multimodal factor graph are calculated using a typed linear mapping and attention mechanism. The calculation expression is as follows: ,in, This represents the strength of the edge connection between node i and node j. Let the initial attention weights be those between node i and node j; For timing consistency; For signal quality, the gate value of the mode to which node i belongs is taken. The gate value of the mode to which node j belongs. geometric mean ; Historical dependence; The degree of agreement between the evidence and the edge; The drift label is a priori for the edge; The weighting coefficients are: the temporal consistency, the signal quality, the historical dependence, the evidence consistency, and the drift label prior, which are all dimensionless scores mapped to [0,1] for their respective features. Based on the initial attention weight, a preset number of message passing rounds are executed, and the initial attention weight is iteratively updated to obtain the influence strength of each edge on the abnormal output node, wherein the abnormal output node is the modal event node corresponding to the abnormal modality; Starting from the candidate root cause node and ending at the abnormal output node, enumerate all candidate paths whose path length does not exceed a preset value. The influence of the path corresponding to the candidate root cause node is determined based on the influence strength of all edges in all candidate paths corresponding to the candidate root cause node. The candidate root cause nodes are subjected to virtual ablation of root cause analysis, and the joint anomaly score value after ablation is calculated. The joint anomaly score value is the sum of the anomaly contribution of each mode under the same data window. The anomaly contribution of each mode is determined based on the anomaly score value of the mode and the corresponding anomaly weight. The drop value corresponding to the candidate root cause node is determined based on the joint anomaly score after ablation and the joint anomaly score before ablation. Based on the drop value and the path influence of the candidate root cause node, the comprehensive root cause score of the candidate root cause node is determined. The final root cause result is determined based on the comprehensive root cause score of each candidate root cause node.

12. The multimodal anomaly testing method for head-up displays according to claim 11, characterized in that, Determining the degree of evidence consistency for each edge includes: The anomaly weight determination algorithm is used to determine the anomaly weight corresponding to each modal event node. The abnormal contribution of the modal event node is determined based on its abnormal score and abnormal weight. The evidence consistency of an edge is determined based on the anomalous contribution of the two modal event nodes of the edge in the multimodal factor graph. The expression for the abnormal weight determination algorithm is: ； in, Let be the anomaly weight of the m-th modal event node. The base weights for the m-th modal event node are: Let be the gate value for the m-th modal event node. This involves summing the product of the gating values ​​and the basic weights for all modal event nodes.

13. The multimodal anomaly testing method for head-up displays according to claim 11, characterized in that, Determine the prior knowledge of edge drift labels, including: Determine the modal type of the modal event node in the multimodal factor graph; Based on the modal types of the two modal event nodes of the edge in the multimodal factor graph and the modal type where drift exists, the drift label prior of the edge is determined.

14. A head-up display multimodal anomaly testing system, characterized in that, The system includes: The prototype vector determination module is used to determine the prototype vector in the prototype library that matches the current multimodal fusion vector. The multimodal fusion vector is obtained by embedding vector fusion of multimodal data related to head-up display. The prototype vector in the prototype library is a standard multimodal fusion vector corresponding to the desired display state. An anomaly scoring module is used to perform anomaly scoring processing on the multimodal fusion vector and the matching prototype vector for each modality, and obtain the anomaly score value corresponding to each modality. The target distribution result determination module is used to determine the corresponding target distribution result based on the abnormal score value sequence of the target mode within a first time period. The target mode is any one of the multimodal modes, and the abnormal score value sequence of the target mode within the first time period is a sequence composed of all abnormal score values ​​of the target mode within the first time period in chronological order. The modal drift determination module is used to verify the target distribution result and baseline distribution of the target modality by using a statistical test method that maximizes the distribution difference, and to determine the modal drift of the target modality. The baseline distribution is the distribution of the target modality when it is normal under the corresponding expected display state. The alarm threshold update module is used to update the alarm threshold according to the modal drift of the target mode; The evaluation module is used to determine the test result of the target modality based on the anomaly score value of the target modality and the currently updated alarm threshold. The alarm threshold update module is specifically used to update the alarm threshold of the target mode by means of a preset threshold update algorithm when the modal drift of the target mode meets its own drift determination condition, so as to obtain the updated alarm threshold. The expression for the preset threshold update algorithm is: ,in, To update the step size; and These are the alarm thresholds for the m-th mode at time t and after the update at time t, respectively; and These are the target distribution and baseline distribution of the m-th mode at time t, respectively; For the effective modal drift amount used for updating, This represents the product of the drift direction and the intensity. Given the mean of the distribution, indicate the direction of the upward or downward shift; The KS distance between the target distribution and the baseline distribution.

15. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and running on the processor, wherein the computer program, when executed by the processor, implements the steps of a head-up display multimodal anomaly testing method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a head-up display multimodal anomaly testing method as described in any one of claims 1 to 13.

Images