Evaluation method for cross-modal input generation based on weak association rules and dynamic gradients

Abstract

This invention relates to the field of large model testing technology, specifically to an evaluation method for cross-modal input generation based on weak association rules and dynamic gradients. The method includes the construction and invocation of a scenario-specific interference library, the generation of scenario-specific trimodal non-ideal inputs, the construction and binding of weak association rules, dynamic gradient adjustment, dual-dimensional evaluation of cross-modal responses, temporal consistency verification and result output, and an interference source tracing and optimization suggestion mechanism. This achieves synchronous distortion simulation of text-visual-auditory trimodals, accurately reproducing the weak association signal relationships of "appearance matching and logical conflict." The deviation between the test scenario and the real application environment is significantly reduced, and the probability of response deviation and functional failure after model deployment is significantly decreased. Based on a scenario-specific interference library and dynamic gradient adjustment, the interference intensity and type are adapted to different scenario characteristics, avoiding the limitations of general fixed interferences and significantly improving the scenario fit of non-ideal inputs.

Description

Technical Field

[0001] This invention relates to the field of large model testing technology, specifically to an evaluation method for cross-modal input generation based on weak association rules and dynamic gradients. Background Technology

[0002] Traditional large-scale model testing uses only random text as input, which cannot reproduce the complex environment of simultaneous distortion in multiple dimensions of text, vision, and hearing. It is also difficult to simulate the weak correlation signal relationship of appearance matching and actual conflict in real scenarios. In addition, it uses general fixed-intensity interference, which cannot be adapted to the specific interference characteristics of different scenarios. This results in a large deviation between the test and the real application environment, and the model is prone to response deviation and functional failure after going online.

[0003] Meanwhile, existing evaluation methods prioritize text over modality, neglect cross-modal processing biases, and are based solely on static input from a single frame, failing to capture the temporal shortcomings of continuous cross-modal input streams. Furthermore, they only output scores and simple reports, without establishing a correlation link between "evaluation metrics - interference types - model defects," making it difficult to translate evaluation results into model optimization actions, leading to a disconnect between evaluation and optimization. Summary of the Invention

[0004] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0005] An evaluation method for cross-modal input generation based on weak association rules and dynamic gradients includes the following steps:

[0006] S1. Construction and invocation of scenario-specific interference library, namely, constructing a scenario-specific interference library with a three-level architecture based on cross-modal scenario interference library, to provide accurate scenario basis for input generation;

[0007] S2, Scenario-based trimodal non-ideal input generation, that is, based on the scenario adaptation results of S1, trimodal non-ideal input is generated according to the process of basic resource loading - submodal distortion generation - temporal alignment binding;

[0008] S3. Weak association rule construction and binding: Based on the feature extraction-weighted mapping-intensity control architecture, weak association binding between text and cross-modal signal appearance matching and logical conflict is realized, and the association strength is stably controlled within a preset range.

[0009] S4. Dynamic gradient adjustment, which is to construct a gradient classification-deviation calculation-closed-loop adjustment-dual-condition verification mechanism, dynamically optimize the interference intensity based on the model response, and accurately locate the anti-interference critical threshold.

[0010] S5, Cross-modal response dual-dimensional evaluation, that is, to build a three-dimensional evaluation framework, covering three levels: text dimension, modal dimension and scene adaptation, evaluate model performance through multiple quantitative indicators, and output the total score by weight fusion;

[0011] S6, Temporal Consistency Verification and Result Output: Specifically, for the needs of temporal scenarios, a module is built to generate temporal input, calculate temporal indicators, and output a visual report, capturing the hidden shortcomings of the model under continuous input.

[0012] S7. Interference tracing and optimization suggestion mechanism, namely, constructing a tracing matrix-defect ranking-suggestion generation-feedback closed-loop mechanism, establishing a direct correlation between evaluation results and optimization actions, and realizing the evaluation-optimization closed loop.

[0013] A further technical solution is that steps S1 and S2 include core logic for scene adaptation, multimodal parameterization generation, and temporal alignment assurance. The core logic for scene adaptation includes feature vector construction, scene similarity calculation, and scene similarity threshold calculation. The scene similarity threshold calculation includes sub-scene feature cohesion. Sub-scene historical effective similarity statistics Scene accuracy requirement coefficient With a preset threshold for scene similarity, the feature vector construction includes ensuring that the feature vector dimensions of the input scene and the sub-scenes in the interference library are unified. The specific structure is as follows: Scene function encoding: describes the core functions of the scene, represented using one-hot encoding. Environmental characteristic values: including environmental parameters such as temperature, light intensity, and noise levels (decibels), expressed as normalized continuous values. Arrange the P normalized environmental parameters in a fixed order to form an environmental feature vector: The interaction method parameter describes the form of interaction between the user and the model, represented by binary encoding. Interference source characteristics: Describe the sources and attributes of typical interference in the scene, represented by multi-label encoding: All feature vectors are adopted Normalization is performed to eliminate the interference of different feature magnitudes on similarity calculation and ensure matching fairness. The above vectors are concatenated in a fixed order of scene function encoding, environmental feature values, interaction mode parameters, and interference source features to obtain the total feature vector of the input scene. Its dimension D = M + P + Q + R, and its mathematical expression is: The scene similarity calculation includes using a cosine similarity metric to quantify the matching degree between the input scene and sub-scenes in the library, as shown in the following formula: Multimodal parameterized generation includes visual signal distortion generation, auditory signal distortion generation, and text signal distortion generation. Visual signal distortion generation employs a layered overlay strategy of basic distortion plus scene-specific distortion to simulate visual interference in different scenarios. This includes basic distortion 1: Gaussian blur, with the formula: Basic Distortion 2: RGB channel offset, the formula is: Scene-specific distortion: designed for the characteristics of sub-scenes; Auditory signal distortion generation includes using a time-frequency domain joint distortion model to simulate the non-stationary characteristics of noise in real-world scenes. Specifically, this includes the first step: Short-Time Fourier Transform (STFT); performing STFT on the original auditory signal to convert the time-domain signal into a time-frequency representation. The second step: generating the noisy signal, the formula is as follows. Step 3: Noise mixing model. Scene noise is dynamically mixed from three types of noise. Formula: Step 4: Global Signal-to-Noise Ratio Constraints. To ensure controllable noise intensity, the global signal-to-noise ratio must meet the following requirements: Timing alignment guarantees include timestamp binding and dynamic delay compensation. Timestamp binding assigns a unique timestamp to each frame / segment of each modal signal, satisfying the following: Dynamic delay compensation is necessary because, due to differences in hardware output delay, the delay values ​​of each mode need to be recorded through pre-testing and compensated in advance when allocating timestamps.

[0014] A further technical solution is that step S3 includes feature extraction and weighted mapping, and weak correlation strength calculation. Feature extraction and weighted mapping includes cross-modal feature extraction, multi-head attention weighted mapping, and optimized feature vectors. Specifically, multi-head attention weighted mapping uses a multi-head attention mechanism to replace traditional linear weighting, improving the precision of surface feature matching through multi-subspace interaction and scene feature fusion. The formula is as follows: A single attention head is calculated as The weak association strength is calculated by balancing surface similarity and logical conflict through a dynamic α coefficient, ensuring that the association strength remains stable at a certain level. The formula is as follows: The range of weak correlation strength is set to the parameters defined for real-world sample data, i.e., for the first... Real-world scenario samples Define its inherent parameters; calculate the statistical distribution characteristics of the sample set, that is, for one sample Calculate the core statistics that reflect the distribution patterns of real-world scenarios; derive the upper and lower limits of preset intervals, specifically through interval derivation based on quantiles or confidence interval derivation based on normal distribution.

[0015] A further technical solution involves a dynamic gradient generation closed-loop mechanism in step S4, including disturbance gradient classification, deviation rate calculation, gradient adjustment decision, and verification. The deviation rate in the deviation rate calculation is a core indicator for measuring the quality of the model response and is used to determine whether the current disturbance intensity needs adjustment. The formula is as follows: .

[0016] A further technical solution is that, in step S5, the cross-modal response dual-dimensional evaluation includes text dimension evaluation, modal dimension evaluation, and total score calculation. Specifically, the modal dimension evaluation includes modality binding error rate, semantic pointing confusion, scene interference adaptation, and modality comprehensive deviation. The semantic pointing confusion is statistically analyzed according to visual and auditory modalities, and the unified basic formula is: The total score is calculated using a modality-first weighting principle, and the calculation formula is as follows: .

[0017] A further technical solution is that, in step S6, the timing consistency verification and result output includes timing input generation parameters, timing index calculation, and basic report output. The timing index calculation involves designing three core indicators to quantify the timing stability of the model under continuous input streams. The three core indicators include cross-frame correlation contradiction rate, semantic update delay time, and correlation drift frequency. The basic report output includes both static and timing results.

[0018] A further technical solution is that, in step S7, the interference tracing and optimization suggestion mechanism includes the construction of a three-dimensional tracing matrix, defect priority ranking, and the generation of three-dimensional optimization suggestions. The defect priority ranking includes the calculation of defect impact weights, which are calculated based on three dimensions: indicator compliance, interference intensity, and impact range (in the range of 0-1). The higher the weight, the higher the defect priority, as shown in the following formula. .

[0019] Compared with the prior art, the beneficial effects of the present invention are:

[0020] 1. Achieve simultaneous distortion simulation across text, vision, and hearing modalities, accurately reproducing weakly correlated signal relationships such as "appearance matching and logical conflict." The deviation between the test scenario and the real application environment is significantly reduced, and the probability of response deviation and functional failure after the model goes live is significantly decreased. Based on a scenario-specific interference library and dynamic gradient adjustment, the interference intensity and type are adapted to different scenario characteristics, avoiding the limitations of general fixed interference and greatly improving the scenario fit for non-ideal inputs.

[0021] 2. Construct a four-dimensional evaluation framework of "text-modality-scene-temporal sequence" to overcome the shortcomings of traditional models that "emphasize text over modality" and "emphasize static data over temporal data," comprehensively capturing the performance shortcomings of models under both static and continuous input conditions. Establish a three-dimensional source matrix of "evaluation metrics-interference types-model defects," outputting actionable optimization suggestions of "structure-parameters-data," and creating a closed loop of "evaluation results → defect localization → model optimization," allowing evaluation to move beyond score output and directly translate into model iteration actions.

[0022] 3. The dynamic gradient adjustment mechanism accurately locates the critical threshold for model anti-interference, avoids ineffective testing resource consumption, and improves testing efficiency compared to traditional fixed-intensity interference. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0025] Example:

[0026] Figure 1 This paper illustrates a preferred embodiment of the evaluation method for cross-modal input generation based on weak association rules and dynamic gradients according to the present invention. The evaluation method for cross-modal input generation based on weak association rules and dynamic gradients in this embodiment specifically includes the following steps:

[0027] S1. Construction and invocation of scenario-specific interference library, namely, constructing a scenario-specific interference library with a three-level architecture based on cross-modal scenario interference library, to provide accurate scenario basis for input generation;

[0028] S2, Scenario-based trimodal non-ideal input generation, that is, based on the scenario adaptation results of S1, trimodal non-ideal input is generated according to the process of basic resource loading - submodal distortion generation - temporal alignment binding;

[0029] S3. Weak association rule construction and binding: Based on the feature extraction-weighted mapping-intensity control architecture, weak association binding between text and cross-modal signal appearance matching and logical conflict is realized, and the association strength is stably controlled within a preset range.

[0030] S4. Dynamic gradient adjustment, which is to construct a gradient classification-deviation calculation-closed-loop adjustment-dual-condition verification mechanism, dynamically optimize the interference intensity based on the model response, and accurately locate the anti-interference critical threshold.

[0031] S5, Cross-modal response dual-dimensional evaluation, that is, to build a three-dimensional evaluation framework, covering three levels: text dimension, modal dimension and scene adaptation, evaluate model performance through multiple quantitative indicators, and output the total score by weight fusion;

[0032] S6, Temporal Consistency Verification and Result Output: Specifically, for the needs of temporal scenarios, a module is built to generate temporal input, calculate temporal indicators, and output a visual report, capturing the hidden shortcomings of the model under continuous input.

[0033] S7. Interference tracing and optimization suggestion mechanism, namely, constructing a tracing matrix-defect ranking-suggestion generation-feedback closed-loop mechanism, establishing a direct correlation between evaluation results and optimization actions, and realizing the evaluation-optimization closed loop.

[0034] Specifically, the construction and invocation of the scenario-based interference library involves building a scenario-specific interference library with a three-level architecture based on cross-modal scenario interference, providing accurate scenario basis for input generation.

[0035] The library architecture is designed as follows: the first level is divided according to the core application domains across modalities; the second level is divided into subdivided scenarios within the application domains; and the third level is divided into interference types, which are further subdivided by modality (text-side interference includes semantic breaks, grammatical errors, etc.; visual-side interference includes Gaussian blur, scene-specific glare, etc.; auditory-side interference includes white noise, scene-specific mechanical noise, etc.). The library stores scene feature labels, real interference instances, and parameter mapping tables (recording typical parameter ranges, occurrence probabilities, and intensity distributions of interference in different scenarios).

[0036] Scene matching mechanism: Extract feature vectors from the input scene (including scene function encoding, environmental feature values, etc., with dimensions consistent with the labels in the library), and match them with sub-scenes in the library through cosine similarity calculation. Based on the matching degree, directly call the interference instance and parameter mapping table of the corresponding sub-scene or extract the basic interference signal of the same application category, and superimpose the high-frequency features of the input scene to generate adapted interference resources.

[0037] Dynamic update mechanism: Connect to the operation and maintenance database to synchronize scenario operation logs, extract new interference data (including signal samples, scenario information, etc.); after preprocessing the new data (noise reduction, feature standardization, etc.), use the K-means algorithm to cluster and remove duplicates (merge high similarity data according to interference feature vectors); supplement the deduplicated data into the database according to the three-level architecture, and update the parameter mapping table.

[0038] In step S2, the scene adaptation results based on S1 are used to generate the trimodal non-ideal input in the scene-based trimodal non-ideal input generation process, which follows the process of "basic resource loading - submodal distortion generation - temporal alignment binding".

[0039] The basic resource loading includes: on the text side, loading a scene-specific vocabulary library (including core terms and commonly used commands) and a general corpus; on the visual side, loading typical visual materials for the scene (uniformly in RGB standard format, with resolution adapted to scene requirements); and on the auditory side, loading basic audio materials for the scene (uniform sampling rate to ensure consistent quality standards).

[0040] Multimodal Non-Ideal Signal Generation: Text Signal: Employs a "scene keyword hybrid extraction + grammatical recombination" strategy, extracting relevant words from a scene-specific vocabulary and randomly selecting words from a general corpus. The extracted words are then processed by adjusting sentence structure, inserting redundant components, and scrambling some semantics, while retaining core keywords. Visual Signal: Employs a layered generation method of "basic distortion superimposed with scene-specific distortion," where the basic distortion includes Gaussian blur and RGB channel offset. Auditory Signal: Employs a "scene noise mixing + parameter adjustment" method, mixing white noise, pink noise, and scene-specific noise (the sum of the three types being 1) according to the scene parameter mapping table proportions. The interference intensity is adjusted through the signal-to-noise ratio.

[0041] Three-modal timing alignment: Achieving parallel output and timing synchronization of three modes through a timestamp binding mechanism, assigning a unique timestamp to each frame / segment of each mode signal, using the Network Time Protocol (NTP) to achieve cross-device clock synchronization, and introducing dynamic compensation for hardware output delay differences (pre-testing and recording delays, and compensating in advance when allocating timestamps), ultimately ensuring that the timestamp synchronization error is less than or equal to the global synchronization error.

[0042] In step S3, the weak association rule construction and binding is based on the "feature extraction-weighted mapping-intensity control" architecture to achieve weak association binding between text and cross-modal signals based on "appearance matching and logical conflict." The core is to stably control the association strength within a preset range, specifically including:

[0043] Cross-modal feature extraction: Prioritize the extraction of scene-related representational and logical features. On the text side, extract keywords, syntactic structure, and core semantics; on the visual side, extract representational features such as color and shape, as well as logical relationships between elements (using CNN extraction); on the auditory side, extract representational features such as frequency and amplitude, as well as signal logical orientation (using MFCC extraction); and extract text features using a word embedding model.

[0044] Weighted matching relationship establishment: Establish a weighted matching relationship between text and cross-modal surface features, and introduce scene correlation coefficient to enhance adaptability. The higher the importance, the larger the scene correlation coefficient.

[0045] Weak association strength regulation: The association strength is controlled by dynamically balancing similarity and logical conflict through the scene adaptation coefficient α. α is categorized according to the degree of cross-modal association dependency, and adjustment trigger conditions include scene switching, deviation of association strength from the target interval by a preset range, and model version updates. The adjustment uses a PID control algorithm to ensure that the strength converges to the target interval. Simultaneously, a semantic consistency judgment model trained on association conflict samples is used to quantify the logical consistency between text and cross-modal signals, ensuring a "appearance matching, logical conflict" effect.

[0046] In step S4, dynamic gradient adjustment is implemented by constructing a "gradient classification - deviation calculation - closed-loop adjustment - dual-condition verification" mechanism. This mechanism dynamically optimizes the interference intensity based on the model response and accurately locates the critical threshold for anti-interference, including:

[0047] Interference gradient classification: The interference intensity of visual and auditory signals is divided into 5 levels to ensure that the differences between gradients are significant and cover the range of interference that the model may face.

[0048] Deviation rate calculation: The deviation rate is calculated based on the model response results and used as the basis for gradient adjustment. Deviation rate judgment criteria: Text response failure refers to any one of the following being lower than the corresponding preset pass standard: semantic understanding accuracy, logical rationality score, output validity, or scene relevance score; cross-modal processing deviation refers to any one of the following: modality binding error rate, semantic pointing confusion exceeding the preset threshold, or scene interference adaptability lower than the preset threshold.

[0049] Gradient adjustment strategy: Based on the model bias rate and the reasonable range of the scene, dynamically enhance / weaken the interference intensity, and locate the critical threshold for anti-interference by double verification of weak correlation strength and scene adaptability.

[0050] Dual-condition verification: After gradient adjustment, two verifications need to be completed: ① weak correlation strength verification; ② scene interference feature matching degree verification. If the verification passes, the current gradient is determined to be an adaptive gradient; if it fails, the "adjustment-verification" process is repeated, and the critical threshold for anti-interference is located by the deviation rate mutation point.

[0051] In step S5, a three-dimensional evaluation framework is constructed in the cross-modal response dual-dimensional evaluation, covering three levels: "text dimension - modal dimension - scene adaptation". The model performance is evaluated through multiple quantitative indicators, and the total score is output by weighted fusion (0-1 range, the closer to 1, the stronger the ability).

[0052] Text-based assessment: Four core metrics are designed to measure the ability to process distorted text.

[0053] Modal dimension assessment: The deviation is quantified by three indicators: modal binding error rate, semantic pointing confusion, and scene interference adaptability. The deviation is then fused into a comprehensive modal deviation by the root mean square.

[0054] Total score calculation: Weights are set according to the "modal priority" principle (the core requirement for cross-modal applications is modal collaboration), and the text comprehensive score and modal positive score are merged (modal comprehensive deviation is converted into positive score), with the weights of the two types of scores summed to 1.

[0055] In step S6, the timing consistency verification and result output are designed to meet the needs of timing scenarios. The module "Timing Input Generation - Timing Index Calculation - Visual Report Output" is constructed to capture the hidden shortcomings of the model under continuous input. Specifically, it includes: Timing cross-modal input stream generation: generating continuous three-modal input streams, with parameters dynamically adjusted according to the timing sensitivity of the scenario (high / medium / low) to ensure synchronization accuracy.

[0056] Time series metric calculation: Quantify time series stability by correlating drift frequency metrics.

[0057] Basic report outputs: output static evaluation results, time-series evaluation results, and high-bias frame visualization annotations.

[0058] In step S7, the interference source tracing and optimization suggestion output constructs a "source tracing matrix - defect ranking - suggestion generation - feedback closed loop" mechanism, establishing a direct correlation between evaluation results and optimization actions, and realizing the "evaluation-optimization" closed loop, specifically including:

[0059] 3D source tracing matrix construction: Establish a 3D matrix of "evaluation index - interference dimension - defect type" and mine the correlation from historical test data using the Apriori algorithm. The matrix is ​​updated by mining again as new data is accumulated.

[0060] Defect priority ranking: Defects are ranked according to their impact on model performance and the intensity of interference.

[0061] 3D optimization suggestion generation: Output structure, parameters, and data 3D optimization suggestions, and update the suggestion library through feedback closed-loop iteration.

[0062] Feedback closed-loop iteration: The strategy library is iterated through suggestion quality assessment - calculate the difference in the total score of the model before and after optimization, suggestions with quality ≥ the preset standard are added to the strategy library (indexed by "defect type-scenario-suggestion"), suggestions with quality < the preset standard are reviewed and analyzed, and the matrix and suggestion generation logic are updated.

[0063] In the cross-modal collaborative generation mechanism, this mechanism constructs composite interference input through a three-level process of "scene adaptation - parameterized generation - temporal alignment". The core is to realize the dynamic binding of scene features and distortion parameters, accurately simulate the multi-dimensional synchronous distortion scene of "text-visual-auditory" in the real environment, and solve the problem of the disconnect between "single modality" and "general interference" and the real scene in traditional testing.

[0064] The core logic of scene adaptation (corresponding to S1 scene matching) is to match the most suitable interference resources for the input scene through feature vector similarity calculation, thereby avoiding the problem of "universal but inaccurate" general interference.

[0065] Steps S1 and S2 include core logic for scene adaptation, multimodal parameterization generation, and temporal alignment assurance. The core logic for scene adaptation includes feature vector construction, scene similarity calculation, and scene similarity threshold calculation. The scene similarity threshold calculation includes sub-scene feature cohesion. Sub-scene historical effective similarity statistics Scene accuracy requirement coefficient With a preset threshold for scene similarity, the feature vector construction includes ensuring that the feature vector dimensions of the input scene and the sub-scenes in the interference library are unified. The specific structure is as follows: Scene function encoding: describes the core functions of the scene, represented using one-hot encoding. ,in Scene function encoding vector, dimension M; environmental feature values: including environmental parameters such as temperature, light intensity, and noise decibels, represented by normalized continuous values. ,in Environmental parameters The original measurement value, For parameters The minimum value in historical scenario data. For parameters The maximum value in historical scenario data. The normalized parameter values ​​range from [0,1] to ensure a consistent numerical range for different parameters. The P normalized environmental parameters are arranged in a fixed order to form an environmental feature vector. The interaction method parameter describes the form of interaction between the user and the model, represented by binary encoding. , The interaction method parameter vector has dimension Q; interference source features describe the sources and attributes of typical interference in the scene, represented by multi-label encoding. , The feature vectors are the interference source features, with dimension R; all feature vectors are adopted. Normalization is performed to eliminate the interference of different feature magnitudes on similarity calculation and ensure matching fairness. The above vectors are concatenated in a fixed order of scene function encoding, environmental feature values, interaction mode parameters, and interference source features to obtain the total feature vector of the input scene. Its dimension D = M + P + Q + R, and its mathematical expression is: The scene similarity calculation includes using a cosine similarity metric to quantify the matching degree between the input scene and sub-scenes in the library, as shown in the following formula: Multimodal parameterized generation includes visual signal distortion generation, auditory signal distortion generation, and text signal distortion generation. Visual signal distortion generation employs a layered overlay strategy of basic distortion plus scene-specific distortion to simulate visual interference in different scenarios. This includes basic distortion 1: Gaussian blur, with the formula: Basic Distortion 2: RGB channel offset, the formula is: Scene-specific distortion: designed for the characteristics of sub-scenes; Auditory signal distortion generation includes using a time-frequency domain joint distortion model to simulate the non-stationary characteristics of noise in real-world scenes. Specifically, this includes the first step: Short-Time Fourier Transform (STFT); performing STFT on the original auditory signal to convert the time-domain signal into a time-frequency representation. The second step: generating the noisy signal, the formula is as follows. Step 3: Noise mixing model. Scene noise is dynamically mixed from three types of noise. Formula: Step 4: Global Signal-to-Noise Ratio Constraints. To ensure controllable noise intensity, the global signal-to-noise ratio must meet the following requirements: ,in The power of the effective signal, To control the power of mixed noise, the total noise intensity is limited by global SNR to prevent excessive local noise from causing complete signal distortion and to ensure that the model still has some processing capacity. Timing alignment guarantees include timestamp binding and dynamic delay compensation. Timestamp binding assigns a unique timestamp to each frame / segment of each modal signal, satisfying the following: Dynamic delay compensation is necessary because, due to differences in hardware output delay, the delay values ​​of each mode need to be recorded through pre-testing and compensated in advance when allocating timestamps.

[0066] The weak association rule quantification design aims to achieve "appearance matching and logical conflict" between text and cross-modal signals. Through a three-level architecture of "feature extraction - weighted mapping - intensity regulation", the association strength is stably controlled within the interval [L, U].

[0067] Step S3 includes feature extraction and weighted mapping, and weak association strength calculation. Feature extraction and weighted mapping includes cross-modal feature extraction, multi-head attention weighted mapping, and optimized feature vectors. Specifically, multi-head attention weighted mapping uses a multi-head attention mechanism to replace traditional linear weighting, and improves the precision of surface feature matching through multi-subspace interaction and scene feature fusion. The formula is as follows: A single attention head is calculated as , These are respectively "query vector (text features)," "key vector (cross-modal surface features)," and "value vector (target features to be fused)," with dimensions of batch size, sequence length, and feature dimension for the adaptation model. No. The learnable projective weights for each attention head, with dimensions that are the fit values ​​between the feature dimension and the subspace dimension (subspace dimension = feature dimension / number of attention heads), serve to... Projecting onto a subspace enables multi-subspace matching. The dimension of the key vector. This is used to alleviate the problem of "softmax gradient vanishing due to excessively large inner product". This is the scene correlation coefficient, and its value range conforms to the setting of the scene's dependence on surface feature matching. The more the scene depends on surface feature matching, the higher the correlation coefficient becomes. The larger, No. Feature vectors of each sub-scene Yes The multilayer perceptron transformation is used to incorporate scene information into the attention score calculation. The normalization function maps the attention scores to the [0,1] interval, ensuring that the sum of the weights is 1. For the concatenation function, The outputs of each attention head are concatenated to form the original feature dimension. The weights are the linear transformation weights after concatenation, with the same dimension as the original features. Their function is to fuse the matching results from multiple subspaces and output the final weighted features. The weak association strength is calculated by balancing surface similarity and logical conflict through a dynamic α coefficient, ensuring the association strength remains stable at a certain level. The formula is as follows: The range of weak correlation strength is set to the parameters defined for real-world sample data, i.e., for the first... Real-world scenario samples Define its inherent parameters; calculate the statistical distribution characteristics of the sample set, that is, for one sample Calculate the core statistics that reflect the distribution patterns of real-world scenarios; derive the upper and lower limits of preset intervals, specifically through interval derivation based on quantiles or confidence interval derivation based on normal distribution.

[0068] Calculate the statistical distribution characteristics of a sample set for N samples. Calculate the core statistics that reflect the distribution patterns in real-world scenarios. The sample mean. Define the sample standard deviation. quantiles are , To round down, quantiles are Since the distribution of weak association strength in real-world scenarios may not be normal, quantile intervals are more robust. The derived result is as follows: lower limit L of the interval: taking the sample quantiles, i.e. Upper limit of the interval U: the sample size quantiles, i.e. Preset interval formula .

[0069] In the dynamic gradient generation closed-loop mechanism, this mechanism uses "gradient classification - deviation calculation - closed-loop adjustment - dual-condition verification" to dynamically optimize the interference intensity based on the real-time response of the model, accurately locate the critical threshold of the model's anti-interference, and solve the problem that the robustness of traditional "fixed intensity interference" cannot be quantified.

[0070] The gradient splitting is based on pre-testing of cross-modal models with different architectures that meet statistical validity requirements, plotting an "interference intensity-deviation rate" curve, and using the point of abrupt change in the slope of the curve as the gradient splitting point to ensure that each level of gradient can distinguish the differences in the model's anti-interference ability.

[0071] The interference gradient parameters are formally defined. First, the multi-dimensional interference parameters are integrated into a single-dimensional intensity value through the "interference intensity quantization function". Then, the gradient boundary threshold is determined through the "slope change point detection". Finally, the parameter range of each gradient (G1~G5) is as follows, specifically for visual and auditory modalities: Interference intensity quantization function, the interference intensity is quantified by the "deviation degree between the parameter and the global mean". The weight is calculated by the "proportion of modal feature labels" in the S1 scene feature vector (e.g., if the visual label contains "Gaussian blur" or "RGB offset", the weight is the proportion of the corresponding label in all visual labels).

[0072] Visual modal interference intensity The visual interference parameter is the standard deviation of Gaussian blur. RGB channel offset mean The interference intensity is the weighted sum of the "relative deviations" of the two parameters: , The mean Gaussian blur parameters of all visual scenes in the interference library ( The mean, ), for standard deviation The average RGB channel offset of all visual scenes in the interference library. for standard deviation , (The weights are the proportion of feature labels in scene S1, and are entirely data-driven.) A larger value indicates that the intensity of visual interference is higher than the global average level.

[0073] Auditory modal interference intensity The auditory interference parameters are the signal-to-noise ratio (SNR) and the noise switching frequency. The logic for quantizing interference intensity is the same as that for vision: , The mean SNR of all auditory scenes in the interference library ( Since SNR is negatively correlated with interference intensity, the deviation is calculated using "mean - current value". for standard deviation The average noise switching frequency of all auditory scenes in the interference library. , for standard deviation , (The weights are derived from the proportion of feature labels in the S1 scene). A higher value indicates that the intensity of auditory interference is higher than the global average level.

[0074] Gradient boundary threshold: Rigorous calculation of slope abrupt change points, sorting pre-test data in ascending order of interference intensity to obtain the sequence. , ( for or First, calculate the slope of adjacent points. Then, based on the cohesion of the S1 sub-scene features Dynamic calculation of slope abrupt change threshold ,specific , It represents the maximum slope of all adjacent points (reflecting the maximum rate of change of the curve). The feature cohesion of the current test sub-scene. This represents the mean of the deviation rate ER in the pretest data.

[0075] The bias rate is a core metric for measuring the quality of a model's response. It is used to determine whether the current disturbance intensity needs adjustment. The formula is as follows: , This represents the number of times the text response was invalid. For the number of cross-modal processing deviations, Total number of tests required The number of tests must be greater than or equal to a predetermined number to ensure statistical validity. This predetermined number is not a generalization requirement, but a quantifiable standard derived from statistical principles and the specificities of cross-modal testing.

[0076] Based on the model bias rate and probability model results, an adjustment strategy is formulated, and the calculation is performed within the statistical baseline interval. Based on the normal distribution characteristics, the "unadjusted baseline interval" at a 95% confidence level is calculated, reflecting the average deviation rate range of the model's normal anti-interference capability under the given scenario. The historical mean effective deviation rate is also included. In this sub-scenario, the average deviation rate when the model is under normal interference resistance reflects the baseline level of the deviation rate, and the standard deviation of the historical effective deviation rate is [missing value]. The standard deviation measures the range of fluctuation in historical deviation rates; the smaller the standard deviation, the more stable the deviation rate.

[0077] During the dual-condition verification process, after adjusting the gradient, two checks are required to ensure that the interference conforms to the "weak correlation rule" and is suitable for the scenario, avoiding ineffective adjustments. Specifically, these are weak correlation strength verification and scenario interference feature matching degree verification. The weak correlation strength verification standard is: the weak correlation strength of the adjusted three-modal signals is within the range of... The purpose of setting the range is to ensure that the adjusted interference still falls within the "weak correlation" category and does not deviate from the characteristics of the real scene. The verification standard for scene interference feature matching degree is: the feature matching degree between the adjusted interference signal and the scene is greater than or equal to... Objective: To ensure that interference is strongly bound to the scene and avoid "gradient adjustment leading to a decrease in scene adaptability".

[0078] The interference gradients are categorized into G1 (weak) to G5 (strong), and the "quantized value of interference intensity" for each gradient increases with increasing gradient. Therefore, the gradient adjustment compensation coefficient is positively correlated with the gradient intensity, and its value is based on the "proportion of quantized interference intensity value" for each gradient. The overall intensity of each gradient is... The visual modality is Auditory modality is Based on the overall strength of G1 (weak interference) Based on the baseline, set according to "gradient intensity ratio" To ensure that the higher the gradient, the larger the coefficient, it is expressed as: , The overall strength of G5 (strong interference), The maximum compensation coefficient is set to 0.5, which is the highest ratio of the G5 threshold. Increased by 50%. Calculate the specific verification thresholds for each gradient and sub-scene using the core formula. If both conditions are met: determine the current gradient as the "fitting gradient" and record the model bias rate under this gradient; if either condition is not met: repeat the gradient adjustment-verification process until it is met.

[0079] Cross-modal response dual-dimensional evaluation system (corresponding to step S5) This system breaks through the limitations of traditional evaluation that "emphasizes text and neglects modality", and constructs a three-dimensional framework of "text quality + modal bias + scene adaptation". Through the weighted fusion of multiple quantitative indicators, it realizes a three-dimensional evaluation of the model's cross-modal processing capability and outputs a total score in the range of 0-1 (the closer to 1, the stronger the capability).

[0080] The text dimension assessment (measuring the ability to process distorted text) focuses on the model's ability to understand semantically fragmented and grammatically chaotic text. Four core indicators are designed (semantic understanding accuracy, logical rationality score, output validity, and scenario relevance score), and the weights are determined by the analytic hierarchy process (AHP).

[0081] Semantic understanding accuracy (SA) is defined as the percentage of correctly identified semantic units out of the total number of semantic units. Preset pass / fail criteria: Benchmark: The historical average SA value extracted from the current sub-scene (…). The lower limit of its 95% confidence interval is taken as the base value. Interference gradient compensation: For each step increase in gradient, the pass / fail standard is lowered by 2% (to avoid the standard becoming unattainable due to an excessively high gradient), but not lower than 70%.

[0082] Logical Reasonableness Score (LS), defined as: calculated based on logical contradictions and grammatical errors, using the following formula: , The number of logical contradictions. The number of syntax errors. To output the number of characters, The text character conversion factor has a value of [30, 80]. The preset pass / fail standard is: Benchmark: Extract the historical LS average of the current sub-scene (...). The 90th percentile is taken as the base value. Interference gradient compensation: For each gradient increase of one level, the acceptable standard is lowered by 2%, but not lower than 70%.

[0083] Output Validity (OE), defined as the percentage of output lengths containing input keywords. Preset acceptance criteria: Benchmark: Extract the historical average OE value for the current sub-scene (…). The 85th percentile is used as the base value. Interference gradient compensation: For each gradient increase of one level, the acceptable standard is lowered by 1.5%, but not lower than 65%.

[0084] Scene Relevance Score (SR) Definition: The proportion of scene-related keywords to the total number of keywords. Preset Pass / Fail Standard: Benchmark: Extract the historical average SR value of the current sub-scene (…). The 90th percentile is used as the base value. Interference gradient compensation: For every 1-level increase in gradient, the acceptable standard is lowered by 1.5%, but not lower than 70%.

[0085] The overall text score is ,in , The weights are determined by AHP and range from 0 to 1.

[0086] Modal dimension assessment (measuring cross-modal collaborative capability) focuses on the collaborative processing bias of text, visual and auditory aspects of the model. It designs three core indicators and fuses them into "Modal Comprehensive Bias (MMP)" through root mean square to quantify the bias.

[0087] Modal binding error rate (MBE) definition: The proportion of text-to-visual / auditory binding errors out of the total number of evaluations. Preset pass / fail criteria: Benchmark: Extract the historical average MBE value of the current sub-scene ( The 95th percentile is used as the base value. Interference gradient compensation: For each gradient increase of one level, the acceptable standard is adjusted upwards by 1.5%, but not exceeding 15%.

[0088] Semantic Pointer Confusion (SDC) is the statistical measure of semantic pointerability (SDC) based on visual and auditory modalities, using a unified basic formula. ,in To identify the number of errors in signal identification, This represents the number of semantic reverse associations. This represents the total number of evaluations for the corresponding modality.

[0089] Preset qualification standards:

[0090] Benchmark: Extract the historical auditory and visual SDC mean values ​​of the current sub-scene ( and ), taking the upper limit of its 95% confidence interval as the base value.

[0091] Interference gradient compensation: For each gradient increase of 1 level, the pass standard is increased by 1.5%, but not exceeding 15%.

[0092] Scene Interference Fit (SAF), defined as: MBE calculated based on scene-specific / general interference, using the following formula: , For scene-specific interference MBE, For general interference MBE. Preset qualification criteria: Benchmark: Extract the historical SAF mean of the current sub-scene ( The lower limit of its 90% confidence interval is taken as the base value. Interference gradient compensation: For each step increase in gradient, the pass / fail standard is lowered by 4%, but not lower than 65%.

[0093] The formula for Modal Synthesis Deviation (MMP) is: All indicators are normalized to [0,1], and the values ​​are [0,2].

[0094] The total score is calculated by assigning weights based on the modality priority principle, and the overall text score is calculated accordingly. ): The weight is the preset text weight (the core of cross-modal application is "modal collaboration", and text is only one of the inputs); Modal positive score: The weight is the preset modal weight (1 - preset text weight).

[0095] Calculation formula The value range is 0-1. The closer it is to 1, the stronger the model's cross-modal processing capability.

[0096] The temporal consistency evaluation module (corresponding to step S6) is designed for temporal scenarios that require processing continuous cross-modal input streams. This module captures the hidden shortcomings of the model in the continuous stream through temporal input generation, temporal index calculation, and visualization reports, thus solving the problem that traditional single-frame static evaluation cannot cover temporal scenarios.

[0097] Timing input generation parameters (corresponding to S6 timing input generation): The timing input stream needs to simulate continuous, dynamically changing cross-modal signals. The parameters are dynamically adjusted according to the scene's timing sensitivity (the higher the sensitivity, the more frequently the signal changes). Basic parameter settings: In the basic parameter settings for timing input generation, the parameters and setting basis for different modalities are as follows: For visual streams, the basic parameter is the frame rate; for auditory streams, the basic parameter is the sampling rate, using the audio industry standard sampling rate; for text streams, the basic parameter is to synchronize with the visual stream frame by frame, with scene timing sensitivity grading.

[0098] Based on the "signal change frequency" and "response delay requirements," the scene's temporal sensitivity is divided into three levels: high, medium, and low. Different levels correspond to different signal change parameters for visual, auditory, and text streams, as detailed below:

[0099] The criteria for determining high sensitivity levels are: signal change frequency ≥ preset high-frequency threshold, and response delay requirement ≤ preset low-latency threshold. The low-latency threshold needs to be set in conjunction with three aspects: ① Matching cross-modal signal characteristics (visual stream: standard frame rate conforming to human visual persistence; auditory stream: audio industry standard sampling rate; text stream: frame-by-frame synchronization with visual stream), ensuring it is not lower than the minimum technical delay of signal generation and output; ② Meeting the real-time target of the scenario, avoiding semantic update lag and correlation drift; ③ Relying on "high-precision clock synchronization protocol + pre-tested hardware delay compensation" to ensure that the three-modal timestamp synchronization error ≤ preset timing accuracy. The following modal parameters are set to the highest: visual stream loss true parameter switching frequency, auditory stream noise type and signal-to-noise ratio switching frequency, and text stream keyword update frequency, to adapt to high-frequency dynamic collaboration.

[0100] The criteria for determining the medium sensitivity level are: the signal change frequency is within a preset medium frequency range, and the response delay requirement is ≤ a preset medium delay threshold (set based on medium real-time requirements, balancing synchronization and resource consumption, and meeting the stability of normal continuous input timing). All modal parameters—visual loss true parameter switching frequency, auditory stream noise and signal-to-noise ratio switching frequency, and text stream keyword update frequency—are set to medium to avoid insufficient timing adaptation or resource waste.

[0101] The criteria for determining low sensitivity level are: signal change frequency < preset low-frequency threshold, and response latency requirement ≤ preset high-latency threshold (set based on low real-time scenarios, high tolerance for response speed, stable heavy signals, and adaptable to scenarios with gradual changes). All modal parameters—visual loss true parameter switching frequency, auditory stream noise and signal-to-noise ratio switching frequency, and text stream keyword update frequency—are set to the lowest possible level to reduce test resource consumption and parameter configuration complexity while meeting basic timing requirements.

[0102] Synchronization accuracy is guaranteed by using high-precision clock synchronization to ensure that the synchronization error of the three-modal timestamps is less than or equal to the preset timing accuracy requirement. Specific measures include:

[0103] 1. A high-precision clock synchronization protocol is adopted to ensure that the clock synchronization accuracy meets the timing test requirements;

[0104] 2. Pre-test and record the hardware latency of each mode, and compensate for it in advance when generating timestamps;

[0105] 3. Monitor synchronization errors in real time; if the error exceeds a preset threshold, trigger resynchronization.

[0106] The time series index is calculated, and the Association Drift Frequency (ADF) is designed to quantify the temporal stability of the model under continuous input streams. It is defined as the number of unexpected changes in the model's cross-modal association within a time period where the three-modal signals show no significant change and this period is ≥ a preset duration. This measures the stability of the model's association. The preset duration must satisfy 1 second ≤ duration ≤ 30 seconds to ensure statistical validity. Its value is determined as follows: ① A basic range is determined based on the scene's temporal sensitivity level (high sensitivity 1~3 seconds, medium sensitivity 3~5 seconds, low sensitivity 5~10 seconds); ② Fine-tuning is performed by combining the scene-specific signal stability threshold in the S1 scene-specific interference library (for high signal stability, the upper limit of the basic range is increased by ≤20%; for low signal stability, the lower limit of the basic range is decreased by ≤20%) with the model's normal inference delay (ensuring duration > inference delay); ③ The average duration of no significant signal change is referenced in the "Time Series Index Statistics" field of the current sub-scene in the S1 library. ,Pick (95% confidence interval) calibration, ultimately covering more than 90% of the real signal stability period duration in the scenario.

[0107] The basic report output (corresponding to the S6 report output) should include both static and time-series results, and use visual annotations to help developers quickly locate problems. The core content is as follows:

[0108] Static evaluation results:

[0109] Text dimension: Specific scores for each indicator and their comparison with the passing threshold;

[0110] Modal dimension: Specific values ​​of each indicator and their comparison with the qualified threshold;

[0111] Total Score: And the level division (different score ranges correspond to different levels);

[0112] Time series evaluation results:

[0113] Specific values ​​for each time-series indicator and criteria for meeting the requirements in different scenarios;

[0114] Time series bottleneck analysis: Analyzing possible technical reasons based on abnormal indicator situations;

[0115] Visual annotation:

[0116] High-deviation frames in the input stream are marked, and the marking content includes frame number, deviation type, and interference parameters;

[0117] Generate a "time series deviation curve" (horizontal axis is time, vertical axis is time series index value) to intuitively display the changes in the model's time series performance.

[0118] The interference source tracing and optimization suggestion mechanism (corresponding to S7) relies on the source tracing matrix, defect ranking, suggestion generation and feedback closed loop construction to establish a direct correlation between evaluation results, interference types, model defects and optimization actions, thereby solving the problem of poor results implementation in traditional evaluation and ultimately accelerating the model iteration process.

[0119] The three-dimensional source tracing matrix is ​​constructed (corresponding to the S6 source tracing matrix). The matrix construction method uses the Apriori association rule mining algorithm to mine the association relationship between "evaluation index - interference dimension - defect type" from historical test data (≥ the preset amount of data to ensure the reliability of association).

[0120] Defect priority ranking (corresponding to S7 defect ranking) is calculated based on the degree of compliance with indicators and the intensity of interference, with the defect impact weight (0-1 range). The higher the weight, the higher the defect priority. The formula is as follows: , The total score for the model. To measure the "performance impact" (the lower the score, the greater the performance impact, and the higher the weight). The current interference gradient (the higher the gradient, the stronger the interference, and the more significant the defect becomes under strong interference, thus the higher the weight). Weight allocation: balancing the "performance bottleneck" and the "bottleneck under strong interference".

[0121] 3D optimization suggestion generation (corresponding to S7 suggestion generation) outputs actionable 3D suggestions in "structure-parameter-data" for high-priority defects, ensuring that developers can directly implement optimizations and avoid "general talk".

[0122] Structural optimization: Structural optimization refers to addressing architectural issues at the underlying level by adjusting the model network structure, thereby improving deficiencies in cross-modal processing. Its output is a specific structural modification plan, clearly specifying the network modules to be adjusted and the new or optimized functional components, providing developers with readily implementable directions for structural adjustments.

[0123] Parameter optimization: Parameter optimization does not require modification of the model network structure. It only adjusts the model's hyperparameters to improve performance for specific tasks and compensate for existing deficiencies. Its output includes a specific parameter adjustment range and a clear optimization target. Developers can adjust the parameters according to the range and verify the optimization effect with the target in mind.

[0124] Data optimization: Data optimization involves building or expanding specific datasets to fine-tune the model, improving its adaptability to defective scenarios and enhancing model performance from a data perspective. Its output includes dataset construction standards and specific fine-tuning strategies. The construction standards specify the data volume, data type, and annotation requirements, while the fine-tuning strategies specify key elements such as the number of freeze layers, training epochs, and validation metrics.

[0125] In this invention, the ability to simulate real cross-modal non-ideal scenarios is significantly improved, achieving simultaneous distortion simulation of text-visual-auditory three modalities, accurately reproducing the weak correlation signal relationship of "appearance matching and logical conflict", greatly reducing the deviation between the test scenario and the real application environment, and significantly reducing the probability of response deviation and functional failure after the model is launched.

[0126] Based on a scene-specific interference library and dynamic gradient adjustment, the interference intensity and type are adapted to different scene characteristics, avoiding the limitations of general fixed interference and greatly improving the scene fit of non-ideal input.

[0127] The cross-modal evaluation system is more complete and more practical. It constructs a four-dimensional evaluation framework of "text-modality-scenario-temporal" to make up for the shortcomings of the traditional "emphasis on text and neglect of modality" and "emphasis on static and neglect of temporal", and comprehensively captures the performance shortcomings of the model under static input and continuous stream input.

[0128] Establish a three-dimensional traceability matrix of "evaluation index - interference type - model defect", output "structure - parameters - data" and implementable optimization suggestions, and open up a closed loop of "evaluation result → defect location → model optimization", so that the evaluation is no longer limited to score output, but directly transformed into model iteration action.

[0129] The testing and optimization efficiency is improved in both directions. The dynamic gradient adjustment mechanism accurately locates the critical threshold of the model's anti-interference ability, avoids the waste of ineffective testing resources, and improves testing efficiency compared to traditional fixed-intensity interference.

[0130] Although the invention has been described herein with reference to several illustrative embodiments, it should be understood that many other modifications and implementations can be devised by those skilled in the art, which will fall within the scope and spirit of the principles disclosed herein. More specifically, various variations and modifications can be made to the components and / or layout of the subject matter arrangement within the scope of the disclosure, drawings, and claims. Besides variations and modifications to the components and / or layout, other uses will be apparent to those skilled in the art.

Claims

1. An evaluation method for cross-modal input generation based on weak association rules and dynamic gradients, characterized in that, Includes the following steps: S1. Contextualized interference library construction and invocation: This involves building a scenario-specific interference library with a three-tiered cross-modal scenario interference library architecture to provide accurate scenario-based information for input generation. S2. Contextualized trimodal non-ideal input generation: Based on the scenario adaptation results of S1, trimodal non-ideal input is generated through a process of basic resource loading, submodal distortion generation, and temporal alignment binding. S3. Weak association rule construction and binding: Based on a feature extraction, weighted mapping, and intensity control architecture, weak association binding is achieved between text and cross-modal signal appearance matching and logical conflicts, stabilizing the association strength within a preset range. S4. Dynamic gradient adjustment: A gradient grading, deviation calculation, closed-loop adjustment, and dual-condition verification mechanism is constructed to dynamically optimize interference strength based on model response and accurately locate the anti-interference critical threshold. S5. Cross-modal response dual-dimensional evaluation: A three-dimensional evaluation framework is constructed, covering text... The evaluation process involves three levels: dimension, modality, and scene adaptation. Model performance is assessed using multiple quantitative indicators, and a total score is output by weighted fusion. S6 involves temporal consistency verification and result output, specifically addressing temporal scenario requirements by constructing a module for temporal input generation, temporal indicator calculation, and visual report output to capture the model's hidden weaknesses under continuous input. S7 involves an interference tracing and optimization suggestion mechanism, constructing a tracing matrix, defect ranking, suggestion generation, and feedback closed-loop mechanism to establish a direct correlation between evaluation results and optimization actions, achieving an evaluation-optimization closed loop. In step S5, the cross-modal response dual-dimensional evaluation includes text dimension evaluation, modality dimension evaluation, and total score calculation. Modality dimension evaluation includes modality binding error rate, semantic confusion, scene interference adaptation, and modality comprehensive deviation. The semantic confusion is statistically analyzed by visual and auditory modalities, using a unified basic formula: ,in To identify the number of errors in signal identification, This represents the number of semantic reverse associations. The total number of evaluations corresponds to the modality; the total score is calculated with weights assigned according to the modality priority principle, and the calculation formula is as follows: .

2. The evaluation method for cross-modal input generation based on weak association rules and dynamic gradients according to claim 1, characterized in that: Steps S1 and S2 include core logic for scene adaptation, multimodal parameterization generation, and temporal alignment assurance. The core logic for scene adaptation includes feature vector construction, scene similarity calculation, and scene similarity threshold calculation. The scene similarity threshold calculation includes sub-scene feature cohesion. Sub-scene historical effective similarity statistics Scene accuracy requirement coefficient And scene similarity preset threshold, The feature vector construction involves unifying the feature vector dimensions of the input scene and the sub-scenes in the interference library. The specific structure is as follows: Scene function encoding: describes the core function of the scene, represented using one-hot encoding. ,in Scene function encoding vector, dimension M; environmental feature values: including environmental parameters such as temperature, light intensity, and noise decibels, represented by normalized continuous values. ,in Environmental parameters The original measurement value, For parameters The minimum value in historical scenario data. For parameters The maximum value in historical scenario data. The normalized parameter values ​​range from [0,1]. The P normalized environmental parameters are arranged in a fixed order to form an environmental feature vector. The interaction method parameter describes the form of interaction between the user and the model, represented by binary encoding. , This is a vector of interaction method parameters, with dimension Q; Interference source characteristics: Describes the sources and attributes of typical interference in the scene, represented by multi-label encoding: , The feature vector of the interference source has dimension R; All feature vectors are adopted Normalization is performed to eliminate the interference of different feature magnitudes on similarity calculation and ensure matching fairness. The above vectors are concatenated in a fixed order of scene function encoding, environmental feature values, interaction mode parameters, and interference source features to obtain the total feature vector of the input scene. Its dimension D = M + P + Q + R, and its mathematical expression is: , The scene similarity calculation includes using a cosine similarity metric to quantify the matching degree between the input scene and sub-scenes in the library, as shown in the following formula: Multimodal parameterized generation includes visual signal distortion generation, auditory signal distortion generation, and text signal distortion generation. Visual signal distortion generation employs a layered overlay strategy of basic distortion plus scene-specific distortion to simulate visual interference in different scenarios. This includes basic distortion 1: Gaussian blur, with the formula: Basic Distortion 2: RGB channel offset, the formula is: Scene-specific distortion: designed for the characteristics of sub-scenes; Auditory signal distortion generation includes using a time-frequency domain joint distortion model to simulate the non-stationary characteristics of noise in real-world scenes. Specifically, this includes the first step: Short-Time Fourier Transform (STFT); performing STFT on the original auditory signal to convert the time-domain signal into a time-frequency representation. The second step: generating the noisy signal, the formula is as follows. Step 3: Noise mixing model. Scene noise is dynamically mixed from three types of noise. Formula: Step 4: Global Signal-to-Noise Ratio Constraints. To ensure controllable noise intensity, the global signal-to-noise ratio must meet the following requirements: ,in The power of the effective signal, For the power of mixed noise; timing alignment guarantees include timestamp binding and dynamic delay compensation. Timestamp binding assigns a unique timestamp to each frame / segment of each modal signal, satisfying: Dynamic delay compensation is necessary because, due to differences in hardware output delay, the delay values ​​of each mode need to be recorded through pre-testing and compensated in advance when allocating timestamps.

3. The evaluation method for cross-modal input generation based on weak association rules and dynamic gradients according to claim 1, characterized in that: Step S3 includes feature extraction and weighted mapping, and calculation of weak correlation strength. Feature extraction and weighted mapping include cross-modal feature extraction, multi-head attention weighted mapping, and optimized feature vectors. Specifically, multi-head attention weighted mapping replaces traditional linear weighting with a multi-head attention mechanism. Through multi-subspace interaction and scene feature fusion, it improves the precision of surface feature matching. The formula is as follows: A single attention head is calculated as , These represent the query vector (text features), the key vector (cross-modal surface features), and the value vector (target features to be fused), respectively. The dimensions are batch size, sequence length, and feature dimension for the adaptation model. No. The learnable projective weights for each attention head, with dimensions that are the fit values ​​between the feature dimension and the subspace dimension, serve to... Projecting onto a subspace enables multi-subspace matching. The dimension of the key vector. This is used to alleviate the problem of vanishing softmax gradients caused by excessively large inner products. This is the scene correlation coefficient, and its value range conforms to the setting of the scene's dependence on surface feature matching. The more the scene depends on surface feature matching, the higher the correlation coefficient becomes. The larger, No. Feature vectors of each sub-scene Yes The multilayer perceptron transformation is used to incorporate scene information into the attention score calculation. The normalization function maps the attention scores to the [0,1] interval, ensuring that the sum of the weights is 1. For the concatenation function, The outputs of each attention head are concatenated to form the original feature dimension. The weights are the linear transformation weights after concatenation, with the same dimension as the original features. The weak association strength is calculated by balancing surface similarity and logical conflict through a dynamic α coefficient, ensuring that the association strength remains stable at a certain level. The formula is as follows: The range of weak correlation strength is set to the parameters defined for real-world sample data, i.e., for the first... Real-world scenario samples Define its inherent parameters; Calculating the statistical distribution characteristics of a sample set is... one sample Calculate the core statistics that reflect the distribution patterns of real-world scenarios; derive the upper and lower limits of preset intervals, specifically through interval derivation based on quantiles or confidence interval derivation based on normal distribution.

4. The evaluation method for cross-modal input generation based on weak association rules and dynamic gradients according to claim 1, characterized in that: In step S4, a closed-loop mechanism for dynamic gradient generation is implemented, including disturbance gradient classification, deviation rate calculation, gradient adjustment decision, and verification. Specifically, the deviation rate in the deviation rate calculation is a core indicator for measuring the quality of the model response and is used to determine whether the current disturbance intensity needs adjustment. The formula is as follows: , This represents the number of times the text response was invalid. For the number of cross-modal processing deviations, Total number of tests.

5. The evaluation method for cross-modal input generation based on weak association rules and dynamic gradients according to claim 1, characterized in that: In step S6, the timing consistency verification and result output includes timing input generation parameters, timing index calculation, and basic report output. The timing index calculation consists of three core indicators designed to quantify the timing stability of the model under continuous input streams. The three core indicators include cross-frame correlation contradiction rate, semantic update delay time, and correlation drift frequency. The basic report output includes both static and timing results.

6. The evaluation method for cross-modal input generation based on weak association rules and dynamic gradients according to claim 1, characterized in that: In step S7, the interference tracing and optimization suggestion mechanism includes the construction of a three-dimensional tracing matrix, defect priority ranking, and the generation of three-dimensional optimization suggestions. The defect priority ranking includes the calculation of defect impact weights, which are calculated based on three dimensions: indicator compliance, interference intensity, and impact range (in the range of 0-1). Higher weights indicate higher defect priority, as shown in the following formula. , This is the model's total score. To measure the performance impact, This represents the current interference gradient.

Images