Multi-modal feature evaluation decision-making system based on attention mechanism

Abstract

The invention discloses a multi-modal feature evaluation decision-making system based on an attention mechanism. The system comprises a data preprocessing module, a feature coding module, a preliminary decision-making module, an attention self-correction module and a final decision-making module. The data preprocessing module is used for acquiring multi-modal to-be-evaluated data and carrying out standardization processing on the multi-modal to-be-evaluated data to form a standardized multi-modal feature set; and the feature coding module is used for performing feature extraction on the standardized multi-modal feature set and generating initial feature representation of each modal. The invention relates to the technical field of artificial intelligence. According to the multi-modal feature evaluation decision-making system based on the attention mechanism, a feedback mechanism is introduced through the attention self-correction module, and secondary adjustment is performed on the weight of the modal features according to a preliminary decision-making result, so that self-adaptive correction of an abnormal modal is realized, the interference of the abnormal modal on the decision-making result is effectively reduced, and the accuracy of the decision-making result is improved. And the consistency and the stability of the final decision are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, more particularly, to a multi-modal feature evaluation decision system based on attention mechanism. BACKGROUND

[0002] With the development of artificial intelligence technology, especially the popularization of deep learning technology, the processing of multi-modal data has become a research hotspot. Multi-modal data covers multiple different types of information sources, such as text, images, audio, numerical data, etc. Different modal data has unique structure and characteristics, how to efficiently fuse these heterogeneous data, especially through deep learning method for comprehensive evaluation and decision, has become the key problem of current multi-modal learning.

[0003] Existing multi-modal learning methods usually adopt simple weighted fusion or feature extraction and fusion strategy based on deep neural network, but these methods have some problems when facing complex multi-modal data. For example, the traditional method may not be able to effectively handle the uneven weight problem between different modalities, resulting in the influence of some modal data being underestimated or overestimated; at the same time, the processing of abnormal modal data is not fine enough, which may affect the accuracy of the final decision result.

[0004] Therefore, a more accurate and intelligent multi-modal feature evaluation decision system is needed, which can allocate appropriate weights according to the data characteristics of different modalities and has self-adaptive correction function to improve the consistency and stability of decision. SUMMARY

[0005] The purpose of the present application is to provide a multi-modal feature evaluation decision system based on attention mechanism, which solves the problem that the existing multi-modal learning method usually adopts simple weighted fusion or feature extraction and fusion strategy based on deep neural network, cannot effectively handle the uneven weight problem between different modalities, resulting in the influence of some modal data being underestimated or overestimated, the processing of abnormal modal data is not fine enough, which may affect the accuracy of the final decision result, and cannot meet the use requirement.

[0006] The present application realizes the above-mentioned purpose through the following technical scheme: a multi-modal feature evaluation decision system based on attention mechanism, the system comprises:

[0007] data preprocessing module, feature encoding module, preliminary decision module, attention self-correction module and final decision module;

[0008] The data preprocessing module is used for obtaining multi-modal data to be evaluated and performing standardization processing to form a standardized multi-modal feature set;

[0009] The feature encoding module is configured to perform feature extraction on the standardized multi-modal feature set to generate initial feature representations of each modality.

[0010] The preliminary decision module utilizes a basic attention network to realize weight distribution and fusion of features of each modality, and outputs a preliminary evaluation decision result.

[0011] The attention self-correction module constructs a feedback self-correction mechanism, and performs secondary adjustment on the weight of the modality feature based on the preliminary decision result, to realize adaptive correction of the abnormal modality.

[0012] The final decision module outputs a final evaluation decision result based on the secondary adjusted fusion features, to improve the consistency and stability of the decision.

[0013] Further, the data preprocessing module comprises:

[0014] a data acquisition unit, a standardization processing unit, a feature set integration unit, and a training data optimization unit.

[0015] The data acquisition unit is configured to collect multi-modal raw data containing multiple data types, including at least two of text, image, numerical value, and audio.

[0016] The standardization processing unit maps the data of each modality to a similar numerical interval by eliminating the differences in dimension and numerical range between different modalities.

[0017] The feature set integration unit is configured to integrate the standardized data of each modality to form a standardized multi-modal feature set with unified structure.

[0018] The training data optimization unit is configured to realize division of data sets and data enhancement processing. The data sets are divided into a training set, a validation set, and a test set according to a predetermined proportion, and the data volume is expanded by using corresponding enhancement strategies for different modalities.

[0019] Further, the feature encoding module comprises a plurality of dedicated modality encoders, which are designed according to the data structure and feature expression characteristics of the corresponding modality, including:

[0020] at least one of a Transformer-based encoder suitable for text data, a convolutional neural network encoder for image data, a multi-layer perceptron encoder adapted for numerical data, and a mel-frequency cepstral coefficient encoder for audio data;

[0021] Each dedicated modality encoder performs convolution operation, full connection mapping, and activation function transformation to deeply mine high-dimensional semantic features and potential associated information of the modality data, and outputs initial feature representations of a unified dimension to provide a basis for cross-modal feature fusion.

[0022] Further, the preliminary decision module comprises:

[0023] a basic attention network unit and a preliminary decision output unit;

[0024] The basic attention network unit adopts a multi-head attention mechanism to calculate the correlation weight between the initial features of each modality by constructing a query matrix, a key matrix, and a value matrix. The weight distribution is constrained by the modality feature variance and the target correlation to ensure that important modalities obtain higher weights.

[0025] The preliminary decision output unit is composed of a multi-layer fully connected network, which maps the weighted and fused preliminary fusion features to the decision space, and outputs the preliminary evaluation decision result according to the evaluation task type, which includes classification tasks and regression tasks.

[0026] Further, the attention self-correction module comprises:

[0027] a feedback signal conversion unit, a self-correction attention network unit, a feature correction unit, and a secondary fusion unit;

[0028] The feedback signal conversion unit converts the preliminary evaluation decision result into a feedback feature vector consistent with the modality feature dimension through a nonlinear conversion network;

[0029] The self-correction attention network unit adds a feedback information interaction channel based on the basic attention network structure, and calculates the secondary correction weight by splicing the modality initial feature and the feedback feature vector;

[0030] The feature correction unit adaptively adjusts each modality initial feature based on the secondary correction weight to reduce abnormal modality interference;

[0031] The secondary fusion unit performs weighted fusion on the corrected modality features to generate secondary fusion features.

[0032] Further, the secondary correction weight calculation of the self-correction attention network unit introduces a decision confidence coefficient;

[0033] The decision confidence coefficient is determined based on the probability distribution of the preliminary decision result, and different weight adjustment amplitude constraints are set according to the confidence level;

[0034] When the confidence is higher than a first threshold, the weight is limited to small adjustment;

[0035] When the confidence is between the first threshold and a second threshold, moderate weight adjustment is allowed;

[0036] When the confidence is lower than the second threshold, large weight reconstruction is supported, and the first threshold is greater than the second threshold.

[0037] Further, the feature correction unit comprises:

[0038] The abnormal mode determination subunit and the feature adjustment subunit;

[0039] The abnormal mode determination subunit determines the abnormal mode through the double conditions of mode weight size and feature similarity;

[0040] The feature adjustment subunit realizes adaptive adjustment of modal features based on secondary correction weight and feature correction increment. The correction strength coefficient is negatively correlated with the decision confidence, and the value range is constrained in a preset interval to avoid feature distortion caused by excessive correction.

[0041] Further, the final decision module comprises:

[0042] The consistency checking unit, the decision optimization unit and the final result output unit;

[0043] The consistency checking unit calculates the similarity of the secondary fusion features and the preliminary fusion features, and compares it with the preset consistency threshold;

[0044] The decision optimization unit selects the feature input mode according to the similarity comparison result. When the similarity meets the threshold requirement, the secondary fusion features are directly input, otherwise the input features are adjusted through iterative optimization or weighted fusion strategy;

[0045] The final result output unit adopts a deep fully connected network structure to convert the optimized fusion features into a final evaluation decision result, meeting the decision consistency constraint requirement.

[0046] Further, the preset consistency threshold of the consistency checking unit is adaptively set based on the feature similarity distribution of the training set, determined by calculating the mean and standard deviation of the similarity of the training set samples, and the threshold is limited in a preset interval;

[0047] The decision consistency constraint sets corresponding constraint standards for different task types. In the regression task, the constraint error is matched with the target value range, and in the classification task, the adjacent decision results are required to be consistent in class.

[0048] Further, the system modules adopt an end-to-end training method, and the optimizer, loss function and training scheduling strategy are uniformly configured;

[0049] The optimizer adopts AdamW optimizer, and each module sets corresponding learning rate and weight decay coefficient according to functional requirements;

[0050] The loss function is selected according to the task type and module function, including cross entropy loss function, mean square error loss function, self-correction loss function and mixed loss function.

[0051] The training scheduling strategy includes early stopping strategy, cosine annealing learning rate scheduling and gradient clipping strategy, and ensures stable convergence of the model.

[0052] The beneficial effects of the present application are:

[0053] 1. By using a multi-modal encoder and an attention mechanism, the features from different modalities are effectively extracted and fused, realizing deep fusion of cross-modal features, and being able to handle structural differences of different modal data, so that the decision system is more robust and adaptable when facing complex data.

[0054] 2. The system introduces a feedback mechanism through the attention self-correction module, and adjusts the weights of the modal features again according to the preliminary decision result, thereby realizing adaptive correction of abnormal modalities, effectively reducing the interference of abnormal modalities on the decision result, and improving the consistency and stability of the final decision.

[0055] 3. In the attention self-correction module, the decision confidence coefficient introduces an adaptive adjustment mechanism, which flexibly adjusts the adjustment amplitude of the weight according to the confidence level of the preliminary decision, effectively avoiding excessive adjustment in the case of low confidence, thereby improving the adaptability of the system to different tasks.

[0056] 4. The system adopts an end-to-end training method, combines the AdamW optimizer, a hybrid loss function and a training scheduling strategy, ensures that the model can stably converge and achieve efficient training effect, and at the same time, improves the quality of the data set and the generalization ability of the model through data enhancement and optimization strategies.

[0057] 5. Each module of the system has a clear division of functions, and a modular structure is adopted in the design, which can be easily expanded and maintained, and the optimization and adjustment of each module will not affect the operation of other modules, enhancing the flexibility and scalability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0058] The accompanying drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0059] Figure 1 is a flow chart of the overall system of the present application;

[0060] Figure 2 is a flow chart of the data preprocessing module of the present application;

[0061] Figure 3 is a flow chart of the attention self-correction module of the present application;

[0062] Figure 4 Flowchart of the final decision module of the present application. DETAILED DESCRIPTION

[0063] The following further describes the present application in conjunction with the accompanying drawings, and it is necessary to point out here that the following detailed description is only used to further illustrate the present application and cannot be understood as limiting the scope of protection of the present application, and those skilled in the art can make some non-essential improvements and adjustments to the present application according to the above application content.

[0064] Example 1:

[0065] Please refer to Figures 1-4 The present application provides a technical solution: a multi-modal feature evaluation decision system based on attention mechanism, the system comprises:

[0066] a data preprocessing module, a feature encoding module, a preliminary decision module, an attention self-correction module and a final decision module;

[0067] The data preprocessing module is used to obtain multi-modal data to be evaluated, and each modality data is preprocessed respectively to obtain a standardized multi-modal feature set.

[0068] Among them, the multi-modal data to be evaluated refers to a data set containing multiple different types of data forms, such as image, text, voice and other different modalities of data, which are the original input of the system for evaluation and decision; preprocessing is a series of operations on each modality data, the purpose is to convert the original data into a form suitable for subsequent processing, which may include data cleaning, normalization, standardization and other operations; the standardized multi-modal feature set is converted into a unified standard form after preprocessing, forming a data set containing multiple modal features;

[0069] The feature encoding module is used to extract features from the standardized multi-modal feature set to generate initial feature representations of each modality.

[0070] Among them, feature extraction is to extract key feature information of each modality data from the standardized multi-modal feature set by using a specific algorithm or model, and to convert the original data into a more concise and more representative feature representation form; the initial feature representation is the preliminary feature form obtained by each modality data after feature extraction, which preliminarily reflects the key information of each modality data, but has not been fused and further processed across modalities;

[0071] The preliminary decision module is used to assign weights to the initial feature representations of each modality by using a basic attention network and to fuse the features to obtain preliminary fusion features, and to output a preliminary evaluation decision result based on the preliminary fusion features.

[0072] The basic attention network is a neural network structure for processing attention mechanism, which can automatically assign different weights according to the importance of input features, highlight key features and suppress secondary features; the weight distribution is a weight value assigned to each modality feature by the basic attention network according to the importance of the initial feature representation of each modality, and the weight size reflects the relative importance of the modality feature in the overall decision; the feature fusion is to integrate the initial feature representation of each modality after weight distribution in a certain way to form a comprehensive preliminary fusion feature, which fuses the key information of different modalities; the preliminary evaluation decision result is the preliminary evaluation and decision information obtained based on the preliminary fusion feature through a specific algorithm or model, which provides a basis for subsequent further optimization decision;

[0073] The attention self-correction module is used to construct an attention feedback type self-correction mechanism, and the preliminary evaluation decision result is used as a reverse feedback signal to guide the secondary adjustment of the distribution weight of each modality feature, so as to realize the self-adaptive correction of abnormal modality input without manual intervention.

[0074] The attention feedback type self-correction mechanism is a method of automatically adjusting attention weight through feedback mechanism, which uses the preliminary evaluation decision result as a feedback signal to guide the system to re-examine the distribution weight of each modality feature, so as to realize the self-adaptive adjustment of abnormal modality input; the reverse feedback signal is the preliminary evaluation decision result transmitted back to the system as the basis for adjusting the weight of each modality feature, which helps the system to find possible abnormal modality input or unreasonable weight distribution; the secondary adjustment is to adjust the distribution weight of each modality feature again according to the reverse feedback signal, so that the system can automatically adapt to different situations without manual intervention, especially to correct abnormal modality input; the abnormal modality input refers to the situation that the data of some modalities in the multi-modal data may have quality problems, such as too much noise, data missing, data error, etc., or is irrelevant to the overall decision, which may interfere with the normal decision of the system.

[0075] The final decision module is used to output the final evaluation decision result based on the fusion feature after secondary adjustment, to enhance the consistency and stability of decision;

[0076] The fusion feature after secondary adjustment is the feature obtained by re-fusing the weight of each modality feature after secondary adjustment by the attention self-correction module, which can more accurately reflect the contribution of each modality data to the final decision than the preliminary fusion feature; the final evaluation decision result is the evaluation and decision information obtained based on the fusion feature after secondary adjustment through the final decision algorithm or model of the system, which is more consistent and stable than the preliminary evaluation decision result.

[0077] It should be noted that, during use, the data preprocessing module standardizes the data across all modalities, laying the foundation for subsequent processing and improving data quality and usability. The feature encoding module extracts initial feature representations, accurately capturing key information from each modality. The preliminary decision-making module uses a basic attention network to allocate weights and fuse features, initially integrating multimodal information and quickly outputting preliminary results, improving decision-making efficiency. The attention self-correction module constructs a feedback mechanism, using the preliminary results as signals to adjust the weights a second time, adaptively correcting abnormal modal inputs without manual intervention, enhancing system robustness. The final decision-making module outputs results based on the fused features after secondary adjustment, further improving decision consistency and stability. The overall design fully utilizes the advantages of multimodal data, achieving efficient, accurate, and adaptive evaluation and decision-making through attention and self-correction mechanisms, making it suitable for complex and ever-changing real-world application scenarios.

[0078] In one embodiment, the data preprocessing module performs the specific process of acquiring multimodal data to be evaluated and preprocessing each modality of data separately, including:

[0079] Acquire multimodal evaluation data, which includes... Different modalities, including common data types such as text, images, numbers, and audio, are denoted as... ,in Indicates the first The original data of the modality, Each modality of data has its own independent data source and data structure characteristics;

[0080] The raw data for each modality are standardized separately. The core purpose is to eliminate the imbalance in feature distribution caused by differences in units and numerical ranges between different modalities. The standardization expression is as follows:

[0081]

[0082] in, Indicates the first The first mode One data sample, Indicates the first The mean of all samples in a given modality. Indicates the first The standard deviation of all samples in the modality For the first The number of samples for each modality. For the standardized first The first mode After standardization, each sample data will be mapped to a similar numerical range, laying the foundation for subsequent cross-modal feature fusion.

[0083] Integrating all modal standardized sample data, forming a standard multi-modal feature set with unified structure and directly used for feature encoding:

[0084]

[0085] wherein, is the standardized feature subset of the th modality, which fully retains the original semantic information of each modality while achieving data compatibility between modalities;

[0086] The preprocessing supplementary steps in the model training stage: the data set is divided into training set, validation set and test set in the ratio of 7:2:1, the training set is used for model parameter iterative update, the validation set is used for hyperparameter tuning and overfitting monitoring, and the test set is used for final model performance evaluation; Data augmentation strategy is used for the training set, synonym replacement and sentence rearrangement are used for text modal to generate expanded samples, random cropping, rotation and flipping are used for image modal to expand data volume, and small Gaussian noise is added to generate incremental samples (Gaussian noise standard deviation is 0.01) for numerical modal, and volume adjustment and time stretching are used for audio modal to realize data augmentation, ensuring the diversity of training data and the generalization ability of the model.

[0087] In this way, the multi-modal data is standardized and divided into data sets, and the training set is also augmented. The dimensions and ranges of different modal data are greatly different, and the standardization process can eliminate this imbalance, map the data to a similar interval, lay a foundation for subsequent cross-modal fusion, and divide the data set for model training, tuning and evaluation. The division of labor in each link is clear, the data augmentation strategy is targeted at the characteristics of different modalities, expands the data volume, increases the data diversity, improves the model generalization ability, makes it better to deal with various complex situations, reduces the risk of overfitting, and improves the accuracy and stability in practical application.

[0088] In an embodiment, the feature encoding module performs feature extraction on the standardized multi-modal feature set to generate the specific process of generating the initial feature representation of each modality, including:

[0089] For the unique data structure and feature expression characteristics of each modality, a dedicated modality encoder is constructed, which includes a Transformer-based text modality encoder suitable for text data, a convolutional neural network image modality encoder for image data, a multi-layer perceptron numerical modality encoder adapted to numerical data, and a Mel frequency cepstrum coefficient audio modality encoder for audio data. Each encoder is optimized in structure to match the feature extraction requirements of the corresponding modality.

[0090] The structure parameters of each dedicated modality encoder are:

[0091] The Transformer-based text encoder uses a 6-layer encoder structure, with 8 multi-head attention heads, 512 hidden layers, 2048 feed-forward network dimensions, and a dropout rate of 0.1.

[0092] The convolutional neural network image encoder uses 5 convolutional layers and 2 max pooling layers. The convolutional kernel sizes are 3×3, 3×3, 5×5, 5×5, and 3×3, and the number of convolutional kernels are 64, 128, 256, 256, and 512, respectively. The pooling kernel size is 2×2, the stride is 2, and the dropout rate is set to 0.2.

[0093] The multilayer perceptron numerical encoder contains three fully connected layers with 256, 512, and 512 neurons respectively, and uses GELU as the activation function.

[0094] The Mel frequency cepstral coefficient audio encoder first extracts 40-dimensional Mel frequency cepstral coefficient features, and then encodes the features through two convolutional layers and two fully connected layers. The convolutional kernel size is 3×3, with 64 and 128 kernels respectively, and the number of neurons in the fully connected layers is 256 and 512.

[0095] Subsets of each modality feature from the standardized multimodal feature set Each modality is input into its corresponding dedicated encoder. Through a series of processing steps, including convolutional operations to extract local key features, dimensionality transformation and feature mapping in fully connected layers, and the introduction of non-linear expressive capabilities using activation functions such as ReLU and GELU, the high-dimensional semantic features and potential correlations inherent in each modality's data are deeply mined, ultimately generating initial feature representations for each modality.

[0096]

[0097] in, Indicates the first A dedicated encoder for each modality,

[0098]

[0099] , for the first Feature dimensions of a modality To unify the encoding dimension, the value range is 64-1024, which is adaptively determined based on the complexity of the evaluation task and the data scale. The larger the data volume and the more complex the task, the higher the dimension value. The unified dimension design ensures that different modal features can be used for subsequent cross-modal attention calculation and fusion operations.

[0100] Training parameters for the encoding module during model training: The AdamW optimizer is used, with an initial learning rate of 1e-4 and a weight decay coefficient of 1e-5. The pre-trained weights of the text encoder are initialized using the weights of the BERT-base model pre-trained on a large-scale general corpus (including Wikipedia + BookCorpus), while the weights of the other encoders are initialized using the Xavier normal distribution. During training, the parameters of the first 3 layers of the text encoder are frozen, the last 3 layers are fine-tuned, and all parameters of the other encoders are used for training to balance feature extraction capability and training efficiency.

[0101] This design involves building dedicated encoders for different modalities and setting detailed structural parameters. Since the data structures and feature representations of different modalities vary greatly, the dedicated encoders can extract features specifically to match the needs of each modality. The detailed structural parameter settings have been optimized to deeply mine the high-dimensional semantics and potential correlation information of each modality. The unified encoding dimension facilitates subsequent cross-modal computation and fusion. During model training, different initialization methods and parameter training strategies balance feature extraction capability and training efficiency, which helps the model converge quickly, improves the accuracy and effectiveness of feature extraction, and enhances model performance.

[0102] In one embodiment, the preliminary decision-making module utilizes a basic attention network to perform weight allocation and feature fusion on the initial feature representations of each modality, and outputs preliminary evaluation decision results, including:

[0103] Construct a basic attention network and employ a multi-head attention mechanism to fully capture the complex relationships between different modalities. The number of attention heads... The determination rule is: based on the unified coding dimension. Adaptive allocation, satisfying ,and This ensures that the feature dimensions of each attention head are balanced, thereby improving the effectiveness of capturing associated features;

[0104] The basic attention network's structural parameters include two multi-head attention layers and two feed-forward networks. The hidden layer dimension of the feed-forward network is [missing information]. The activation function used is GELU, and the dropout rate is set to 0.1.

[0105] By computing the feature interaction information of multiple attention heads in parallel, the association weights between the initial feature representations of each modality are generated. The weight calculation expression is as follows:

[0106]

[0107] in,

[0108]

[0109]

[0110]

[0111] Query matrix, key matrix and value matrix respectively, , , is a learnable parameter matrix, initialized as an orthogonal matrix, constantly optimized through model training to adapt to multi-modal feature correlation learning, is the dimension of a single attention head, introduces for scaling to avoid the problem of gradient disappearance of the Softmax function caused by the result of matrix multiplication being too large, is a normalization function to ensure that the sum of the weights of each modality is 1;

[0112] Based on the calculated correlation weight, the initial feature representation of each modality is weighted and fused, and the size of the weight value directly reflects the importance of the corresponding modality to the evaluation decision task. The supplementary rule of weight allocation is:

[0113] Through the modality contribution degree evaluation function:

[0114]

[0115] is constrained, where is the variance of the th modality feature, is the similarity between the th modality feature and the true decision target feature , to ensure that modalities with large variance and strong correlation with the target obtain higher initial weights, and obtain preliminary fusion features that can integrate key information from each modality , where is the attention weight of the th modality, satisfying , this fusion process realizes the effective integration of cross-modal information and highlights the contribution of important modality features;

[0116] The preliminary fusion feature is input into the decision prediction layer, which is composed of multiple fully connected networks, and the structure parameters are:

[0117] contains 2 fully connected layers and 1 output layer, and the number of neurons in the fully connected layer is , , the activation function uses GELU, and the dropout rate is set to 0.2; through the nonlinear mapping ability of the network, the fusion feature is converted to a specific decision space, and the preliminary evaluation decision result is output according to the type of the evaluation task:

[0118]

[0119] The evaluation tasks include regression tasks and classification tasks. This indicates the initial decision decoder. The initial decision results, whether continuous values ​​(suitable for regression tasks) or discrete category labels (suitable for classification tasks), will serve as the core feedback basis for subsequent self-correction mechanisms.

[0120] Preliminary training parameters for the decision module during model training: The same AdamW optimizer as the encoding module is used, with an initial learning rate of 5e-5 and a weight decay coefficient of 1e-5; the loss function is selected based on the task type: for classification tasks, the cross-entropy loss function is used.

[0121]

[0122] in, For the first The true label of each sample The class probability output for the initial decision;

[0123] The regression task uses the mean squared error loss function:

[0124]

[0125] in, For the first The true value of each sample The predicted values ​​output for the initial decision.

[0126] This design constructs a basic attention network, specifying its structure and parameters in detail, and providing rules for weight calculation and allocation. The multi-head attention mechanism can fully capture complex associations between different modalities, improving the effectiveness of capturing association features. Reasonable structural parameter settings ensure network performance. The weight calculation and allocation rules comprehensively consider the variance of modal features and similarity to the target, ensuring that important modalities receive higher weights and highlighting their contributions. Selecting a loss function based on the task type can more accurately measure the gap between the model output and the true value, guiding model optimization and making the initial decision results more accurate and reliable, providing a good foundation for subsequent self-correction.

[0127] In one embodiment, the specific process by which the attention self-correction module constructs an attention feedback self-correction mechanism and implements adaptive correction of abnormal modal features includes:

[0128] An attention-feedback self-correction mechanism is constructed. This mechanism is a core innovative module for evaluating the stability of decision results. It consists of three parts: a feedback signal conversion unit, a self-correcting attention network, and a feature correction unit. These units work together to achieve a closed-loop feedback from decision results to feature adjustment. The feedback signal conversion unit adopts a 3-layer MLP structure with the following number of neurons: , , The activation function is GELU, and the dropout rate is set to 0.1.

[0129] The preliminary evaluation decision results are converted through the feedback signal conversion unit. Converted into a feedback feature vector that can directly interact with modal features:

[0130]

[0131] in, As a nonlinear transformation function, it is implemented using the above-mentioned MLP network. Its core function is to map the abstract decision result into a feedback vector consistent with the modal feature dimension, ensuring that the feedback information can be effectively transmitted to the modal feature adjustment process. The feedback vector contains key information such as the confidence level and deviation direction of the preliminary decision result.

[0132] A self-correcting attention network is constructed, which adds a feedback information interaction channel to the structure of the basic attention network, while maintaining the same structural parameters as the basic attention network (including the number of attention heads). (Feed-Forward network dimensions, etc.), only adding feedback feature concatenation operations at the input layer; representing the initial features of each modality. With feedback feature vector By splicing the data, we obtain the feedback enhancement features that integrate the feedback information:

[0133]

[0134] By concatenating features, the effects of the initial decision are perceived, and then secondary attention weights are calculated. The calculation of these secondary weights fully considers the impact of decision feedback information on modal importance; the expression is as follows:

[0135]

[0136] in, These are bias parameters, initially set to 0, and optimized iteratively through model training. As an activation function, the weight values ​​are constrained to the range of 0-1, which facilitates precise control over the adjustment range of modal features. For the first The secondary correction weights for each mode;

[0137] The supplementary determination rule of the secondary correction weight is:

[0138] Introducing the decision confidence coefficient , calculated by the maximum value of the Softmax probability distribution of the preliminary decision decoder output layer,

[0139]

[0140] The original logical value of the decoder output;

[0141] When , the difference degree of the secondary correction weight and the initial attention weight is constrained to , that is, a small weight adjustment under a high confidence decision;

[0142] When , the difference degree is relaxed to , allowing moderate weight adjustment;

[0143] When , the difference degree has no upper limit constraint, supporting a large weight reconstruction to correct possible decision bias, and the rule ensures that the weight adjustment matches the decision confidence, avoiding invalid adjustment;

[0144] Based on the secondary correction weight, the initial feature representation of each modality is adaptively adjusted. For the abnormal modality reflected in the preliminary decision, the abnormal modality determination rule is:

[0145] When the modality weight or and , it is determined as an abnormal modality;

[0146] Through the correction mechanism, the interference of the abnormal modality is automatically reduced or its effective information is enhanced, realizing the adaptive correction of the abnormal modality feature. The corrected modality feature representation is:

[0147]

[0148] Wherein, is the feature correction increment, is the correction strength coefficient, and its determination rule is: That is, the lower the decision confidence, the higher the correction strength, and the value range is strictly constrained between 0-1 to avoid feature distortion caused by excessive correction. The correction process does not require human intervention and is completely completed by the model according to the feedback signal;

[0149] Fuse all the corrected modality feature representations to obtain the secondary fusion feature that fuses the decision feedback information and eliminates the interference of the abnormal modality:

[0150]

[0151] This feature has higher reliability and effectiveness compared to the initial fusion feature;

[0152] The self-calibration mechanism training parameters during model training are as follows: The optimizer for the self-calibrating attention network and feedback signal conversion unit is AdamW, with an initial learning rate of 3e-5 and a weight decay coefficient of 1e-5; a self-calibrating loss function is used.

[0153]

[0154] in, For the first Secondary fusion features of individual samples For the first The model incorporates the true target features of each sample; during training, a gradient clipping strategy is employed, with the upper limit of the gradient norm set to 1.0 to prevent gradient explosion; the parameters of the self-calibration mechanism are synchronously and iteratively updated with the parameters of the preceding modules to ensure end-to-end training of the entire model.

[0155] This design constructs an attention-feedback self-correction mechanism, clearly defining the composition and function of each unit and providing detailed calculation rules. This mechanism forms a closed-loop feedback from decision results to feature adjustment, effectively transforming decision result information into feedback vectors and transmitting them to the feature adjustment process. Secondary weight calculation considers decision feedback, ensuring that weight adjustment matches decision confidence and avoiding ineffective adjustment. Abnormal modality identification and correction rules automatically reduce abnormal modality interference or enhance effective information without manual intervention. The self-correction loss function and training strategy ensure effective learning of the self-correction mechanism, improving the reliability and effectiveness of secondary fusion features and enhancing the model's ability to cope with abnormal situations.

[0156] In one embodiment, the specific process by which the final decision module outputs the final evaluation decision result based on the fusion features after secondary adjustment, thereby enhancing the consistency and stability of the decision, includes:

[0157] A decision optimization module is constructed, comprising a consistency verification layer and a result output layer. Its core function is to ensure the consistency and reliability of the final decision result, avoiding decision fluctuations caused by feature correction. The result output layer (final decision decoder) adopts a 4-layer fully connected network structure with the following number of neurons: , , , , As a decision output dimension, in classification tasks For the number of categories in a regression task The activation function used is GELU, and the dropout rate is set to 0.2.

[0158] Secondary fusion features Input consistency check layer, calculate the similarity of the preliminary fusion feature and the feature:

[0159]

[0160] The similarity value range is [-1, 1], and the closer the value is to 1, the higher the consistency of the twice fusion feature;

[0161] Consistency check threshold The supplementary determination rule is: based on the adaptive setting of the feature similarity distribution of the training set, calculate the mean and standard deviation of the similarity of the preliminary fusion feature and the twice fusion feature of all samples in the training set and , then , and Need to be limited in interval, if the calculation result exceeds the interval, take the interval boundary value, ensure that the threshold meets the data distribution rule and meets the actual decision consistency requirement;

[0162] Judge whether the feature similarity meets the preset threshold , this check process can effectively detect the rationality of feature correction;

[0163] If , it means that the twice fusion feature optimizes the influence of abnormal mode on the basis of retaining the core decision information, and the consistency meets the requirements, then directly input the twice fusion feature to the result output layer, output the final evaluation decision result:

[0164]

[0165] If , it means that the twice fusion feature is quite different from the preliminary fusion feature, and there may be over-correction risk, then the twice fusion feature is iteratively optimized through the result output layer, and the correction strength coefficient is gradually reduced by in the iteration process, the twice correction weight and the fusion feature are recalculated, and the upper limit of the iteration number is set to 3 times, if it still does not meet after 3 iterations, the weighted fusion result of the initial fusion feature and the twice fusion feature is used as input, and the final evaluation decision result is output again;

[0166] Among them, represents the final decision decoder, which uses a deeper fully connected network structure to improve the decision accuracy;

[0167] The final evaluation decision result satisfies a decision consistency constraint, and the constraint threshold value The supplementary determination rule is:

[0168] In the regression task Wherein is the true decision result, ensuring that the error matches the target value range;

[0169] In the classification task That is, the category of the decision result of the adjacent two times is completely consistent, and through the constraint, the stability of the decision result is ensured, and the large change of the decision result caused by data fluctuation or modal anomaly is avoided;

[0170] The decision optimization module in the model training stage trains the parameters: the optimizer of the final decision decoder uses AdamW, the initial value of the learning rate is set to 2e-5, and the weight decay coefficient is 1e-5; the loss function uses a hybrid loss:

[0171]

[0172] Wherein, is the task loss, the cross-entropy loss is used for the classification task, and the mean square error loss is used for the regression task;

[0173]

[0174] is the L1 loss, is the L2 loss, which is selected according to the task requirement;

[0175] is the consistency loss weight, the value range is 0.3-0.7, and the value is determined by the verification set optimization;

[0176] The overall training process of the model: the mini-batch training method is used, and the batch size is set according to the GPU memory self-adaptation, and the value range is 16-64;

[0177] The number of training rounds is set to 100, and the early stopping strategy is used, that is, when the verification set loss does not decrease for 10 consecutive rounds, the training is stopped and the optimal model parameters are saved;

[0178] The learning rate scheduling uses the cosine annealing strategy, the initial learning rate is set according to the above values of each module, the learning rate is decayed to 0.5 of the current value every 20 rounds, and the minimum learning rate is not less than 1e-7; during the training process, the loss value, decision accuracy (classification task) or R² coefficient (regression task) of the training set and the verification set are monitored in real time, to ensure that the model converges and has no overfitting.

[0179] In this way, the decision optimization module is designed, the structure and function thereof are defined, and detailed calculation and judgment rules are given. The consistency check layer can detect feature correction rationality, avoid decision fluctuation caused by excessive correction, adopt different processing modes according to the similarity judgment result, ensure that the final decision result retains core information and optimizes abnormal influence, the decision consistency constraint is set according to the task type, the stability of the decision is ensured, the mixed loss function comprehensively considers the task loss and consistency loss, and the model is guided to optimize. The overall training process is reasonable, the model can converge without overfitting, the accuracy and stability of the final decision result are improved, and the practical application requirements are met.

[0180] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment methods can be completed by programs instructing relevant hardware, therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product in the form of being implemented on one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program codes.

[0181] The above embodiments are used to introduce the present application in detail, and the principles and embodiments of the present application are described by applying specific examples. The above embodiment is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific embodiments and application range will be changed, and the above description should not be understood as the limitation of the present application.

Claims

1. An attention mechanism based multi-modal feature evaluation decision system, characterized in that, The system comprises: a data preprocessing module, a feature encoding module, a preliminary decision module, an attention self-correction module, and a final decision module; The data preprocessing module is configured to acquire multi-modal to-be-evaluated data and perform standardization processing to form a standardized multi-modal feature set; The feature encoding module is configured to perform feature extraction on the standardized multi-modal feature set to generate initial feature representations of each modality; The preliminary decision module is configured to realize weight distribution and fusion of features of each modality by using a basic attention network, and output a preliminary evaluation decision result; The attention self-correction module is configured to construct a feedback self-correction mechanism, and perform secondary adjustment on the weight of the modality feature based on the preliminary decision result, to realize adaptive correction of the abnormal modality; The final decision module is configured to output a final evaluation decision result based on the secondary adjusted fusion features, to improve the consistency and stability of the decision. 2.The attention mechanism based multi-modal feature evaluation decision system according to claim 1, wherein, The data preprocessing module comprises: a data acquisition unit, a standardization processing unit, a feature set integration unit, and a training data optimization unit; The data acquisition unit is configured to collect multi-modal original data containing multiple data types, and the multiple data types include at least two of text, image, numerical value, and audio; The standardization processing unit is configured to map the data of each modality to a similar numerical value interval by eliminating the differences in dimension and numerical value range between different modalities; The feature set integration unit is configured to integrate the standardized data of each modality to form a standardized multi-modal feature set with unified structure; The training data optimization unit is configured to realize division and data enhancement processing of the data set, and the data set is divided into a training set, a validation set, and a test set according to a preset proportion, and the data amount is expanded by using corresponding enhancement strategies for different modalities. 3.The attention-mechanism-based multi-modal feature evaluation decision system according to claim 1, wherein, The feature encoding module comprises a plurality of dedicated modality encoders, and each dedicated modality encoder is designed according to the data structure and feature expression characteristics of the corresponding modality, and comprises at least one of a Transformer-based encoder suitable for text data, a convolutional neural network encoder for image data, a multi-layer perceptron encoder adapted to numerical data, and a mel-frequency cepstral coefficient encoder for audio data; Each dedicated modality encoder deeply excavates high-dimensional semantic features and potential associated information of the modality data through convolution operation, full connection mapping, and activation function transformation, and outputs initial feature representations of a unified dimension to provide a basis for cross-modality feature fusion. The preliminary decision module comprises: 4.The attention-mechanism-based multi-modal feature evaluation decision system of claim 1, wherein, a basic attention network unit and a preliminary decision output unit; The basic attention network unit adopts a multi-head attention mechanism, calculates the correlation weight between the initial features of each modality by constructing a query matrix, a key matrix, and a value matrix, and the weight distribution is constrained by the feature variance and the target correlation of the modality, to ensure that important modalities obtain higher weights; The preliminary decision output unit is composed of a multi-layer fully connected network, maps the preliminary fusion features after weighted fusion to a decision space, and outputs a preliminary evaluation decision result according to the evaluation task type, and the evaluation task type includes a classification task and a regression task. The attention self-correction module comprises:

5. The attention mechanism based multi-modal feature evaluation decision system according to claim 1, wherein, ​ The feedback signal conversion unit, the self-correcting attention network unit, the feature correction unit and the secondary fusion unit; The feedback signal conversion unit converts the preliminary evaluation decision result into a feedback feature vector consistent with the modal feature dimension through a nonlinear conversion network; The self-correcting attention network unit adds a feedback information interaction channel based on the basic attention network structure, calculates the secondary correction weight by splicing the modal initial feature and the feedback feature vector; The feature correction unit adaptively adjusts each modal initial feature based on the secondary correction weight to reduce abnormal modal interference; The secondary fusion unit weights and fuses the corrected modal features to generate secondary fusion features.

6. The attention mechanism-based multi-modal feature evaluation decision system according to claim 5, characterized in that: The secondary correction weight calculation of the self-correcting attention network unit introduces a decision confidence coefficient; The decision confidence coefficient is determined based on the probability distribution of the preliminary decision result, and different weight adjustment amplitude constraints are set according to the confidence level; When the confidence is higher than the first threshold, the weight is limited to small adjustment; When the confidence is between the first threshold and the second threshold, moderate weight adjustment is allowed; When the confidence is lower than the second threshold, large weight reconstruction is supported, and the first threshold is greater than the second threshold.

7. The attention mechanism based multi-modal feature evaluation decision system according to claim 5, wherein, The feature correction unit includes: An abnormal modal judgment subunit and a feature adjustment subunit; The abnormal modal judgment subunit judges the abnormal modal through the dual conditions of modal weight size and feature similarity; The feature adjustment subunit realizes adaptive adjustment of modal features based on the secondary correction weight and feature correction increment, and the correction strength coefficient is negatively correlated with the decision confidence, and the value range is restricted within a preset interval to avoid feature distortion caused by excessive correction. 8.The attention-mechanism-based multi-modal feature evaluation decision system of claim 1, wherein, The final decision module includes: A consistency checking unit, a decision optimization unit and a final result output unit; The consistency checking unit calculates the similarity between the secondary fusion features and the preliminary fusion features, and compares it with a preset consistency threshold; The decision optimization unit selects the feature input mode according to the similarity comparison result, directly inputs the secondary fusion features when the similarity meets the threshold requirement, otherwise adjusts the input features through iterative optimization or weighted fusion strategy; The final result output unit adopts a deep fully connected network structure to convert the optimized fusion features into a final evaluation decision result, meeting the decision consistency constraint requirement.

9. The attention mechanism-based multi-modal feature evaluation decision system according to claim 8, characterized in that: The preset consistency threshold of the consistency checking unit is adaptively set based on the feature similarity distribution of the training set, determined by calculating the mean and standard deviation of the similarity of the training set samples, and the threshold is limited within a preset interval; The decision consistency constraint sets corresponding constraint standards for different task types, matches the constraint error with the target value range in regression tasks, and requires the adjacent decision results to be consistent in classification tasks.

10. The attention mechanism-based multi-modal feature evaluation decision system according to any one of claims 1-9, characterized in that: The system modules are trained in an end-to-end manner, and an optimizer, a loss function and a training scheduling strategy are uniformly configured; The optimizer is an AdamW optimizer, and each module sets a corresponding learning rate and a weight decay coefficient according to functional requirements; The loss function is selected according to the task type and the module function, and includes a cross-entropy loss function, a mean square error loss function, a self-correction loss function and a mixed loss function; The training scheduling strategy includes an early stopping strategy, a cosine annealing learning rate scheduling and a gradient clipping strategy, to ensure stable convergence of the model.

Images