A multi-modal data fusion classification method

Abstract

The application relates to a multi-modal data fusion classification method, comprising the following steps: acquiring three-modal data, respectively encoding the three-modal data to obtain encoded data, wherein the three-modal data comprises sensor time series data, operation and maintenance log documents, device association topology and process knowledge graphs; performing cross-modal fusion on the encoded data to obtain fusion data; performing time series compression and classification on the fusion data to obtain an original vector; based on Dirichlet distribution, performing evidence output and selective prediction on the original vector to obtain a classification result; the classification result comprises a category prediction probability, a nullity degree and a waiver result. The scheme of the application introduces an uncertainty head for an encoder, and introduces the precision signal output in the fusion process in two layers of attention and gating, so that the fusion robustness is good; through Dirichlet evidence output and selective prediction, the sample is automatically handed over to manual review when the sample is not sure, so that the calibration and operability are good.

Description

Technical Field

[0001] This invention relates to the field of multimodal data fusion and deep learning technology, specifically to a multimodal data fusion and classification method. Background Technology

[0002] Condition monitoring and fault diagnosis of industrial equipment typically require the integration of information from multiple sources. In a typical equipment health management system, at least three types of data are available: first, numerical time-series data collected by sensors, including continuous signals reflecting the physical state of equipment operation such as vibration amplitude, bearing temperature, and lubricating oil pressure; second, textual records generated by maintenance personnel during inspections and repairs, including daily inspection logs, fault repair reports, and equipment alarm descriptions; and third, equipment domain knowledge encoded in a graph format, such as mechanical transmission relationships between equipment, upstream and downstream connections within the same process, and semantic descriptions generated based on equipment specifications and historical cases. Effectively integrating these three types of heterogeneous data is a fundamental requirement for improving the accuracy of fault diagnosis.

[0003] The existing data fusion classification output has the following problems: (1) Data fusion is carried out using the gated cross-attention method, but the input of the gated signal only contains the modal feature vector itself and does not contain any information about whether the feature is reliable. Therefore, it lacks the ability to perceive the reliability of each modality. When the data reliability of a certain modality decreases significantly at a certain time step (for example, the sensor drifts due to aging, or there are no operation and maintenance logs at all during a certain period), the fusion mechanism cannot reduce the contribution weight of the modality accordingly, resulting in the fusion result being polluted by low-quality data. In severe cases, the fusion performance is worse than that of a single modality. (2) The probability values ​​output by classifiers based on softmax normalization lack calibrability, meaning that the magnitude of the softmax probability cannot truly reflect the reliability of the prediction. For two samples with the same output probability of 0.85, the model may be very confident about one and actually guessing about the other, but softmax cannot distinguish between these two situations. In industrial fault diagnosis, maintenance personnel need to allocate limited maintenance resources based on the model's confidence level, and the uncalibrable confidence level directly affects the quality of decision-making. (3) The labeling cost of industrial equipment monitoring data is high, and fault samples are particularly scarce, resulting in a limited training set size. Embedded spatial data augmentation (such as linear interpolation in the encoder output space) is an effective means to alleviate overfitting. However, existing methods do not process uncertainty information when interpolating the encoder output. For fusion models that do not have the ability to perceive uncertainty, after directly applying embedding augmentation, the features received by the gating network that have been perturbed by interpolation are no different from the regular features, and it is impossible to distinguish between augmented noise and real signals, resulting in a significant degradation in classification performance. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a multimodal data fusion classification method that enables the encoder to report the reliability of the output feature representation at the same time, and enables the fusion module to use this reliability signal for data fusion at both the attention calculation and gating modulation levels, and makes the uncertainty measure of the classification output both calibrable and operable while maintaining the efficiency of single forward inference.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A multimodal data fusion and classification method, comprising: The three modal data are acquired and encoded to obtain encoded data. The three modal data include sensor time-series data, operation and maintenance log documents, and equipment association topology and process knowledge graph; the three modalities are sensor modality, document modality, and knowledge modality. The encoded data is fused across modes to obtain fused data; The fused data is then subjected to temporal compression and classification to obtain the original vector; Based on the Dirichlet distribution, the original vector is subjected to evidence-based output and selective prediction to obtain classification results; the classification results include class prediction probability, emptyness degree and abstention result.

[0006] Based on the above technical solution, the present invention can be further improved as follows.

[0007] Furthermore, the acquisition of trimodal data, and the encoding of the trimodal data to obtain encoded data, includes: The sensor time-series data is encoded using a sensor encoder and a corresponding uncertainty header to obtain sensor encoded data; the sensor encoded data includes mean, logarithmic variance, time-step accuracy, and scalar accuracy. The operation and maintenance log document is encoded using a document encoder and a corresponding uncertainty header to obtain document encoded data; the document encoded data includes mean, logarithmic variance, time-step precision, and scalar precision. Knowledge embedding data is obtained by encoding the device association topology and process knowledge graph using a knowledge encoder.

[0008] Furthermore, the step of encoding the device association topology and process knowledge graph using a knowledge encoder to obtain knowledge embedding data includes: A knowledge encoder is used to encode the device association topology and process knowledge graph to obtain a graph embedding vector; A semantic description embedding vector is generated from the equipment association topology and process knowledge graph using a large language model; The knowledge embedding data is obtained by gating and fusing the graph embedding vector and the semantic description embedding vector.

[0009] Furthermore, the cross-modal fusion of the encoded data to obtain fused data includes: In the first stage of fusion, the sensor encoded data and the document encoded data are subjected to precision weighting and precision gating fusion to obtain the first fused data; In the second stage of fusion, the first fused data and the knowledge embedded data are subjected to precision weighting and precision gating fusion to obtain the final fused data.

[0010] Furthermore, in the first stage of fusion, precision-weighted and precision-gated fusion is performed on the sensor encoded data and the document encoded data to obtain the first fused data, including: Set the sensor mode as the first primary mode and the document mode as the first auxiliary mode; The first auxiliary mode mean is obtained by precision weighting. The first content data is calculated based on the mean of the first primary mode and the weighted mean of the first auxiliary mode; The first gated data is obtained based on the mean of the first primary mode, the scalar precision of the first primary mode, the mean of the first auxiliary mode, and the scalar precision of the first auxiliary mode. First fused data is obtained based on the first content data and the first gating data.

[0011] Further, the first content data is calculated based on the mean of the first primary mode and the weighted mean of the first auxiliary mode, including: The first attention head correlation value between the main mode and the auxiliary mode is calculated based on the first main mode mean and the first auxiliary mode weighted mean; Calculate the first multi-head cross-attention output value based on the first attention head correlation value; The first content data is calculated based on the first multi-head cross-attention output value.

[0012] Furthermore, in the second-stage fusion, the first fused data and the knowledge embedding data are subjected to precision-weighted and precision-gated fusion to obtain the final fused data, including: The first-stage fusion output is set as the second primary modality, and the knowledge modality is set as the second auxiliary modality; The second principal mode mean, the second principal mode scalar precision, the second auxiliary mode mean, and the second auxiliary mode scalar precision are determined based on the first fused data and the knowledge embedded data. The second auxiliary mode mean is obtained by precision weighting. The second content data is calculated based on the mean of the second primary mode and the weighted mean of the second auxiliary mode; The second gating data is obtained based on the mean of the second primary mode, the scalar precision of the second primary mode, the mean of the second auxiliary mode, and the scalar precision of the second auxiliary mode; The final fused data is obtained based on the second content data and the second gating data.

[0013] Further, determining the second primary mode mean, the second primary mode scalar precision, the second auxiliary mode mean, and the second auxiliary mode scalar precision based on the first fused data and the knowledge embedding data includes: The first fused data is determined as the mean of the second dominant mode; The arithmetic mean of the logarithmic variances of the first principal mode and the first auxiliary mode is determined as the logarithmic variance of the second principal mode, and the scalar accuracy of the second principal mode is calculated based on the logarithmic variance of the second principal mode. The knowledge-embedded data is copied along the time dimension and then used for time-series fusion to obtain the mean of the second auxiliary mode; The knowledge embedding data includes a fixed precision, and the precision of the second auxiliary modality is calculated based on the fixed precision.

[0014] Furthermore, the step of obtaining classification results by performing evidence-based output and selective prediction on the original vector based on the Dirichlet distribution includes: Calculate the Dirichlet concentration parameters for each category based on the original vector; Calculate the total Dirichlet intensity, class prediction probability, and emptyness degree based on the concentration parameters; The abstention result is obtained by selectively predicting based on the emptyness degree and the preset abstention threshold. The predicted probability of the category, the degree of emptiness, and the result of abandonment are output as the classification result.

[0015] Furthermore, the method also includes: Acquire sensor coding training data, document coding training data, and sampling interpolation coefficients; The sampling interpolation coefficients are used to synchronously interpolate the mean representation and its log-variance in the training to obtain augmented sample data and corresponding augmented label data; The enhanced sample data is fused across modes to obtain fused enhanced data; The loss is calculated based on the fused enhanced data, the enhanced label data, and the sampling interpolation coefficients; The loss is used to optimize the training of the encoder and the learnable parameters in the fusion classification process.

[0016] The advantages of adopting the above scheme are: introducing an uncertainty head into the encoder enables the output of a learnable accuracy signal at each time step, and introducing this signal into the fusion process at both the attention level and the gating level, resulting in good fusion robustness; through Dirichlet evidence-based output, the model outputs a higher null value when evidence is insufficient, selective prediction is performed using the abandonment threshold, and samples are automatically transferred to manual review when uncertain, resulting in good model calibration and operability; only a single forward inference can simultaneously output class prediction and uncertainty measurement, resulting in low computational cost. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the multimodal data fusion and classification method of the present invention.

[0018] Figure 2 This is a schematic diagram of the electronic device of the present invention. Detailed Implementation

[0019] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0020] In existing technologies, multimodal fusion methods have evolved from simple concatenation to architectures with interactive modeling capabilities. Early solutions concatenated feature vectors from various modalities and fed them into a unified classifier, but this approach ignored the interaction relationships between modalities. Subsequently, attention-based fusion methods emerged, utilizing cross-attention to model the association between two modalities—for example, allowing sensor features to query operational log features, thus enabling sensor encoding to absorb relevant textual semantic information. Building on this, gated cross-attention methods introduce an additional sigmoid gating signal to control the integration strength of cross-modal information: the output of the cross-attention is first subjected to a linear transformation and then multiplied element-wise with the sigmoid gating vector. The gating vector is obtained by activating the main modality features after another linear transformation. This mechanism is superior to simple attention concatenation in terms of fusion flexibility and performs well in multimodal fusion methods.

[0021] However, the aforementioned gated cross-attention method has a structural deficiency: the input to the gated signal only contains the modal feature vector itself, without any information about whether the feature is reliable. Regardless of whether the sensor reading at a certain time step is normally acquired or severely distorted due to sensor drift, the gated network only sees an encoded vector, and it has no way of judging whether this vector is trustworthy.

[0022] In industrial settings, fluctuations in data reliability are the norm, not the exception. Take, for example, a continuously operating rotating machine: its vibration sensors are initially calibrated well, but after several months of operation, due to loose mounting bases or probe wear, the acquired signals gradually exhibit baseline drift. At this point, the characteristics output by the sensor channels deviate from the actual equipment condition. Simultaneously, the coverage of maintenance logs is uneven—during normal equipment operation, inspection personnel often do not generate any written records, leaving the documented modalities essentially blank during these periods; while during operational transitions (such as sudden load changes or start-stop transitions), historical characteristic patterns may temporarily become invalid. When these low-quality or missing modal data are integrated with equal weight, the fusion result may actually be worse than using only a reliable single modality, resulting in the so-called "fusion degradation" phenomenon.

[0023] At the classification output level, existing methods also have limitations. Currently, most multi-class classification models use the softmax function to normalize the raw output values ​​of the classification network in the output layer into a probability distribution, with the sum of the probability values ​​being 1. The category corresponding to the highest probability is the prediction result. However, the magnitude of the softmax output value is not equivalent to the reliability of the prediction. In practical applications, situations may arise where the model encounters an abnormal pattern that has never appeared in the training set, yet the softmax still gives a "probability" of 0.9 or even higher. In industrial fault diagnosis scenarios, maintenance personnel need to rely on the "confidence" of the model's output to decide whether to trigger downtime for maintenance or dispatch repair personnel. An uncalibrated high-confidence prediction may lead to false negatives (the model determines the equipment is normal, but it is actually in the early stages of a fault, missing the optimal maintenance window) or false positives (the model issues fault alarms for normally operating equipment, leading to unnecessary downtime and production losses).

[0024] Existing methods attempt to address the uncertainty of classification outputs. Bayesian approximation methods estimate uncertainty by performing multiple random forward propagations during the inference phase (e.g., repeatedly inferring dozens of times after enabling a random deactivation layer), using the discreteness of the output distribution. While theoretically sound, this approach incurs computational costs dozens of times greater than a single forward inference, making it difficult to deploy in real-time industrial online monitoring scenarios. Furthermore, the randomness of multiple samplings necessitates additional engineering management and result aggregation logic, increasing the complexity of system integration.

[0025] To address the aforementioned issues, this application provides a multimodal data fusion and classification method, applicable to application scenarios such as industrial equipment health management and predictive maintenance, which require making reliable decisions under conditions of uneven modal data quality.

[0026] Figure 1 This is a flowchart illustrating the multimodal data fusion and classification method of the present invention, as shown below. Figure 1This application provides a multimodal data fusion and classification method, including steps S1-S4.

[0027] In one embodiment, this application may also provide a model or system for performing the following multimodal data fusion classification method.

[0028] S1. Obtain the three-modal data and encode the three-modal data respectively to obtain encoded data. The three-modal data includes sensor time series data, operation and maintenance log documents, equipment association topology and process knowledge graph.

[0029] In this embodiment, the three modalities are named: sensor modality, document modality, and knowledge modality. This step maps the raw data of the three heterogeneous modalities to a unified d-dimensional latent space, and introduces uncertainty estimation for the required encoder, as detailed in steps S11-S13.

[0030] S11. Using the sensor encoder and the corresponding uncertainty head, the sensor time series data is encoded to obtain sensor encoded data; the sensor encoded data includes mean, logarithmic variance, time step accuracy, and scalar accuracy.

[0031] Specifically, input sensor timing data , where T is the number of monitoring cycles (e.g., time steps). For each time step, based on sensor feature dimensions (such as vibration amplitude, temperature, pressure, and other multi-channel data), the sensor encoder extracts the feature vector of the sensor time-series data. The uncertainty head then projects the sensor time-series data through linear projection. The input is mapped to a d-dimensional latent space to obtain the mean representation. and logarithmic variance :

[0032] in, For encoder functions, and These are the weight matrix and bias vector of the uncertainty head, respectively. The clamp operation will convert the log-variance... The output value is limited to the clamping range. In one embodiment of the present invention, .

[0033] The initialization strategy of the uncertain header directly affects the training stability of the system. This invention uses the weight matrix... Initialized as an all-zero matrix, biased Initialize to -1.0. At the start of training, the log-variance output for all time steps is -1.0, corresponding to a precision of approximately 2.7. This starting point was chosen because it falls within the clamping interval [-3, 2] at a relatively high precision level, but not excessively so. If the initial precision is too high (e.g., close to the upper limit),... Gated networks tend to over-rely on the embedding vectors output by encoders that are not yet fully trained, amplifying initial noise; if the initial accuracy is too low (e.g., close to the lower limit), the network may fail to achieve its intended effect. The fused signal will be almost completely suppressed in the early stages of training, making it difficult for the encoder to obtain effective gradients from the fused loss.

[0034] The upper and lower bounds of the clamping interval each have clear engineering implications. Lower bound Limit the maximum accuracy to This prevents any single time step from gaining absolute dominance in the fusion computation—without this constraint, a very small number of anomalous batches have been observed to push accuracy to the order of 10³, leading to training failure after several training epochs. Upper bound Limit the minimum precision to This ensures that even the most unreliable time step retains some information contribution and is not completely "silenced".

[0035] In log variance Based on this, define the precision for each time step and each dimension. :

[0036] Higher precision indicates more reliable encoding at that location. The scalar precision is obtained by averaging the precision along the feature dimension d. :

[0037] in, The exponential function, where d is the feature dimension and j is the precision. The index along the feature dimension.

[0038] S12. Using a document encoder and the corresponding uncertainty header, the operation and maintenance log document is encoded to obtain document encoded data; the document encoded data includes mean, logarithmic variance, time-step precision, and scalar precision.

[0039] Precoded vector sequence of input operation and maintenance log text The vector at each time step is an aggregated representation of the operation and maintenance logs for that time period, encoded by a pre-trained language model. If the document encoder's structure is symmetrical to the sensor encoder, then the output mean... and logarithmic variance :

[0040] According to the mean and logarithmic variance Define the precision per time step and per dimension. And calculate scalar precision :

[0041] During periods when the equipment is operating normally and there are no inspection records, the document input may be a zero vector or a noise-padded vector. At this time, the uncertainty head of the document encoder tends to output a higher log-variance (i.e., lower precision). The subsequent fusion module automatically reduces the contribution of the document modality at this time step, thereby avoiding the blank input from misleading the fusion result.

[0042] S13. Use a knowledge encoder to encode the equipment association topology and process knowledge graph to obtain knowledge embedding data.

[0043] The equipment association topology and process knowledge graph G = (V, E, R) describes the relationship structure between equipment. V is the set of equipment nodes, E is the set of edges, and R is the set of relationship types (such as "coaxial connection", "upstream and downstream of the same process", etc.). This invention uses a two-layer relational graph convolutional network (R-GCN, hereinafter referred to as the knowledge encoder) to encode the graph, where the relation weight matrix uses basis decomposition to reduce the number of parameters.

[0044] Specifically, a knowledge encoder is used to encode the equipment association topology and process knowledge graph to obtain graph embedding vectors. The Large Language Model (LLM) is used to generate semantic description embedding vectors for each device. Semantic description of embedded vectors This is used to summarize information such as the device's specifications and historical failure modes.

[0045] Embedding a graph into a vector Semantic description embedding vector Knowledge embedding data is obtained by performing sigmoid-gated fusion. :

[0046] in, For the sigmoid function, For element-wise multiplication, and Let be a learnable linear transformation matrix. Where, and These are the linear transformation matrices acting on the semantic embedding and graph embedding of the large language model, respectively. They are not directly calculated from external rules or manual statistics, but rather are learnable parameters within the neural network. The semantic embedding dimension of the large language model is set to... The graph embedding dimension is The knowledge fusion output dimension is The corresponding bias vector and These are also learnable parameters, and their dimensions are all... Then we have: , , .

[0047] It's important to note that the knowledge encoder does not attach an uncertainty head. This is because R-GCN generates the same embedding vector for the same device across all monitoring periods—the graph structure does not change with monitoring time, and its temporal variance is structurally zero. LLM semantic embeddings, generated based on a fixed template, also lack time-varying characteristics. Attaching a learnable variance parameter to a source with no meaningful variation in the temporal dimension only increases model parameters without providing additional information. Therefore, the log-variance of the knowledge modality is fixed at -1.0, corresponding to a fixed precision of 2.7.

[0048] S2. Perform cross-modal fusion on the encoded data to obtain fused data.

[0049] In the cross-modal fusion process of this step, the precision signals obtained in step S1 are fused at the attention level and the gating level respectively, so that low-reliability information is automatically suppressed, see steps S21-S22 for details.

[0050] S21. In the first stage of fusion, the sensor encoded data and document encoded data are subjected to precision weighting and precision gating fusion to obtain the first fused data.

[0051] In the first phase of fusion, the sensor modality is set as the first primary modality (main), and the document modality is set as the first auxiliary modality (aux).

[0052] Specifically, for the mean of the first auxiliary mode The first auxiliary mode weighted mean is obtained by performing precision weighting. First, calculate the time mean of the accuracy of the first auxiliary mode. :

[0053] Then calculate the reliability weight relative to the mean at each time step. :

[0054] Where T is the number of monitoring periods and t is the number of time steps. It is a learnable temperature scalar, initialized to 5.0. The sigmoid function maps the precision deviation to the interval [0, 1], so that time steps with precision higher than the time mean receive a weight greater than 0.5, and time steps with precision lower than the mean receive a weight less than 0.5.

[0055] Weighted mean of the first auxiliary mode Represented as: .

[0056] Using the first principal mode mean Weighted mean of the first auxiliary mode Calculate the query vector for multi-head cross-attention output. (Query), key vector (Key) and value vector (Value):

[0057]

[0058]

[0059] in, , , These are the projection matrices of the query, key, and value for the m-th attention head, respectively, and are all learnable parameters.

[0060] The m-th attention head uses scaled dot product attention to calculate the correlation between the primary and secondary modes, obtaining the correlation value of the first attention head with a certain number in the first-stage fusion. :

[0061] The outputs of all attention heads are aggregated to obtain the multi-head cross-attention output value:

[0062] Where m is the attention head number, M is the total number of attention heads, and d′ is the feature dimension of a single attention head; This is a scaling factor used to prevent the softmax function from becoming oversaturated due to excessively large dot product values.

[0063] Multi-head cross-attention output value The first content data is obtained through linear transformation. :

[0064] Based on the mean of the first dominant mode First principal mode scalar accuracy Mean of the first auxiliary mode First auxiliary mode scalar accuracy Obtain the first gating data :

[0065] in, This is a concatenation operation along the feature dimension. The total dimension of the gated input is 2d+2 (i.e., two d-dimensional mean vectors plus two scalar precision values). (The transformed result). (Take a certain precision). This is to compress its numerical range, making scalar precision more compatible with d-dimensional eigenvectors in terms of numerical scale. and These are the learnable parameters used for the first-stage fusion. The bias is initialized to 0.5, so that the sigmoid output is slightly higher than 0.5 in the early stage of training, which encourages information flow.

[0066] First content data and the first gate control data After element-wise multiplication, the first fused data is obtained through layer normalization and dropout regularization. :

[0067] In this embodiment, by incorporating the accuracy signal into the gate input, the gating network obtains an additional criterion beyond the pure feature vectors: when both modalities report high accuracy, the gating tends to fully fuse them; when the primary modality has high accuracy but the secondary modality has low accuracy, the gating tends to weaken the fusion strength, preserving the original information of the primary modality. This behavior cannot be achieved by the feature vectors themselves alone, because a low-reliability feature vector and a high-reliability feature vector may be completely similar in numerical appearance.

[0068] S22. In the second stage of fusion, the first fused data and the knowledge embedded data are subjected to precision weighting and precision gating fusion to obtain the final fused data.

[0069] The second-stage fusion still uses a two-layer fusion of attention layer and gating layer. The learnable parameters in the second stage and the first stage are learned independently and are not shared.

[0070] Specifically, in the second-stage fusion, the output of the first-stage fusion is used as the mean of the second principal mode, that is: The knowledge modality is used as the second auxiliary modality. Here, (1) represents the first stage of fusion, and (2) represents the second stage of fusion.

[0071] Log-variance of the second principal mode The result is obtained by taking the arithmetic mean of the logarithmic variances of the first principal mode and the first auxiliary mode:

[0072] Based on the log-variance of the second principal mode Dimensional accuracy of calculating the second principal mode :

[0073] The scalar precision is obtained by taking the mean value of the precision along the feature dimension d. :

[0074] Embedding knowledge into data The second auxiliary mode mean is obtained by copying along the time dimension and participating in time series fusion. :

[0075] The knowledge mode does not have a separate uncertainty head, therefore the log-variance of the second auxiliary mode is... It is a fixed value, that is:

[0076] Based on the log-variance of the second auxiliary mode Calculate the dimension-by-dimensional accuracy of the second auxiliary mode :

[0077] Based on the dimension-wise accuracy of the second auxiliary mode Calculate the accuracy of the second auxiliary mode scalar for:

[0078] Based on the above calculation formula, according to the first fused data and knowledge embedded data Determine the mean of the second principal mode Second principal mode scalar accuracy Mean of the second auxiliary mode Second auxiliary mode scalar accuracy .

[0079] The accuracy of the second auxiliary mode is the same at all time steps, therefore the weights are calculated based on both precision and scalar precision. Simplified to:

[0080] Based on weight Calculate the weighted mean of the second auxiliary mode. Represented as:

[0081] Weighted average of the second auxiliary mode Key vectors for cross-attention Sum value vector The first-stage fusion output serves as the query vector. This allows us to obtain the second multi-head cross-attention output value in the second-stage fusion. (The calculation method is the same as in the first stage, and will not be described in detail here):

[0082] Second multi-head cross-attention output value The second content data is obtained through linear transformation. :

[0083] Based on the mean of the second dominant mode Second principal mode scalar accuracy Mean of the second auxiliary mode Second auxiliary mode scalar accuracy Obtain the second gating data :

[0084] in, , , , These are the learnable parameters for the second fusion stage.

[0085] The second content data Second gated data After element-wise multiplication, the second fused data is obtained through layer normalization and dropout regularization. Second fusion data This is also the final fused data. :

[0086] In the second-stage fusion, the knowledge modality is static, so the logarithmic variance of the knowledge modality is a fixed value, and the precision weighting at the attention level degenerates to a fixed scaling of 0.5. The reliability modulation that truly varies with the samples in the second-stage fusion mainly occurs at the gating level.

[0087] As described above, the cross-modal fusion process involves a cascaded design of first-stage fusion and second-stage fusion. The interaction information between the sensor and the document is first extracted and filtered, and then integrated with relatively stable domain knowledge information to form the final multimodal fusion representation.

[0088] In this embodiment, an uncertainty head is introduced into the encoder in step S1 so that a learnable precision signal is output at each time step. This signal is then introduced into the fusion process at both the attention level and the gating level in step S2 so that the information stream can be modulated simultaneously.

[0089] At the attention level, the deviation of the accuracy of each time step of the auxiliary modality relative to the time mean is calculated. After being weighted by a sigmoid mapping controlled by a learnable temperature parameter, the value vector is automatically attenuated, thus reducing the contribution of low-reliability time steps to the attention output. At the gating level, the scalar accuracy of each of the two modalities is concatenated into the input vector of the gating network, making the generation of the gating signal dependent not only on the modal feature content but also on the reliability of the features. The two levels complement each other: the attention level performs fine-grained, dimension-wise, and time-step-wise modulation; the gating level performs macroscopic fusion strength control. Thus, cross-modal fusion has the ability to perceive modal quality changes caused by sensor drift, log missing, etc., and helps maintain a relatively stable fusion effect under conditions of uneven data reliability.

[0090] S3. Perform time-series compression and classification on the fused data to obtain the original vector.

[0091] Specifically, the fused data output in step S2 (Where B is the batch size, T is the number of time steps, and d is the feature dimension) The time dimension is compressed using a two-layer gated recurrent unit (GRU) network, and the hidden state at the last time step of the last layer is taken as the convergent representation of the entire monitoring window, with dimension d. Then, a two-layer multilayer perceptron (MLP) is used for classification to output the original classification output vector. (Where C is the number of categories, and in the three-category classification of equipment status, C = 3, corresponding to the three states of normal, sub-health, and fault respectively).

[0092] S4. Based on the Dirichlet distribution, the original vector is subjected to evidence-based output and selective prediction to obtain the classification result; the classification result includes the category prediction probability, the degree of emptiness, and the abstention result.

[0093] This step replaces the traditional softmax classifier with an evidence-based output structure based on the Dirichlet distribution, enabling the model to output both the class prediction result and the uncertainty measure in a single forward inference.

[0094] The steps for converting the original vector into evidence output are as follows: The classification network MLP first outputs the original vector z, and then uses the softplus function to convert the original vector z into non-negative evidence. Then, by adding 1 to each category of evidence, the concentration parameters of the Dirichlet distribution are obtained. :

[0095]

[0096] in, A vector of all 1s, used to represent the Dirichlet concentration parameters for each category. It satisfies the above formula and corresponds to a uniform Dirichlet prior, where C is the total number of categories in the classification task.

[0097] Based on the Dirichlet concentration parameters for each category, calculate the total Dirichlet intensity S (the sum of concentration parameters for all categories) and the predicted probability for each category. And the uncertainty of emptiness degree u:

[0098]

[0099]

[0100] In this embodiment, the Dirichlet distribution provides point estimates of the probabilities of each class and can characterize the model's confidence in these estimates. When the model has limited evidence for a particular sample, the evidence for each class... Both are close to 0. At this point, the total Dirichlet strength S is close to the total number of categories C in the classification task, and the emptiness degree u is close to 1, indicating that the model has high uncertainty. As the model accumulates more evidence for the correct category, the total Dirichlet strength S increases, and the emptiness degree u decreases accordingly and approaches 0.

[0101] By using Dirichlet evidence-based output, the model outputs a higher degree of emptiness even when evidence is insufficient. In quantitative evaluation, the expected calibration error (ECE) is lower than that of the softmax method in the same framework, indicating a better correlation between the uncertainty metric u of the evidence-based output and the actual error rate, and good calibrability of the uncertainty metric.

[0102] In this step, based on the emptiness degree u and the preset abstention threshold... Selective prediction is performed to obtain the abstention result. Specifically, during the inference phase, the emptiness degree u is calculated for each sample. An abstention threshold is preset. The waiver threshold is configured by the user based on specific business needs; when In such cases, the system does not output an automatic diagnostic conclusion, but instead marks the sample as "awaiting manual review" or triggers additional sensor sampling. Optionally, the "awaiting manual review" or "additional sensor sampling" information is output as an abstention result. A threshold uncertainty can be set, automatically marking a sample as "awaiting manual review" when the model is not sufficiently confident in its judgment, thus assigning a higher risk of actual misjudgment to abstention samples, and making the uncertainty measurement highly operable.

[0103] Based on the above, the method in this embodiment ultimately outputs a classification result including the predicted category probability. The degree of emptiness u and the result of abstention.

[0104] The selective prediction in this embodiment enables the model's diagnostic risk (e.g., the proportion of incorrect predictions) to decrease as the abandonment coverage rate decreases, meaning the model tends to prioritize abandoning samples with a higher probability of misjudgment.

[0105] Existing softmax-based maximum probability abstention strategies lack stable monotonicity in similar tasks. That is, after abstaining according to softmax probability, the risk does not decrease continuously with decreasing coverage, indicating that softmax probability as a reliability indicator has limitations. The Dirichlet emptyness-based abstention mechanism in this embodiment shows a better monotonic trend in experiments: samples that abstain first typically correspond to a higher risk of misjudgment. This has practical value for industrial applications. Operations personnel can set abstention threshold, allowing the model to automatically transfer samples to manual review when uncertain, rather than forcibly providing potentially incorrect diagnoses. This results in good diagnostic calibration and reduced diagnostic risk.

[0106] Optionally, the method in this embodiment further includes training the learnable parameters in the fusion classification method process. This application uses a mixup interpolation method to perform embedding enhancement in the encoder output space. The embedding enhancement is performed only during the training phase and only applies to sensor encoding and document encoding. The knowledge embedding is a device-level static prior and does not participate in sample-level enhancement.

[0107] Specifically, in each training batch of sensor-encoded training data and document-encoded training data, an interpolation coefficient is randomly sampled from a symmetric Beta distribution. :

[0108] The Beta distribution is a continuous probability distribution defined on the interval (0,1). Used to determine the mixing ratio of two paired samples within a training batch. Let be the shape parameter of this distribution. In one embodiment of the invention, take . Because a symmetrical Beta distribution is used, the two samples have the same sampling rules; when When sampling results are closer to 0 or 1, the augmented sample usually retains the main features of one of the original samples while incorporating some information from the other sample.

[0109] After random pairing within a batch, interpolation coefficients are used. Calculate the augmented sample mean for the i-th and j-th samples randomly paired within the batch. and the corresponding enhanced sample log-variance :

[0110]

[0111] and Let be the mean and logarithmic variance of the i-th sample, respectively. and Let be the mean and logarithmic variance of the j-th sample, respectively. Logarithmic variance synchronous interpolation ensures that the augmented sample retains uncertainty information consistent with its mixing source.

[0112] For the i-th sample and the j-th sample, the sample label enhancement can be represented by the same interpolation coefficients:

[0113]

[0114] in, and These are the true class labels of the i-th and j-th original samples in the training dataset, obtained through one-hot encoding, and are not generated by model prediction.

[0115] In actual training, instead of directly inputting the above-mentioned mixed soft labels into the evidence loss function, the two original ground truth labels are used instead. Calculate the losses separately and weight them according to the mixing coefficients to obtain the evidence-based classification loss for a given true label. This is the total EDL loss used for a single backpropagation when using Mixup:

[0116] in, To enhance the original classification output obtained from the samples through the fusion network and the classification network, Loss is classified as evidence.

[0117] Embedding enhancement is used to enable the model to learn the category information of the i-th and j-th original samples simultaneously, and the contribution of the two parts of the loss is consistent with its mixing ratio.

[0118] As described above, the mean-variance synchronous embedding enhancement with uncertainty protection performs embedding fusion enhancement in the encoder output space by synchronously interpolating the mean representation and log-variance with the same interpolation coefficients, ensuring that the enhanced sample retains uncertainty information consistent with its fusion source. This is a necessary condition for accuracy-aware fusion and embedding enhancement to work synergistically. Control experiments show that embedding enhancement without variance synchronous interpolation can lead to a significant degradation in classification performance; with variance synchronous interpolation, it can identify and moderately attenuate enhanced noise through accuracy signal recognition, transforming the originally harmful embedding perturbation into an effective regularization method, which helps improve generalization performance.

[0119] Alternatively, the training loss can be explained in another way. For the two original real labels The sum of the calculated losses each includes a Bayesian risk term and a KL divergence regularization term. It's worth noting that in the supervision and parameter update stages of EDL training loss: the augmented samples undergo two-stage fusion, GRU, and a classification network to generate evidence-based outputs. Then, the Bayesian risk term and KL regularization term are calculated based on the true class labels. These two terms together constitute a total loss, and a backpropagation and parameter update are performed based on the total loss model.

[0120] Calculate Bayesian risk data items :

[0121] in, The component of the true category one-hot label in the c-th category. Let S be the predicted probability of the c-th category, S be the total Dirichlet intensity, and C be the total number of categories.

[0122] Used to measure the deviation between the predicted probability mean and the true class label. Let be the variance term of the c-th class probability in the Dirichlet posterior, and the sum of these two terms constitutes the expected squared error loss in the Dirichlet posterior.

[0123] Calculate the KL divergence regularization term :

[0124] in, , This refers to the Dirichlet concentration parameter.

[0125] use The operation involves clearing the evidence corresponding to the true class label to zero before calculating the KL divergence regularization term. This ensures that the KL regularization term only penalizes evidence placed on the wrong class. Specifically, it is used to constrain the model from accumulating too much evidence on the wrong class. That is, if the model accumulates a large amount of evidence on a certain wrong class, the KL term will penalize it, prompting the model to maintain a low evidence output for the wrong class.

[0126] Based on the above, let's explain the training loss from another perspective. Based on the calculated Bayesian risk data items and KL divergence regularization term Calculate total loss :

[0127] Where N is the total number of samples in the current batch, and n is the sample number. The true class label number for the nth training sample. This represents the category weight corresponding to the true category.

[0128] Category weights satisfy C represents the total number of categories. The number of samples belonging to class C in the current mini-batch, and the class weight. This is used to recalculate the class distribution of the current mini-batch at each training step, to accommodate situations where the number of normal and faulty samples is imbalanced. In this embodiment, for the nth sample, take... Then there is .

[0129] The KL annealing factor increases gradually during training, and t represents the current training period. Let KL be the number of annealing cycles, satisfying . Used to prioritize the weights of KL regularization The value increases linearly from 0 to 1 within a training cycle. In a preferred embodiment of the invention, The value is set to 100.

[0130] During training, the Mixup stage generates enhanced feature samples and paired labels. After the enhanced samples complete the entire forward propagation (i.e., through fusion, temporal compression, and classification networks), the total EDL loss is calculated uniformly, and then this loss is used for backpropagation to continuously optimize and update the learnable parameters in the entire network model. The learnable parameters in this embodiment mainly include the encoder and its uncertainty head, the knowledge fusion layer, the two-stage attention and gating fusion module, the GRU, and the classification network. The pre-generated LLM semantic embeddings, the topological data of the knowledge graph, and the preset fixed variance do not participate in the update, but the network parameters that process these input data are also trained.

[0131] Optionally, during the training process described above, since the accuracy signal is sensitive to numerical values, the method in this embodiment utilizes strategies such as log-variance clamping, gradient pruning, dynamic class weights, and KL annealing scheduling to ensure the numerical stability of the training.

[0132] In log-variance clamping, the log-variance of the uncertainty head output is hard-limited to... Interval. Due to precision It is an exponential function of the logarithmic variance, and small changes in the logarithmic variance can lead to drastic fluctuations in accuracy.

[0133] In gradient clipping, the L2 norm of the full model gradients is clipped to 1.0 after each backpropagation of the total loss and before parameter updates. Gradient clipping complements log-variance clamping: clamping limits the numerical range of accuracy during forward propagation, while gradient clipping limits the magnitude of the gradients during backpropagation. Together, they prevent accuracy-related parameters from experiencing runaway updates during training.

[0134] In dynamic class weights, the class weights satisfy the following condition: It is important to note that the weights are calculated entirely based on the statistics of the current batch, without using the prior distribution of the global training set, to avoid introducing information that is not currently visible.

[0135] In KL annealing scheduling, the weights of the KL regularization terms are determined by... Linear growth. Linear annealing scheduling can give the model sufficient freedom to explore in the early stages of training, allowing the model to learn to distinguish categories first, and then gradually apply penalty constraints to false evidence, which has a good classification effect on the reasoning process.

[0136] This application demonstrates the computational chain of multimodal data fusion classification: raw classification output → softplus → evidence quantity e → Dirichlet parameter α = e + 1 → predicted probability p = α / S → emptiness degree u = C / S, as well as the two branches of training loss (Bayesian risk data item). and KL divergence regularization term Compared to uncertainty methods based on Bayesian sampling (such as the Monte Carlo random inactivation method, which requires dozens of forward inferences), this application can output both category prediction and uncertainty measure in a single forward inference. Furthermore, the number of additional parameters introduced by the encoder uncertainty head is small, and the additional computational overhead of the evidence output head (one softplus and one summation) is negligible. This results in low computational cost, good robustness of the fusion classification, and the ability to meet the inference latency requirements of industrial online monitoring scenarios.

[0137] In some embodiments, the present invention provides a multimodal data fusion classification system, which can be implemented using a combination of hardware and software. As an example, the multimodal data fusion classification system of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the multimodal data fusion classification method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0138] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0139] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned multimodal data fusion classification methods. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the multimodal data fusion classification method shown in any embodiment of the present invention by calling the computer program.

[0140] In one alternative embodiment, an electronic device is provided, such as Figure 2 As shown, Figure 2The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0141] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0142] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 2 The bus 4002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0143] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0144] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0145] Among them, electronic devices can also be terminal devices, which can be any device that can install applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0146] It should be noted that, Figure 2 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0147] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described multimodal data fusion and classification methods.

[0148] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0149] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned multimodal data fusion classification method.

[0150] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0152] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0153] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0154] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0155] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0156] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0157] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A multimodal data fusion and classification method, characterized in that, include: Acquire trimodal data, and encode the trimodal data respectively to obtain encoded data; The three-modal data includes sensor time-series data, operation and maintenance log documents, and equipment association topology and process knowledge graph; the three modalities are sensor modality, document modality, and knowledge modality. The encoded data is fused across modes to obtain fused data; The fused data is then subjected to temporal compression and classification to obtain the original vector; Based on the Dirichlet distribution, the original vector is subjected to evidence-based output and selective prediction to obtain classification results; the classification results include class prediction probability, emptyness degree and abstention result.

2. The method according to claim 1, characterized in that, The acquisition of trimodal data, and the encoding of the trimodal data to obtain encoded data, includes: The sensor time-series data is encoded using a sensor encoder and a corresponding uncertainty header to obtain sensor encoded data; the sensor encoded data includes mean, logarithmic variance, time-step accuracy, and scalar accuracy. The operation and maintenance log document is encoded using a document encoder and a corresponding uncertainty header to obtain document encoded data; the document encoded data includes mean, logarithmic variance, time-step precision, and scalar precision. Knowledge embedding data is obtained by encoding the device association topology and process knowledge graph using a knowledge encoder.

3. The method according to claim 2, characterized in that, The process of encoding the device association topology and process knowledge graph using a knowledge encoder to obtain knowledge embedding data includes: A knowledge encoder is used to encode the device association topology and process knowledge graph to obtain a graph embedding vector; A semantic description embedding vector is generated from the equipment association topology and process knowledge graph using a large language model; The knowledge embedding data is obtained by gating and fusing the graph embedding vector and the semantic description embedding vector.

4. The method according to claim 3, characterized in that, The process of performing cross-modal fusion on the encoded data to obtain fused data includes: In the first stage of fusion, the sensor encoded data and the document encoded data are subjected to precision weighting and precision gating fusion to obtain the first fused data; In the second stage of fusion, the first fused data and the knowledge embedded data are subjected to precision weighting and precision gating fusion to obtain the final fused data.

5. The method according to claim 4, characterized in that, In the first stage of fusion, the sensor encoded data and the document encoded data are subjected to precision-weighted and precision-gated fusion to obtain the first fused data, including: Set the sensor mode as the first primary mode and the document mode as the first auxiliary mode; The first auxiliary mode mean is obtained by precision weighting. The first content data is calculated based on the mean of the first primary mode and the weighted mean of the first auxiliary mode; The first gated data is obtained based on the mean of the first primary mode, the scalar precision of the first primary mode, the mean of the first auxiliary mode, and the scalar precision of the first auxiliary mode. First fused data is obtained based on the first content data and the first gating data.

6. The method according to claim 5, characterized in that, The first content data is calculated based on the mean of the first primary mode and the weighted mean of the first auxiliary mode, including: The first attention head correlation value between the main mode and the auxiliary mode is calculated based on the first main mode mean and the first auxiliary mode weighted mean; Calculate the first multi-head cross-attention output value based on the first attention head correlation value; The first content data is calculated based on the first multi-head cross-attention output value.

7. The method according to claim 6, characterized in that, In the second stage of fusion, the first fused data and the knowledge embedded data are subjected to precision-weighted and precision-gated fusion to obtain the final fused data, including: The first-stage fusion output is set as the second primary modality, and the knowledge modality is set as the second auxiliary modality; The second principal mode mean, the second principal mode scalar precision, the second auxiliary mode mean, and the second auxiliary mode scalar precision are determined based on the first fused data and the knowledge embedded data. The second auxiliary mode mean is obtained by precision weighting. The second content data is calculated based on the mean of the second primary mode and the weighted mean of the second auxiliary mode; The second gating data is obtained based on the mean of the second primary mode, the scalar precision of the second primary mode, the mean of the second auxiliary mode, and the scalar precision of the second auxiliary mode; The final fused data is obtained based on the second content data and the second gating data.

8. The method according to claim 7, characterized in that, The step of determining the second primary mode mean, the second primary mode scalar precision, the second auxiliary mode mean, and the second auxiliary mode scalar precision based on the first fused data and the knowledge embedded data includes: The first fused data is determined as the mean of the second dominant mode; The arithmetic mean of the logarithmic variances of the first principal mode and the first auxiliary mode is determined as the logarithmic variance of the second principal mode, and the scalar accuracy of the second principal mode is calculated based on the logarithmic variance of the second principal mode. The knowledge-embedded data is copied along the time dimension and then used for time-series fusion to obtain the mean of the second auxiliary mode; The knowledge embedding data includes a fixed precision, and the precision of the second auxiliary modality is calculated based on the fixed precision.

9. The method according to claim 1, characterized in that, The process of obtaining classification results by performing evidence-based output and selective prediction on the original vector based on the Dirichlet distribution includes: Calculate the Dirichlet concentration parameters for each category based on the original vector; Calculate the total Dirichlet intensity, class prediction probability, and emptyness degree based on the concentration parameters; The abstention result is obtained by selectively predicting based on the emptyness degree and the preset abstention threshold. The predicted probability of the category, the degree of emptiness, and the result of abandonment are output as the classification result.

10. The method according to claim 9, characterized in that, The method further includes: Acquire sensor coding training data, document coding training data, and sampling interpolation coefficients; The mean representation and its log-variance in the training data are simultaneously interpolated using the sampling interpolation coefficients to obtain augmented sample data and corresponding augmented label data. The enhanced sample data is fused across modes to obtain fused enhanced data; The loss is calculated based on the fused enhanced data, the enhanced label data, and the sampling interpolation coefficients; The loss is used to optimize the training of the encoder and the learnable parameters in the fusion classification process.

Images