A multi-modal data fusion method, system and device applied to charging evaluation

Abstract

The application discloses a multi-modal data fusion method, system and device applied to charging evaluation, and is used for improving the precision of charging module health evaluation. The multi-modal data fusion method comprises the following steps: collecting multi-modal original data of a charging module; processing the multi-modal original data to construct a deep representation tensor; performing forward semantic correlation mapping on the deep representation tensor based on a bidirectional cross attention mechanism to generate a matrix in a first direction; performing reverse semantic backflow inference on the bidirectional cross attention mechanism to generate a matrix in a second direction; modulating and fusing the matrices in the first direction and the second direction according to a modal confidence factor to obtain an inter-modal correlation strength matrix; weighting the deep representation tensor based on the inter-modal correlation strength matrix, and aggregating by using a preset cross-modal sparse projection network to obtain a low-dimensional fusion semantic representation, which is simultaneously input into a preset charging health evaluation model for evaluation.

Description

Technical Field

[0001] This application relates to the field of charging technology, and in particular to a multimodal data fusion method, system and device for charging assessment. Background Technology

[0002] With the rapid development of new energy vehicles and energy storage systems, charging modules, as the main modules for energy transmission and storage, directly determine the overall safety and lifespan of the equipment due to their operational reliability. During long-term, high-frequency charging and discharging, charging modules are prone to performance degradation due to issues such as cell aging, circuit losses, and abnormal heat dissipation, which can even lead to safety accidents such as short circuits and fires. Therefore, accurate assessment of the health status of charging modules has become a primary industry requirement. To comprehensively reflect the health status of charging modules, the assessment process needs to integrate multimodal data. This multimodal data characterizes the operating status of the charging module from different dimensions. Only through effective integration can the one-sidedness of single-data assessment be avoided, providing a data basis for health status assessment.

[0003] In existing technologies, multimodal data is typically cleaned and then fused using either data-level stitching or feature-level overlay. Specifically, data-level stitching involves directly stitching the data from each modality according to timestamps or the order of acquisition into a high-dimensional multimodal data matrix. For example, parameters such as voltage, temperature, and vibration amplitude at a specific moment are combined into a feature vector. Feature-level overlay fusion first extracts basic features from individual modal data, then overlays these basic features to form a multimodal fusion feature set. Finally, the fused high-dimensional multimodal data matrix or multimodal fusion feature set is used as the data input for an evaluation model to assess the health status of the charging module.

[0004] However, whether data-level splicing or feature-level overlay is used for fusion, the fusion is only performed on the surface of the data. This will cause the evaluation model to fail to accurately capture the hidden features that reflect potential health risks during the evaluation, thereby reducing the accuracy and reliability of the health evaluation of the charging module. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a multimodal data fusion method, system, and apparatus for charging evaluation.

[0006] The technical solution provided in this application is described below: The first aspect of this application provides a multimodal data fusion method for charging evaluation, the multimodal data fusion method comprising: Collect multimodal raw data when the charging module is in operation. The multimodal raw data includes visual domain data, acoustic domain data, and electrical parameter domain data. Cross-domain semantic activation processing is performed on the original multimodal data to construct the corresponding deep representation tensor, which carries visual subdomain features, acoustic subdomain features and electrical parameter quantum domain features respectively. Based on a pre-defined bidirectional cross-attention mechanism, forward semantic association mapping is performed on the deep representation tensors of each subdomain feature to generate a cross-correlation weight matrix in the first direction. In the bidirectional cross-attention mechanism, reverse semantic backflow inference is further performed to construct an inverse correlation mapping for the deep representation tensors of each subdomain to generate a cross-correlation weight matrix in the second direction; Based on the preset modal confidence factor, the cross-correlation weight matrix of the first direction and the cross-correlation weight matrix of the second direction are modulated and fused to obtain the intermodal correlation strength matrix. The modal confidence factor is used to characterize the effective participation degree of the corresponding subdomain features under the condition of sudden interference. The deep representation tensor is weighted based on the intermodal correlation strength matrix, and the weighted deep representation tensor is mapped to a low-dimensional semantic subspace for aggregation using the adaptive sparse projection matrix in the preset cross-modal sparse projection network to obtain a low-dimensional fused semantic representation, which is then input into a preset charging health assessment model for evaluation.

[0007] Optionally, performing cross-domain semantic activation processing on the multimodal raw data to construct the corresponding deep representation tensor includes: The visible data is processed by multi-layer convolution through a pre-trained convolutional neural network to generate a first initial feature vector. The acoustic domain data is processed by time-frequency analysis using a pre-trained time-frequency convolutional network to generate a second initial feature vector. The electrical parameter domain data is processed by time-series modeling using a pre-trained long short-term memory network to generate a third initial feature vector; The first initial feature vector, the second initial feature vector, and the third initial feature vector are respectively input into a preset gated loop unit. Redundant information is filtered out by the forget gate mechanism of the preset gated loop unit, and key features are retained by the update gate mechanism of the preset gated loop unit to obtain the deep representation tensor.

[0008] Optionally, the step of performing multi-layer convolution processing on the visible data using a pre-trained convolutional neural network to generate a first initial feature vector includes: Brightness correction is performed on the visible field data, and low-level edge features, mid-level texture features, and high-level semantic features are extracted layer by layer from the convolutional layers of a pre-trained convolutional neural network. The pooling layers of the pre-trained convolutional neural network are used to reduce the dimensionality of the low-level edge features, the mid-level texture features, and the high-level semantic features, respectively. The dimensionality-reduced low-level edge features, mid-level texture features, and high-level semantic features are mapped to a first initial feature vector through the fully connected layer of the pre-trained convolutional neural network.

[0009] Optionally, before modulating and fusing the cross-correlation weight matrices of the first and second directions according to a preset modal confidence factor, the multimodal data fusion method further includes: The preset modal confidence factor is dynamically adjusted based on the visual subdomain features, the acoustic subdomain features, and the electrical parameter quantum domain features. The dynamic adjustment of the preset modal confidence factor based on the visible subdomain features, the acoustic subdomain features, and the electrical parameter quantum domain features includes: Real-time recording of the signal-to-noise ratio fluctuation data of the visible subdomain features, the acoustic subdomain features, and the electrical parameter quantum domain features within a preset time window; The stability index of each subdomain feature is calculated based on the signal-to-noise ratio fluctuation data, and the preset modal confidence factor is dynamically adjusted based on the stability index.

[0010] Optionally, dynamically adjusting the preset modal confidence factor based on the stability index includes: When the stability index is greater than or equal to a preset first threshold, the preset modal confidence factor is reduced based on a preset first attenuation coefficient. When the stability index is less than the preset first threshold and greater than or equal to the preset second threshold, the preset modal confidence factor is reduced based on the preset second attenuation coefficient. When the stability index is less than the preset second threshold, the preset modal confidence factor is increased based on the preset recovery rate until the preset modal confidence factor reaches the preset benchmark value.

[0011] Optionally, the step of modulating and fusing the cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction according to a preset modal confidence factor to obtain the intermodal correlation strength matrix includes: The cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction are standardized to obtain the forward correlation matrix and the backward correlation matrix. A confidence modulation matrix is ​​constructed based on a preset modal confidence factor, and the forward correlation matrix and the reverse correlation matrix are bidirectionally fused with the confidence modulation matrix to obtain an initial fusion matrix; The initial fusion matrix is ​​subjected to eigenvalue decomposition to obtain the principal feature components; Based on the principal feature components, construct the intermodal correlation strength matrix.

[0012] Optionally, the standardization of the cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction to obtain the forward correlation matrix and the backward correlation matrix includes: Calculate the L2 norm of each row of the weight vector in the cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction, respectively; Regularize each element in each row of the weight vector based on the L2 norm; Based on the Softmax function and combined with a preset temperature coefficient, the cross-correlation weight matrix of the first direction and the cross-correlation weight matrix of the second direction after processing are transformed by probability distribution. The transformed cross-correlation weight matrix of the first direction is used as the forward correlation matrix, and the transformed cross-correlation weight matrix of the second direction is used as the backward correlation matrix.

[0013] A second aspect of this application provides a multimodal data fusion system for charging assessment, the multimodal data fusion system comprising: The acquisition unit is used to acquire multimodal raw data when the charging module is in operation. The multimodal raw data includes visual domain data, acoustic domain data, and electrical parameter domain data. The construction unit is used to perform cross-domain semantic activation processing on the multimodal raw data to construct the corresponding deep representation tensor, wherein the deep representation tensor carries visual subdomain features, acoustic subdomain features and electrical parameter quantum domain features respectively; The first generation unit is used to perform forward semantic association mapping on the deep representation tensors of each subdomain feature based on a preset bidirectional cross-attention mechanism, so as to generate a cross-correlation weight matrix in the first direction. The second generation unit is used to further perform reverse semantic backflow inference in the bidirectional cross-attention mechanism, and construct reverse correlation mapping for the deep representation tensors of each subdomain to generate a cross-correlation weight matrix in the second direction. The fusion unit is used to modulate and fuse the cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction according to the preset modal confidence factor to obtain the intermodal correlation strength matrix. The modal confidence factor is used to characterize the effective participation degree of the corresponding subdomain features under the condition of sudden interference. The evaluation unit is used to weight the deep representation tensor based on the intermodal correlation strength matrix, and to map the weighted deep representation tensor to a low-dimensional semantic subspace for aggregation using the adaptive sparse projection matrix in the preset cross-modal sparse projection network to obtain a low-dimensional fused semantic representation, which is then input into the preset charging health evaluation model for evaluation.

[0014] Optionally, the building unit is specifically used for: The visible data is processed by multi-layer convolution through a pre-trained convolutional neural network to generate a first initial feature vector. The acoustic domain data is processed by time-frequency analysis using a pre-trained time-frequency convolutional network to generate a second initial feature vector. The electrical parameter domain data is processed by time-series modeling using a pre-trained long short-term memory network to generate a third initial feature vector; The first initial feature vector, the second initial feature vector, and the third initial feature vector are respectively input into a preset gated loop unit. Redundant information is filtered out by the forget gate mechanism of the preset gated loop unit, and key features are retained by the update gate mechanism of the preset gated loop unit to obtain the deep representation tensor.

[0015] Optionally, the building unit is specifically used for: Brightness correction is performed on the visible field data, and low-level edge features, mid-level texture features, and high-level semantic features are extracted layer by layer from the convolutional layers of a pre-trained convolutional neural network. The pooling layers of the pre-trained convolutional neural network are used to reduce the dimensionality of the low-level edge features, the mid-level texture features, and the high-level semantic features, respectively. The dimensionality-reduced low-level edge features, mid-level texture features, and high-level semantic features are mapped to a first initial feature vector through the fully connected layer of the pre-trained convolutional neural network.

[0016] Optionally, an adjustment unit may also be included, specifically for: The preset modal confidence factor is dynamically adjusted based on the visual subdomain features, the acoustic subdomain features, and the electrical parameter quantum domain features. The dynamic adjustment of the preset modal confidence factor based on the visible subdomain features, the acoustic subdomain features, and the electrical parameter quantum domain features includes: Real-time recording of the signal-to-noise ratio fluctuation data of the visible subdomain features, the acoustic subdomain features, and the electrical parameter quantum domain features within a preset time window; The stability index of each subdomain feature is calculated based on the signal-to-noise ratio fluctuation data, and the preset modal confidence factor is dynamically adjusted based on the stability index.

[0017] Optionally, the adjustment unit is specifically used for: When the stability index is greater than or equal to a preset first threshold, the preset modal confidence factor is reduced based on a preset first attenuation coefficient. When the stability index is less than the preset first threshold and greater than or equal to the preset second threshold, the preset modal confidence factor is reduced based on the preset second attenuation coefficient. When the stability index is less than the preset second threshold, the preset modal confidence factor is increased based on the preset recovery rate until the preset modal confidence factor reaches the preset benchmark value.

[0018] Optionally, the fusion unit is specifically used for: The cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction are standardized to obtain the forward correlation matrix and the backward correlation matrix. A confidence modulation matrix is ​​constructed based on a preset modal confidence factor, and the forward correlation matrix and the reverse correlation matrix are bidirectionally fused with the confidence modulation matrix to obtain an initial fusion matrix; The initial fusion matrix is ​​subjected to eigenvalue decomposition to obtain the principal feature components; Based on the principal feature components, construct the intermodal correlation strength matrix.

[0019] Optionally, the fusion unit is specifically used for: Calculate the L2 norm of each row of the weight vector in the cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction, respectively; Regularize each element in each row of the weight vector based on the L2 norm; Based on the Softmax function and combined with a preset temperature coefficient, the cross-correlation weight matrix of the first direction and the cross-correlation weight matrix of the second direction after processing are transformed by probability distribution. The transformed cross-correlation weight matrix of the first direction is used as the forward correlation matrix, and the transformed cross-correlation weight matrix of the second direction is used as the backward correlation matrix.

[0020] A third aspect of this application provides a multimodal data fusion device for charging evaluation, the multimodal data fusion device comprising: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor invokes to execute the first aspect and any optional multimodal data fusion method within the first aspect.

[0021] The fourth aspect of this application provides a computer-readable storage medium storing a program that, when executed on a computer, performs the first aspect and any optional multimodal data fusion method of the first aspect.

[0022] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: 1. Collect multimodal raw data in the visual domain, acoustic domain, and electrical parameter domain, covering all dimensions of the charging module's operation. Then, perform cross-domain semantic activation processing on the multimodal raw data to construct a deep representation tensor carrying features of each subdomain, breaking the limitations of shallow data processing in existing technologies. 2. Based on the preset bidirectional cross-attention mechanism, the deep representation tensor carrying the features of each subdomain is generated into cross-correlation weight matrices in the first and second directions respectively through forward semantic association mapping and reverse semantic backflow inference, so as to fully capture the implicit associations between modes. 3. By modulating and fusing the cross-correlation weight matrices of the first and second directions through a preset modal confidence factor, the intermodal correlation strength matrix is ​​obtained. This can filter out effective features under interference and reduce the impact of interference on the fusion effect. 4. Based on the intermodal correlation strength matrix and the pre-set cross-modal sparse projection network, the deep representation tensor is aggregated into a low-dimensional fused semantic representation, which simplifies the data dimension while further preserving key semantic information.

[0023] This method of evaluating charging modules through the synergistic effects of multimodal data, cross-domain semantic activation, bidirectional cross-attention mechanism, modal confidence factor, and cross-modal sparse projection network can overcome the limitations of shallow fusion in existing technologies, accurately uncover implicit correlations between modalities, screen effective features under interference, significantly improve the accuracy and reliability of charging module health assessment, and effectively ensure the safe operation of the equipment. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A schematic flowchart of an embodiment of the multimodal data fusion method provided in this application; Figure 2A schematic flowchart illustrating one implementation of step S102 in the multimodal data fusion method provided in this application; Figure 3 A schematic flowchart illustrating one implementation of step S105 in the multimodal data fusion method provided in this application; Figure 4 A schematic flowchart of an embodiment of the multimodal data fusion method provided in this application for dynamically adjusting the preset modality confidence factor; Figure 5 A schematic diagram of an embodiment of the multimodal data fusion system provided in this application; Figure 6 This is a schematic diagram of an embodiment of the multimodal data fusion device provided in this application. Detailed Implementation

[0026] In the multimodal data fusion method provided in this application embodiment, the subject executing the multimodal data fusion method is not specifically limited. This multimodal data fusion method can be implemented by any device, system, or platform with data acquisition, communication, and computing capabilities, including but not limited to smart charging piles, on-board chargers, battery management systems (BMS), edge computing nodes, cloud servers, independent monitoring instruments, or general-purpose computing devices.

[0027] The implementation of multimodal data fusion methods can be accomplished through embedded programs, software modules, hardware circuits, or combinations thereof. Regardless of whether the executing entity is a single device, a distributed cluster, or a virtualized service, as long as it can complete multimodal data acquisition, cross-domain semantic activation, bidirectional attention calculation, confidence modulation fusion, and health assessment according to the steps of the embodiment, it is applicable to multimodal data fusion methods.

[0028] In the aforementioned application scenarios, the operational status assessment of the charging module relies on real-time information acquired from multi-source heterogeneous sensors, and its health is jointly characterized by the fusion of multi-modal data. Due to the complex and variable nature of the actual operating environment, such as sudden changes in illumination, background noise interference, instantaneous sensor failures, or electrical noise fluctuations, data from a single modality may contain noise, lack information, or even be contradictory. This can lead to deviations, delays, or decreased confidence in assessment results based on a single or simply pieced-together multi-source data, directly affecting the timeliness of charging safety warnings, the accuracy of fault diagnosis, and the reliability of health status prediction.

[0029] This application's multimodal data fusion method achieves robust alignment, complementary enhancement, and efficient fusion of multimodal information through a series of collaborative techniques, including constructing deep representations by performing cross-domain semantic activation, generating bidirectional cross-correlation weights based on a bidirectional cross-attention mechanism, performing modulation fusion based on dynamic confidence factors, and utilizing adaptive sparse projection to achieve low-dimensional aggregation. This method enables accurate perception of operating status and early anomaly identification in complex charging evaluation scenarios. Even if the quality of some modal data temporarily deteriorates, reliable inferences can still be made based on high-confidence modalities, thereby improving the overall anti-interference capability, diagnostic accuracy, and stability in variable environments. For example, in the fault warning scenario of DC fast charging piles for electric vehicles, when a sudden strong background noise severely interferes with the acoustic modes, this application's multimodal data fusion method can dynamically reduce the confidence weight of acoustic features based on the real-time signal-to-noise ratio, while strengthening the correlation between high-definition visual features and stable electrical parameter features. This accurately identifies the potential risk of "overheating of connection terminals" and triggers a precise warning, avoiding missed or false alarms due to information conflicts or omissions.

[0030] To facilitate accurate understanding of this application by those skilled in the art, the relevant terms or concepts involved in the embodiments of this application are explained as follows: Charging module: refers to the core physical unit or subsystem that performs functions such as power conversion, transmission, management, or storage during the charging process. Specific forms include charging piles for electric vehicles, on-board chargers (OBC), battery packs and their management systems (BMS), charging circuits in power storage converters (PCS), or wireless charging transmitter / receiver modules for consumer electronic devices.

[0031] Multimodal raw data: refers to a heterogeneous set of raw data collected from different physical domains or sensing channels that describes the state of the charging module at the same operating moment or time period.

[0032] Deep representation tensors are multidimensional data arrays that represent high-level semantic features of various modalities after cross-domain semantic activation processing. Each tensor corresponds to a modality, and its elements are learned through a deep learning model, containing information meaningful for determining the charging state.

[0033] Cross-correlation weight matrix: A matrix calculated through an attention mechanism, reflecting the strength or importance of the correlation between specific feature pairs from different modalities. The matrices in the first and second directions capture the inter-modal dependencies from different perspectives.

[0034] Cross-modal sparse projection networks: a special type of neural network layer or structure, which is a learnable "adaptive sparse projection matrix". This adaptive sparse projection matrix has sparsity constraints, which efficiently projects high-dimensional, weighted multimodal fusion features into a lower-dimensional semantic subspace. Aggregation is performed within the semantic subspace to remove redundancy and retain the most complementary and discriminative information, forming a compact low-dimensional fusion semantic representation.

[0035] Please see Figure 1 This application first provides an embodiment of a multimodal data fusion method for charging evaluation, which includes: S101. Collect multimodal raw data when the charging module is in operation. The multimodal raw data includes visual domain data, acoustic domain data, and electrical parameter domain data. In this embodiment, multimodal raw data of the charging module during operation are collected, specifically including visual domain data, acoustic domain data, and electrical parameter domain data. These three types of data comprehensively cover all dimensions of the charging module during operation.

[0036] Specifically, high-definition industrial cameras can be used to collect data in the visible field, capturing the appearance of the charging module in real time, such as the temperature distribution of the heat sink, whether the interface is loose, and the status of the indicator lights. High-sensitivity microphones can be used to collect data in the acoustic field, recording audio information such as fan rotation noise, current noise, and abnormal noises generated during charging. Voltage sensors, current sensors, or power sensors can be used to collect data in the electrical parameter field, such as charging voltage fluctuations, operating current stability, input and output power changes, and power factor.

[0037] It is important to note that during the data collection process, the sampling frequency should be set according to actual needs to synchronously record these three types of data, so as to ensure that the data timestamps are consistent and form a multimodal raw dataset, providing complete data for subsequent processing.

[0038] S102. Perform cross-domain semantic activation processing on the multimodal raw data to construct the corresponding deep representation tensor. The deep representation tensor carries visual subdomain features, acoustic subdomain features and electrical parameter quantum domain features respectively. In this embodiment, cross-domain semantic activation processing needs to be performed on the collected multimodal raw data to construct a deep representation tensor carrying features of each subdomain. Cross-domain semantic activation processing can use feature association algorithms to uncover potential semantic relationships between features of different domains. Subsequently, based on these multimodal raw data that have undergone cross-domain semantic activation processing, a corresponding deep representation tensor is constructed. The deep representation tensor simultaneously retains features of the visual subdomain, acoustic subdomain, and electrical parameter quantum domain.

[0039] See Figure 2The following provides a specific implementation method for step S102, which includes: S1021. Perform brightness correction on the visible field data, and extract low-level edge features, mid-level texture features and high-level semantic features from the corrected visible field data layer by layer through the convolutional layers of a pre-trained convolutional neural network. Brightness correction is performed on the visible field data in the multimodal raw data. Brightness correction can mainly eliminate the problem of brightness imbalance caused by differences in lighting conditions in the visible field data. For example, in a monitoring scene, there is a significant difference in brightness between images of the same object taken on cloudy and sunny days. Brightness correction can make the brightness levels of the two types of images more consistent, providing a unified basis for subsequent feature extraction.

[0040] While performing brightness correction, features are extracted layer by layer through the convolutional layers of a pre-trained convolutional neural network. Specifically, shallow convolutional layers capture low-level edge features in the visible data, such as the outlines and corners of objects; mid-level convolutional layers integrate edge information based on shallow features to extract mid-level texture features, such as stripes and spots on the object's surface; deep convolutional layers further abstract and fuse mid-level texture features to extract high-level semantic features, such as recognizing the semantic concept of "car" from an image containing vehicles. This layered extraction approach avoids the problems of focusing only on details and losing semantics or focusing only on semantics and ignoring details, and improves feature extraction efficiency through pre-trained convolutional neural networks, providing a high-quality visible feature foundation for constructing accurate deep representation tensors.

[0041] S1022. Dimensionality reduction is performed on low-level edge features, mid-level texture features, and high-level semantic features through the pooling layers of a pre-trained convolutional neural network. The obtained low-level edge features, mid-level texture features, and high-level semantic features are respectively input into the pooling layer of the pre-trained convolutional neural network. The pooling layer performs dimensionality reduction by sampling local regions of these features using built-in sampling rules. The sampling method can be max pooling or average pooling. The pooling layer divides these features into multiple non-overlapping local regions, and only the maximum value of each region is retained as the feature representative of that region. When average pooling is used, the average value of each local region is calculated as the feature representative.

[0042] This dimensionality reduction is performed because the low-level edge features, mid-level texture features, and high-level semantic features extracted by the convolutional layer contain a lot of redundant information and repeated parameters. For example, adjacent regions in low-level edge features may have similar edge information. The pooling layer can reduce the size and number of parameters of the feature map while retaining key feature information through local sampling, thereby achieving dimensionality reduction.

[0043] S1023. The low-level edge features, mid-level texture features, and high-level semantic features after dimensionality reduction are mapped to the first initial feature vector through the fully connected layer of the pre-trained convolutional neural network.

[0044] Since the features extracted by convolutional and pooling layers are mostly local or hierarchical features, after dimensionality reduction, fully connected layers are needed to break the locality limitation of these features, associate features from different levels and different localities, and form a vector that can represent the overall features of the visible data.

[0045] Specifically, the dimensionality-reduced low-level edge features, mid-level texture features, and high-level semantic features are sequentially concatenated to form an integrated visual feature matrix. This matrix includes comprehensive visual information from details to semantics. Subsequently, the visual feature matrix is ​​input into the fully connected layer of a pre-trained convolutional neural network. Each neuron in the fully connected layer is connected to all neurons in the visual feature matrix. Through built-in weight parameters, the visual feature matrix undergoes linear transformation and non-linear activation processing, converting the two-dimensional matrix into a one-dimensional vector form, i.e., the first initial feature vector.

[0046] S1024. Perform time-frequency analysis on the acoustic domain data using a pre-trained time-frequency convolutional network to generate a second initial feature vector; For acoustic domain data in multimodal raw data, since these acoustic domain data are often distributed in both time and frequency dimensions, traditional time-domain or frequency-domain analysis alone is difficult to capture completely. Therefore, time-frequency analysis is mainly performed through pre-trained time-frequency convolutional networks. Time-frequency convolutional networks combine the completeness of time-frequency analysis with the feature extraction capabilities of convolutional networks. Moreover, the pre-trained network has learned general acoustic feature patterns, which can efficiently process different types of acoustic data.

[0047] Specifically, the time-frequency convolutional network receives acoustic domain data and performs time-frequency analysis on it. Time-frequency analysis transforms the acoustic signals in the acoustic domain data from the traditional time domain to the time-frequency domain, simultaneously revealing the frequency distribution characteristics of the signal at different time points. For example, converting a speech signal into a spectrogram clearly shows the pitch changes of the speech at different times. Subsequently, the convolutional layers in the time-frequency convolutional network extract features from the time-frequency features obtained from the time-frequency analysis. After extraction, the extracted time-frequency features are mapped into a one-dimensional second initial feature vector through fully connected layers in the pre-trained time-frequency convolutional network.

[0048] S1025. The electrical parameter domain data is processed by time series modeling through a pre-trained long short-term memory network to generate a third initial feature vector; Because electrical parametric domain data has significant temporal correlations, traditional neural networks struggle to capture long-term temporal dependencies. Pre-trained long short-term memory networks have learned general temporal modeling rules and can efficiently process different types of electrical parametric time-series data. Therefore, for electrical parametric domain data in multimodal raw data, temporal modeling processing is performed through pre-trained long short-term memory networks.

[0049] Specifically, the electrical parameter domain data is first organized into a time-series data sequence according to time order. For example, the current data of a certain industrial equipment in a day is divided into minutes to form [I1, I2, ..., I 1440 The time series data sequence is then input into a pre-trained Long Short-Term Memory (LSTM) network. The pre-trained LSM network processes the time series data through built-in memory units, input gates, forget gates, and output gates.

[0050] Furthermore, the forget gate determines whether to forget the historical information in the memory unit from the previous time step based on the current input and the hidden state from the previous time step. For example, when the device current is in a stable phase, the forget gate will retain the memory information of the previously stable current. The input gate controls the update intensity of the input data at the current time step. For example, when the current suddenly increases when the device starts up, the input gate will increase the update weight of the current data at that time step and store it in the memory unit. The output gate generates the hidden state at the current time step based on the memory unit and the current hidden state. Through multiple time steps of iteration, the entire electrical parameter time series is modeled, and the time-series correlation features are extracted. After extraction, the time-series correlation features are mapped to a one-dimensional third initial feature vector through the fully connected layer of a pre-trained Long Short-Term Memory network.

[0051] S1026. Input the first initial feature vector, the second initial feature vector, and the third initial feature vector into the preset gated loop unit respectively. Use the forget gate mechanism of the preset gated loop unit to filter out redundant information and use the update gate mechanism of the preset gated loop unit to retain key features, thereby obtaining the deep representation tensor.

[0052] The first, second, and third initial feature vectors contain a large amount of redundant information that is irrelevant to the fusion target. Direct fusion will lead to a decrease in tensor quality. Therefore, a dual-gating mechanism with a pre-defined gated loop unit is used to retain effective information and remove invalid information.

[0053] Specifically, the generated first, second, and third initial feature vectors are input into a preset gating loop unit. The preset gating loop unit processes the three types of feature vectors through built-in forget and update gate mechanisms.

[0054] The forget gate mechanism first evaluates the information in each feature vector, determining which information is redundant and filtering it out. For example, in a smart home appliance monitoring scenario, the forget gate filters out background features of the wall, subtle wind noise in the environment, and instantaneous current spikes in the appliance image. The update gate mechanism, on the other hand, combines the current input feature vector with its hidden state to evaluate and retain key features valuable for multimodal fusion, such as the core appearance features of the appliance, normal noise features during operation, and current features during stable operation. After the redundancy filtering by the forget gate and the key retention by the update gate, a pre-gated gated loop unit integrates and optimizes the three types of feature vectors, outputting a deep representation tensor containing key information from the three modalities.

[0055] S103. Based on the preset bidirectional cross-attention mechanism, forward semantic association mapping is performed on the deep representation tensors of each subdomain feature to generate the cross-correlation weight matrix in the first direction. In this embodiment, after constructing the corresponding deep representation tensor, a forward semantic association mapping is performed on the visual subdomain features, acoustic subdomain features, and electrical parameter quantum domain features in the deep representation tensor based on a preset bidirectional cross-attention mechanism to generate a cross-correlation weight matrix in the first direction.

[0056] Specifically, taking one of the subdomain features as the reference domain, the semantic association strength between the reference domain and the other two target subdomain features is calculated. For example, taking the electronically parametric quantum domain feature as the reference, the degree of association between the visual subdomain feature and the electronically parametric quantum domain feature, as well as the degree of association between the acoustic subdomain feature and the electronically parametric quantum domain feature, is measured through the attention weight calculation in the preset bidirectional cross-attention mechanism.

[0057] During the mapping process, weights are assigned through the attention network in the pre-defined bidirectional cross-attention mechanism. Feature pairs with high correlation receive higher weights, while feature pairs with low correlation receive lower weights. Semantic association mapping is performed sequentially with visual subdomain features and acoustic subdomain features, respectively, with the other two subdomains. The results of the three mappings are integrated to generate a cross-correlation weight matrix in the first direction. The matrix elements can intuitively reflect the positive correlation strength between features in different subdomains.

[0058] S104. Further perform reverse semantic backflow inference in the bidirectional cross-attention mechanism, construct reverse correlation mapping for the deep representation tensors of each subdomain, and generate the cross-correlation weight matrix in the second direction. In this embodiment, after generating the cross-correlation weight matrix in the first direction, reverse semantic backflow inference is performed in the bidirectional cross-attention mechanism to construct an inverse correlation mapping for the deep representation tensors of the visual subdomain features, acoustic subdomain features, and electrical parameter quantum domain, thereby generating the cross-correlation weight matrix in the second direction.

[0059] The reverse semantic backflow inference follows the opposite logic to the forward semantic association mapping, which generates the first-direction cross-correlation weight matrix. The forward semantic association mapping uses the target subdomain in the forward mapping as a baseline to deduce its association strength with the original baseline domain, forming a reverse verification. However, in the reverse semantic backflow inference process, the reliability of the forward semantic association mapping is primarily verified. If a feature pair has a high weight in the forward semantic association mapping but an extremely low weight in the reverse semantic backflow inference, the association is flagged as potentially false; if both weights are high, the association is confirmed to be genuine. All reverse semantic backflow inference results are integrated to generate the second-direction cross-correlation weight matrix, which complements the first-direction matrix.

[0060] S105. Modulate and fuse the cross-correlation weight matrix of the first direction and the cross-correlation weight matrix of the second direction according to the preset modal confidence factor to obtain the intermodal correlation strength matrix. The modal confidence factor is used to characterize the effective participation of the corresponding subdomain features under the condition of sudden interference. The preset modal confidence factors are used to characterize the effective participation of each subdomain feature under sudden interference conditions. For example, the confidence factor decreases when the visual domain is affected by illumination interference, the confidence factor decreases when the acoustic domain is affected by environmental noise interference, and the confidence factor decreases when the electrical parameter domain is affected by electromagnetic interference. These confidence factors are mainly determined through training with historical interference data.

[0061] Specifically, the system first retrieves preset modal confidence factors corresponding to the visible subdomain, acoustic subdomain, and electrical parameter quantum domain. For example, if a subdomain has no interference, its modal confidence factor is set to 1; if a subdomain has slight interference, its modal confidence factor is set to 0.8; and if a subdomain has severe interference, its modal confidence factor is set to 0.3. Then, the cross-correlation weight matrix in the first and second directions is multiplied by the corresponding modal confidence factors of the subdomains to attenuate and modulate the correlation weights of the interfered subdomains. The modulated weight matrices in the two directions are then fused to generate an intermodal correlation strength matrix. This generated intermodal correlation strength matrix retains the true correlations of two-way verification while filtering out invalid correlations caused by interference.

[0062] See Figure 3 The following provides a specific implementation method for step S105, which includes: S1051. Calculate the L2 norm of each row of the weight vector in the cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction, respectively. Since the L2 norm can quantify the overall strength of each row of weight vectors and reflect the comprehensive correlation level between the subdomain and other subdomains, this embodiment mainly calculates the L2 norm of each row of weight vectors in the first direction cross-correlation weight matrix and the second direction cross-correlation weight matrix respectively.

[0063] Specifically, in the first-direction cross-correlation weight matrix and the second-direction cross-correlation weight matrix, each row corresponds to a subdomain feature, and the elements in each row represent the correlation weights between that subdomain and other subdomain features. For each row of each matrix, the L2 norm of all elements in that row is calculated sequentially. For example, in the electrical parameter quantum domain row of the first-direction cross-correlation weight matrix, the elements in the electrical parameter quantum domain row are the correlation weights for "electrical parameter-visual" and "electrical parameter-acoustic". According to the L2 norm calculation rules, the squares of each element in this row are summed, and the square root of the sum is taken to obtain the L2 norm of that row. This ensures that the correlation weights of different subdomains are processed at the same scale, avoiding fusion bias caused by differences in matrix size.

[0064] S1052. Regularize each element in each row of the weight vector based on the L2 norm; Regularization can eliminate the size difference of the weight vector in each row, so that the weight vectors in different rows are in the same numerical range. The size of each element in a row only represents the relative importance of the association in that row, rather than the absolute strength.

[0065] Specifically, after calculating the L2 norm of each row of weight vectors, for each row of the cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction, each element in that row is divided by the corresponding L2 norm of that row, so that the L2 norm of the weight vector in that row becomes 1 after processing. For example, in the cross-correlation weight matrix in the first direction, the original elements of the electric parameter quantum domain row are [0.8, 0.6], and the L2 norm is 1. After regularization, the elements are still [0.8, 0.6]; the original elements of the acoustic subdomain row are [0.3, 0.4], and the L2 norm is 0.5. After regularization, the elements become [0.3 / 0.5, 0.4 / 0.5] = [0.6, 0.8].

[0066] S1053. Based on the Softmax function and combined with a preset temperature coefficient, the cross-correlation weight matrix of the first direction and the cross-correlation weight matrix of the second direction after processing are transformed by probability distribution. The transformed cross-correlation weight matrix of the first direction is used as the forward correlation matrix, and the transformed cross-correlation weight matrix of the second direction is used as the backward correlation matrix.

[0067] Based on the Softmax function and a preset temperature coefficient, the probability distribution transformation is performed on the two cross-correlation weight matrices after processing. The probability distribution transformation makes the correlation weights interpretable and flexible, solving the problem that traditional weights can only reflect intensity and cannot reflect relative proportion.

[0068] The Softmax function is applied separately to each row of the cross-correlation weight matrix in the first and second directions after processing. Further, the difference between each element in the row is amplified by exponential operation, and then divided by the sum of the exponential results of all elements in the row, so that the sum of the elements in the row after transformation is 1, forming a probability distribution. Simultaneously, a preset temperature coefficient is introduced to adjust the smoothness of the distribution; the smaller the temperature coefficient, the steeper the probability distribution; the larger the temperature coefficient, the smoother the distribution. For example, in charging evaluation, if a row of elements is [0.8, 0.6] after regularization, and the temperature coefficient is set to 0.5, the elements after Softmax transformation are [0.65, 0.35]; if the temperature coefficient is set to 2, the transformed elements are [0.55, 0.45].

[0069] After the transformation, the cross-correlation weight matrix in the first direction is used as the forward correlation matrix, and the cross-correlation weight matrix in the second direction is used as the backward correlation matrix.

[0070] S1054. Construct a confidence modulation matrix based on the preset modal confidence factor, and then perform bidirectional fusion of the forward correlation matrix and the reverse correlation matrix with the confidence modulation matrix to obtain an initial fusion matrix; The modal confidence factor is predetermined based on the actual interference situation of each subdomain. For example, the visual subdomain is less affected by illumination interference, so the modal confidence factor is set to 1; the acoustic subdomain is greatly affected by noise interference, so the modal confidence factor is set to 0.3; and the electric parameter quantum domain has no electromagnetic interference, so the modal confidence factor is set to 1.

[0071] After pre-determining the modal confidence factors, a confidence modulation matrix with the same dimension as the correlation matrix is ​​constructed based on these factors. In this confidence modulation matrix, each row / column element corresponding to a subdomain represents the confidence factor for that subdomain. Simultaneously, each element in the forward correlation matrix is ​​multiplied by the corresponding factor in the modulation matrix, and the same applies to the backward correlation matrix. Finally, the modulated forward and backward correlation matrices are weighted and summed to obtain the initial fusion matrix. This fusion method is both interference-resistant and allows for two-way verification, thus improving the reliability of the correlation information.

[0072] S1055. Perform eigenvalue decomposition on the initial fusion matrix to obtain the principal feature components; Eigenvalue decomposition of the initial fusion matrix can condense the high-dimensional structural information in the initial fusion matrix into a few principal feature components, and eliminate secondary components corresponding to redundancy and noise.

[0073] Specifically, eigenvalue decomposition is performed on the initial fusion matrix to obtain its principal eigenvalues. Each principal eigenvalue consists of eigenvalues ​​and their corresponding eigenvectors. The eigenvalues ​​represent the explanatory power of the eigenvectors for the variations in the initial fusion matrix; the larger the eigenvalue, the more information the corresponding eigenvector contains. Subsequently, all eigenvalues ​​are sorted in descending order, and the k largest eigenvalues ​​are selected as the principal eigenvalues ​​of the initial fusion matrix. The value of k is determined primarily based on the dimensionality of the initial fusion matrix and the information retention requirements.

[0074] S1056. Construct the intermodal correlation strength matrix based on the principal feature components.

[0075] Based on the magnitude of the eigenvalues ​​corresponding to the principal feature components, a weight is assigned to each principal feature component. The larger the eigenvalue, the higher the weight, which means that the principal feature component has a higher contribution to the initial fusion matrix. For example, the eigenvalue of principal feature component 1 is 4.5, and the weight is set to 4.5 / (4.5+2.1)=0.68; the eigenvalue of principal feature component 2 is 2.1, and the weight is set to 2.1 / (4.5+2.1)=0.32.

[0076] After assigning weights, each principal feature component is scaled according to the weights, and then the scaled principal feature components are concatenated by column or row to form an intermodal correlation strength matrix with the same dimension as the initial fusion matrix. Each element in the intermodal correlation strength matrix represents the final correlation strength between two subdomain features.

[0077] S106. The deep representation tensor is weighted based on the intermodal correlation strength matrix, and the weighted deep representation tensor is mapped to a low-dimensional semantic subspace for aggregation using the adaptive sparse projection matrix in the preset cross-modal sparse projection network to obtain a low-dimensional fused semantic representation, which is then input into the preset charging health assessment model for evaluation.

[0078] In this embodiment, based on the weight distribution of the intermodal correlation strength matrix, the visible subdomain features, acoustic subdomain features, and electrical parameter quantum domain features in the deep representation tensor are weighted. Features with high correlation strength are given higher weights, while features with low correlation strength and interference are given lower weights, thus highlighting key and effective features.

[0079] While performing weighting, the weighted deep representation tensor is input into a pre-defined cross-modal sparse projection network. The adaptive sparse projection matrix in this network, through learning and training, automatically selects the most discriminative core features, eliminating redundant and interfering features, thus mapping the high-dimensional deep representation tensor to a low-dimensional semantic subspace. During the mapping process, sparsity constraints ensure that the low-dimensional vector retains only key semantic information, forming a low-dimensional fused semantic representation. This low-dimensional fused semantic representation is then input into a pre-defined charging health assessment model to comprehensively evaluate the aging degree, failure risk, and operational stability of the charging module.

[0080] Before step S105 above, it is also necessary to dynamically adjust the preset modal confidence factor. Please refer to [link / reference needed]. Figure 4 , Figure 4 An embodiment of the multimodal data fusion method provided in this application, which dynamically adjusts the preset modality confidence factor, includes: S201. Real-time recording of signal-to-noise ratio fluctuation data of visible subdomain features, acoustic subdomain features, and electrical parameter quantum domain features within a preset time window; In this embodiment, the signal-to-noise ratio (SNR) fluctuation data of the visible subdomain features, acoustic subdomain features, and electrical parameter quantum domain features are recorded in real time within a preset time window. The SNR fluctuation data provides a basis for subsequent stability analysis.

[0081] Specifically, based on the real-time requirements of the charging module, a reasonable preset time window is first set, such as 10 seconds. This not only balances the timeliness and representativeness of the data but also avoids large data fluctuations due to an excessively short window and delayed data due to an excessively long window. Subsequently, the calculation logic for signal-to-noise ratio fluctuation data is defined for the visual subdomain features, acoustic subdomain features, and electrical parameter quantum domain features, respectively.

[0082] Furthermore, in the visual subdomain features, the signal is the intensity value of the effective features of the image, and the noise is the intensity value of invalid pixels caused by illumination interference and lens smudges. The ratio of the two is the visual domain signal-to-noise ratio. In the acoustic subdomain features, the signal is the intensity of the normal operation sound of the charging module, and the noise is the intensity of the ambient noise. The ratio of the two is calculated to obtain the acoustic domain signal-to-noise ratio. In the electrical parameter domain, the signal is the stable standard voltage / current parameter value, and the noise is the parameter fluctuation value caused by electromagnetic interference. The ratio is the electrical parameter domain signal-to-noise ratio.

[0083] Within each time window, signal-to-noise ratio (SNR) data for each subdomain feature is collected once per second. The maximum, minimum, average, and fluctuation amplitude of the SNR within the window are recorded synchronously to form complete SNR fluctuation data.

[0084] S202. Calculate the stability index of each subdomain feature based on the signal-to-noise ratio fluctuation data; The stability index of a subdomain feature is calculated based on the ratio of the fluctuation amplitude to the average value in the signal-to-noise ratio (SNR) fluctuation data within the time window. For each subdomain feature, the fluctuation amplitude and average value of the corresponding window are extracted from the SNR fluctuation data. Dividing the fluctuation amplitude by the average value yields the stability index of that subdomain feature. Following the same logic, the stability indices of the other two subdomain features can be calculated separately.

[0085] Calculating stability metrics ensures that the stability judgment criteria are consistent across different subdomains and different interference scenarios, avoiding human bias.

[0086] S203. When the stability index is greater than or equal to the preset first threshold, the preset modal confidence factor is reduced based on the preset first attenuation coefficient. When the stability index is greater than or equal to the preset first threshold, it indicates that the subdomain feature corresponding to the stability index is extremely unstable. Then, the preset modal confidence factor corresponding to this extremely unstable subdomain feature is reduced based on the preset first attenuation coefficient. For example, if the preset first threshold is set to 0.8, and the stability index of a certain subdomain feature is 0.91, and 0.91≥0.8, then it indicates that the subdomain feature is extremely unstable, and the preset modal confidence factor corresponding to it needs to be significantly reduced based on the preset first attenuation coefficient.

[0087] It should be noted that the preset first threshold and preset first attenuation coefficient are determined according to actual needs and are not limited here.

[0088] S204. When the stability index is less than the preset first threshold and greater than or equal to the preset second threshold, the preset modal confidence factor is reduced based on the preset second attenuation coefficient. When the stability index is less than a preset first threshold and greater than or equal to a preset second threshold, it indicates that the subdomain feature corresponding to the stability index is relatively unstable. In this case, the preset modal confidence factor corresponding to the relatively unstable subdomain feature is reduced based on the preset second attenuation coefficient. For example, if the preset first threshold is set to 0.8 and the preset second threshold is 0.4, and the stability index of a certain subdomain feature is 0.6, 0.6 < 0.8 and 0.6 ≥ 0.4, then it indicates that the subdomain feature is relatively unstable and the corresponding preset modal confidence factor needs to be reduced based on the preset second attenuation coefficient.

[0089] It should be noted that the preset second threshold and preset second attenuation coefficient are determined according to actual needs and are not limited here.

[0090] S205. When the stability index is less than the preset second threshold, the preset modal confidence factor is increased based on the preset recovery rate until the preset modal confidence factor reaches the preset benchmark value.

[0091] When the stability index is less than the preset second threshold, it indicates that the subdomain feature corresponding to the stability index is stable. Then, the preset modal confidence factor corresponding to this stable subdomain feature is increased based on the preset recovery rate. For example, if the preset first threshold is set to 0.4, and the stability index of a certain subdomain feature is 0.3, 0.3 < 0.4, then it indicates that the subdomain feature is stable. The preset modal confidence factor corresponding to this subdomain feature is increased based on the preset recovery rate until the preset benchmark value is reached and then the adjustment stops.

[0092] It should be noted that the preset recovery rate and preset baseline value are determined according to actual needs and are not limited here.

[0093] The following provides a detailed description of the multimodal data fusion system for charging evaluation provided in this application. Please refer to [link / reference]. Figure 5 , Figure 5 One embodiment of the multimodal data fusion system provided in this application includes: The acquisition unit 501 is used to acquire multimodal raw data when the charging module is in operation. The multimodal raw data includes visual domain data, acoustic domain data and electrical parameter domain data. The construction unit 502 is used to perform cross-domain semantic activation processing on the multimodal raw data to construct the corresponding deep representation tensor. The deep representation tensor carries visual subdomain features, acoustic subdomain features and electrical parameter quantum domain features respectively. The first generation unit 503 is used to perform forward semantic association mapping on the deep representation tensors of each subdomain feature based on a preset bidirectional cross-attention mechanism, so as to generate a cross-correlation weight matrix in the first direction. The second generation unit 504 is used to further perform reverse semantic backflow inference in the bidirectional cross-attention mechanism, constructing a reverse correlation mapping for the deep representation tensors of each subdomain to generate a cross-correlation weight matrix in the second direction. The fusion unit 505 is used to modulate and fuse the cross-correlation weight matrix of the first direction and the cross-correlation weight matrix of the second direction according to the preset modal confidence factor to obtain the intermodal correlation strength matrix. The modal confidence factor is used to characterize the effective participation degree of the corresponding subdomain features under the interference burst situation. Evaluation unit 506 is used to weight the deep representation tensor based on the intermodal correlation strength matrix, and to map the weighted deep representation tensor to a low-dimensional semantic subspace for aggregation using the adaptive sparse projection matrix in the preset cross-modal sparse projection network, so as to obtain a low-dimensional fused semantic representation, which is then input into the preset charging health evaluation model for evaluation.

[0094] Optionally, building unit 502 is specifically used for: The visible data is processed by multi-layer convolution through a pre-trained convolutional neural network to generate the first initial feature vector. The acoustic domain data is processed by time-frequency analysis using a pre-trained time-frequency convolutional network to generate a second initial feature vector. The electrical parameter domain data is processed by time-series modeling through a pre-trained long short-term memory network to generate a third initial feature vector; The first initial feature vector, the second initial feature vector, and the third initial feature vector are respectively input into a preset gated loop unit. Redundant information is filtered out by the forget gate mechanism of the preset gated loop unit, and key features are retained by the update gate mechanism of the preset gated loop unit to obtain the deep representation tensor.

[0095] Optionally, building unit 502 is specifically used for: Brightness correction is performed on the visible field data, and low-level edge features, mid-level texture features, and high-level semantic features are extracted layer by layer from the convolutional layers of a pre-trained convolutional neural network. The pooling layers of a pre-trained convolutional neural network are used to reduce the dimensionality of low-level edge features, mid-level texture features, and high-level semantic features respectively. The pre-trained fully connected layers of a convolutional neural network map the dimensionality-reduced low-level edge features, mid-level texture features, and high-level semantic features into a first initial feature vector.

[0096] Optionally, it also includes an adjustment unit 507, specifically used for: The preset modal confidence factor is dynamically adjusted based on the visual subdomain features, acoustic subdomain features, and electrical parameter quantum domain features. The preset modal confidence factor is dynamically adjusted based on visual subdomain features, acoustic subdomain features, and electrical parameter quantum domain features, including: Real-time recording of signal-to-noise ratio fluctuations in visual subdomain features, acoustic subdomain features, and electrical parameter quantum domain features within a preset time window; The stability index of each subdomain feature is calculated based on the signal-to-noise ratio fluctuation data, and the preset modal confidence factor is dynamically adjusted based on the stability index.

[0097] Optionally, the adjustment unit 507 is specifically used for: When the stability index is greater than or equal to the preset first threshold, the preset modal confidence factor is reduced based on the preset first attenuation coefficient. When the stability index is less than a preset first threshold and greater than or equal to a preset second threshold, the preset modal confidence factor is reduced based on the preset second attenuation coefficient. When the stability index is less than the preset second threshold, the preset modal confidence factor is increased based on the preset recovery rate until the preset modal confidence factor reaches the preset benchmark value.

[0098] Optionally, the fusion unit 505 is specifically used for: The cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction are standardized to obtain the forward correlation matrix and the backward correlation matrix. A confidence modulation matrix is ​​constructed based on a preset modal confidence factor, and the forward correlation matrix and the reverse correlation matrix are bidirectionally fused with the confidence modulation matrix to obtain an initial fusion matrix. The initial fusion matrix is ​​subjected to eigenvalue decomposition to obtain the principal feature components; Construct an intermodal correlation strength matrix based on the principal feature components.

[0099] Optionally, the fusion unit 505 is specifically used for: Calculate the L2 norm of each row of the weight vector in the cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction, respectively; Regularize each element in each row of the weight vector based on the L2 norm; Based on the Softmax function and combined with a preset temperature coefficient, the cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction are transformed by probability distribution. The transformed cross-correlation weight matrix in the first direction is used as the forward correlation matrix, and the transformed cross-correlation weight matrix in the second direction is used as the backward correlation matrix.

[0100] For details on the implementation method, please refer to [link / reference]. Figures 1-4 Examples will not be described in detail here.

[0101] This application also provides a multimodal data fusion device for charging evaluation. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 One embodiment of the multimodal data fusion apparatus provided in this application includes: Processor 601, memory 602, input / output unit 603, bus 604; The processor 601 is connected to the memory 602, the input / output unit 603, and the bus 604; The memory 602 stores a program, and the processor 601 calls the program to execute any of the multimodal data fusion methods described above.

[0102] This application also relates to a computer-readable storage medium storing a program, characterized in that when the program is run on a computer, it causes the computer to execute any of the above-described multimodal data fusion methods.

[0103] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0104] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0105] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0106] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0107] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A multimodal data fusion method for charging evaluation, characterized in that, include: Collect multimodal raw data when the charging module is in operation. The multimodal raw data includes visual domain data, acoustic domain data, and electrical parameter domain data. Cross-domain semantic activation processing is performed on the original multimodal data to construct the corresponding deep representation tensor, which carries visual subdomain features, acoustic subdomain features and electrical parameter quantum domain features respectively. Based on a pre-defined bidirectional cross-attention mechanism, forward semantic association mapping is performed on the deep representation tensors of each subdomain feature to generate a cross-correlation weight matrix in the first direction. In the bidirectional cross-attention mechanism, reverse semantic backflow inference is further performed to construct an inverse correlation mapping for the deep representation tensors of each subdomain to generate a cross-correlation weight matrix in the second direction; Based on the preset modal confidence factor, the cross-correlation weight matrix of the first direction and the cross-correlation weight matrix of the second direction are modulated and fused to obtain the intermodal correlation strength matrix. The modal confidence factor is used to characterize the effective participation degree of the corresponding subdomain features under the condition of sudden interference. The deep representation tensor is weighted based on the intermodal correlation strength matrix, and the weighted deep representation tensor is mapped to a low-dimensional semantic subspace for aggregation using the adaptive sparse projection matrix in the preset cross-modal sparse projection network to obtain a low-dimensional fused semantic representation, which is then input into a preset charging health assessment model for evaluation.

2. The multimodal data fusion method according to claim 1, characterized in that, The step of performing cross-domain semantic activation processing on the multimodal raw data to construct the corresponding deep representation tensor includes: The visible data is processed by multi-layer convolution through a pre-trained convolutional neural network to generate a first initial feature vector. The acoustic domain data is processed by time-frequency analysis using a pre-trained time-frequency convolutional network to generate a second initial feature vector. The electrical parameter domain data is processed by time-series modeling using a pre-trained long short-term memory network to generate a third initial feature vector; The first initial feature vector, the second initial feature vector, and the third initial feature vector are respectively input into a preset gated loop unit. Redundant information is filtered out by the forget gate mechanism of the preset gated loop unit, and key features are retained by the update gate mechanism of the preset gated loop unit to obtain the deep representation tensor.

3. The multimodal data fusion method according to claim 2, characterized in that, The step of performing multi-layer convolution processing on the visible data using a pre-trained convolutional neural network to generate a first initial feature vector includes: Brightness correction is performed on the visible field data, and low-level edge features, mid-level texture features, and high-level semantic features are extracted layer by layer from the convolutional layers of a pre-trained convolutional neural network. The pooling layers of the pre-trained convolutional neural network are used to reduce the dimensionality of the low-level edge features, the mid-level texture features, and the high-level semantic features, respectively. The dimensionality-reduced low-level edge features, mid-level texture features, and high-level semantic features are mapped to a first initial feature vector through the fully connected layer of the pre-trained convolutional neural network.

4. The multimodal data fusion method according to claim 1, characterized in that, Before modulating and fusing the cross-correlation weight matrices of the first and second directions according to a preset modal confidence factor, the multimodal data fusion method further includes: The preset modal confidence factor is dynamically adjusted based on the visual subdomain features, the acoustic subdomain features, and the electrical parameter quantum domain features. The dynamic adjustment of the preset modal confidence factor based on the visible subdomain features, the acoustic subdomain features, and the electrical parameter quantum domain features includes: Real-time recording of the signal-to-noise ratio fluctuation data of the visible subdomain features, the acoustic subdomain features, and the electrical parameter quantum domain features within a preset time window; The stability index of each subdomain feature is calculated based on the signal-to-noise ratio fluctuation data, and the preset modal confidence factor is dynamically adjusted based on the stability index.

5. The multimodal data fusion method according to claim 4, characterized in that, The step of dynamically adjusting the preset modal confidence factor based on the stability index includes: When the stability index is greater than or equal to a preset first threshold, the preset modal confidence factor is reduced based on a preset first attenuation coefficient. When the stability index is less than the preset first threshold and greater than or equal to the preset second threshold, the preset modal confidence factor is reduced based on the preset second attenuation coefficient. When the stability index is less than the preset second threshold, the preset modal confidence factor is increased based on the preset recovery rate until the preset modal confidence factor reaches the preset benchmark value.

6. The multimodal data fusion method according to any one of claims 1 to 5, characterized in that, The step of modulating and fusing the cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction according to a preset modal confidence factor to obtain an intermodal correlation strength matrix includes: The cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction are standardized to obtain the forward correlation matrix and the backward correlation matrix. A confidence modulation matrix is ​​constructed based on a preset modal confidence factor, and the forward correlation matrix and the reverse correlation matrix are bidirectionally fused with the confidence modulation matrix to obtain an initial fusion matrix; The initial fusion matrix is ​​subjected to eigenvalue decomposition to obtain the principal feature components; Based on the principal feature components, construct the intermodal correlation strength matrix.

7. The multimodal data fusion method according to claim 6, characterized in that, The standardization process for the cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction to obtain the forward correlation matrix and the backward correlation matrix includes: Calculate the L2 norm of each row of the weight vector in the cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction, respectively; Regularize each element in each row of the weight vector based on the L2 norm; Based on the Softmax function and combined with a preset temperature coefficient, the cross-correlation weight matrix of the first direction and the cross-correlation weight matrix of the second direction after processing are transformed by probability distribution. The transformed cross-correlation weight matrix of the first direction is used as the forward correlation matrix, and the transformed cross-correlation weight matrix of the second direction is used as the backward correlation matrix.

8. A multimodal data fusion system for charging assessment, characterized in that, include: The acquisition unit is used to acquire multimodal raw data when the charging module is in operation. The multimodal raw data includes visual domain data, acoustic domain data, and electrical parameter domain data. The construction unit is used to perform cross-domain semantic activation processing on the multimodal raw data to construct the corresponding deep representation tensor, wherein the deep representation tensor carries visual subdomain features, acoustic subdomain features and electrical parameter quantum domain features respectively; The first generation unit is used to perform forward semantic association mapping on the deep representation tensors of each subdomain feature based on a preset bidirectional cross-attention mechanism, so as to generate a cross-correlation weight matrix in the first direction. The second generation unit is used to further perform reverse semantic backflow inference in the bidirectional cross-attention mechanism, and construct reverse correlation mapping for the deep representation tensors of each subdomain to generate a cross-correlation weight matrix in the second direction. The fusion unit is used to modulate and fuse the cross-correlation weight matrix in the first direction and the cross-correlation weight matrix in the second direction according to the preset modal confidence factor to obtain the intermodal correlation strength matrix. The modal confidence factor is used to characterize the effective participation degree of the corresponding subdomain features under the condition of sudden interference. The evaluation unit is used to weight the deep representation tensor based on the intermodal correlation strength matrix, and to map the weighted deep representation tensor to a low-dimensional semantic subspace for aggregation using the adaptive sparse projection matrix in the preset cross-modal sparse projection network to obtain a low-dimensional fused semantic representation, which is then input into the preset charging health evaluation model for evaluation.

9. A multimodal data fusion device for charging evaluation, characterized in that, include: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor calls to execute the multimodal data fusion method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, the program executing, when executed on a computer, the multimodal data fusion method as described in any one of claims 1 to 7.

Images