Method and corresponding product for enterprise data monitoring with multimodal feature fusion

Abstract

The application relates to the field of industrial planning, and provides an enterprise data monitoring method based on multi-modal feature fusion and a corresponding product. The method comprises the following steps: acquiring multi-modal monitoring data of an enterprise; performing feature extraction on the multi-modal monitoring data to obtain an initial feature set corresponding to different modes; based on the initial feature set of each mode, evaluating the availability of the initial feature set in a current monitoring task, and screening out core modes participating in subsequent fusion according to the evaluation; based on a learned inter-modal correlation relationship, fusing the initial feature set corresponding to the core modes to generate a unified multi-modal fusion feature representation; performing enterprise state analysis based on the multi-modal fusion feature representation to output a monitoring result; and according to feedback information of the monitoring result, adjusting the basis for evaluating the availability of the core modes participating in subsequent fusion, and / or adjusting the inter-modal correlation relationship for fusing the initial feature set corresponding to the core modes.

Description

Technical Field

[0001] This application relates to the field of industrial planning, and in particular to a method and product for enterprise data monitoring that integrates multimodal features. Background Technology

[0002] With the development of information technology, the data generated in the daily operations of enterprises is not only massive in quantity but also diverse in form, encompassing structured and unstructured data from various sources such as financial systems, production sensors, market reports, internal communications, and supply chain logs. This information from different sources, with different formats and semantics, is called multimodal data, and together they form an information mosaic that comprehensively portrays the state of an enterprise.

[0003] To extract valuable information from these heterogeneous multimodal data, existing technologies typically employ data fusion methods. However, these existing solutions have the following significant limitations in practical applications: 1) They often overlook the quality differences and correlation strengths of different modal data under specific monitoring tasks; 2) Most existing fusion methods are based on static and fixed association assumptions between modal features, lacking the ability to learn and adapt to complex and dynamic intermodal interactions. This leads to a significant decrease in fusion effectiveness and unreliable monitoring results when data distribution changes or noise occurs; 3) The entire monitoring process is often open-loop, making it difficult for the system to adapt to dynamic changes in the internal and external environment of the enterprise, and in the long run, monitoring accuracy and robustness are difficult to guarantee. Summary of the Invention

[0004] This application provides a multimodal feature fusion method and corresponding product for enterprise data monitoring. Through a closed-loop optimization mechanism, it achieves the self-adaptation and accuracy improvement of enterprise monitoring under multimodal data fusion.

[0005] On the one hand, this application provides a multimodal feature fusion method for enterprise data monitoring, the method comprising:

[0006] Step S1: Obtain the enterprise's multimodal monitoring data, which includes data from at least two different sources or formats;

[0007] Step S2: Extract features from the multimodal monitoring data to obtain an initial feature set corresponding to different modes;

[0008] Step S3: Based on the initial feature set of each modality, evaluate its usability under the current monitoring task, and select the core modalities to participate in subsequent fusion accordingly;

[0009] Step S4: Based on the learned intermodal relationships, the initial feature sets corresponding to the core modalities are fused to generate a unified multimodal fusion feature representation;

[0010] Step S5: Perform enterprise status analysis based on the multimodal fusion feature representation and output monitoring results;

[0011] Step S6: Based on the feedback information of the monitoring results, adjust the usability assessment criteria used for screening in step S3, and / or adjust the intermodal correlations used for weighted integration in step S4.

[0012] On the other hand, this application provides an enterprise data monitoring device that integrates multimodal feature fusion, the device comprising:

[0013] The acquisition module is used to acquire multimodal monitoring data of the enterprise, wherein the multimodal monitoring data includes data from at least two different sources or formats;

[0014] The extraction module is used to extract features from the multimodal monitoring data to obtain an initial feature set corresponding to different modalities;

[0015] The evaluation module is used to assess the usability of each modality under the current monitoring task based on the initial feature set of each modality, and to select the core modalities to participate in subsequent fusion accordingly.

[0016] The fusion module is used to fuse the initial feature sets corresponding to the core modalities based on the learned inter-modal relationships to generate a unified multimodal fusion feature representation;

[0017] The analysis module is used to perform enterprise status analysis based on the multimodal fusion feature representation and output monitoring results;

[0018] The adjustment module is used to adjust the availability assessment criteria used for screening in step S3 and / or adjust the intermodal correlations used for weighted integration in step S4 based on the feedback information of the monitoring results.

[0019] Thirdly, this application provides an apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the technical solution of the enterprise data monitoring method for multimodal feature fusion as described above.

[0020] Fourthly, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described enterprise data monitoring method for multimodal feature fusion.

[0021] As can be seen from the technical solution provided in this application, on the one hand, by acquiring and fusing multimodal monitoring data from at least two different sources or formats, the information from heterogeneous data sources within the enterprise can be comprehensively utilized, overcoming the limitations of a single modal data perspective. This provides a more comprehensive and three-dimensional data foundation for enterprise status analysis, making subsequent monitoring results more comprehensive and reliable. Furthermore, by evaluating the usability of the initial feature sets of each modality under the current monitoring task and selecting core modalities to participate in subsequent fusion, low-quality, low-relevance modal data interference can be effectively filtered out, thus avoiding the problem of "garbage data in, garbage results out." This ensures that the data input to the fusion stage is high-value information that has undergone task adaptability screening, laying the foundation for generating high-quality fusion feature representations. On the other hand, by weighted integration of the core modal features based on the learned inter-modal relationships, the technical solution of this application can dynamically capture and quantify the complex, non-linear relationships between different modal features, rather than using fixed fusion rules. This makes the fusion process more efficient and reliable. By adapting to different task scenarios and the inherent patterns of data, the generated unified multimodal fusion feature representation can more accurately reflect the essence of enterprise status, thereby significantly improving the accuracy of subsequent status analysis. Thirdly, enterprise status analysis based on the aforementioned high-quality, highly adaptable fusion feature representation outputs monitoring results derived from in-depth collaborative mining of multi-source heterogeneous information. Therefore, it is more reliable than analysis results based on a single modality or simple fusion methods. Furthermore, since the fusion weight correlation is learned, it provides a traceable basis for analytical decisions to a certain extent. Fourthly, based on the feedback information from the monitoring results, the usability assessment criteria used for modality screening and / or the inter-modality correlation used for feature weighting integration are dynamically adjusted, forming a closed-loop optimization loop for the entire monitoring system. This mechanism allows the system to learn from historical monitoring experience and continuously correct its data selection and fusion strategies. Thus, it can maintain and continuously improve monitoring performance when facing data distribution drift, new modalities, or task changes, possessing robustness and adaptability for long-term application. In summary, the technical solution of this application achieves adaptiveness and improved accuracy in enterprise monitoring under multimodal data fusion through a closed-loop optimization mechanism. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of the enterprise data monitoring method using multimodal feature fusion provided in this application embodiment;

[0024] Figure 2 This is a schematic diagram of the structure of the enterprise data monitoring device with multimodal feature fusion provided in the embodiments of this application;

[0025] Figure 3 This is a schematic diagram of the device provided in the embodiments of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] In this specification, adjectives such as "first" and "second" are used only to distinguish one element or action from another, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element or component or step (etc.) should not be construed as being limited to only one of the elements, components, or steps, but may be one or more of the elements, components, or steps, etc.

[0028] For ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale.

[0029] Real-time and accurate monitoring of enterprise operational status is fundamental for scientific decision-making, risk warning, and process optimization. With the development of information technology, the data generated in daily enterprise operations is not only massive in quantity but also diverse in form, encompassing structured and unstructured data from various sources such as financial systems, production sensors, market reports, internal communications, and supply chain logs. This information from different sources, with different formats and semantics, is called multimodal data, which together constitute an information mosaic for comprehensively depicting the enterprise's status. To extract valuable information from this heterogeneous multimodal data, existing technologies typically employ data fusion methods. A common approach is to first process the data of different modalities using their respective feature extractors, then perform simple concatenation or weighted averaging of the extracted features, and finally input them into an analytical model for status assessment or anomaly detection. Another, more complex approach is to pre-set fixed weights for different modalities or use correlation analysis based on traditional statistical methods to determine the fusion method when designing the fusion strategy. However, the aforementioned existing technical solutions have the following obvious limitations in practical applications: 1) They usually ignore the quality differences and correlation strengths of different modal data under specific monitoring tasks. For example, when assessing supply chain risks, real-time logistics sensor data may be more timely and reliable than lagging monthly report texts, but static fusion strategies cannot dynamically reflect this difference; 2) Most existing fusion methods are based on the assumption of static and fixed correlations between modal features, lacking the ability to learn and adapt to complex and dynamic intermodal interactions, resulting in a significant decrease in fusion effect and unreliable monitoring results when data distribution changes or noise occurs; 3) The entire monitoring process is often open-loop, that is, it lacks a mechanism to continuously optimize the feature selection and fusion strategy at the front end based on the feedback of monitoring results, making it difficult for the system to adapt to the dynamic changes in the internal and external environment of the enterprise, and in the long run, the monitoring accuracy and robustness are difficult to guarantee.

[0030] To address the aforementioned problems in existing technologies, this application proposes a multimodal feature fusion method for enterprise data monitoring, the flowchart of which is attached. Figure 1 As shown, the main steps include S101 to S106, which are detailed below:

[0031] Step S101: Obtain the enterprise's multimodal monitoring data, wherein the multimodal monitoring data includes data from at least two different sources or formats.

[0032] In the enterprise operating environment, data naturally possesses multi-source and heterogeneous characteristics. Traditional monitoring methods often rely on a single data source (such as only financial data or only log data), resulting in a one-sided analytical perspective and difficulty in comprehensively reflecting the complex state of the enterprise. This application first collects raw monitoring data synchronously or asynchronously from multiple internal and external information systems and sensing devices within the enterprise. These data differ significantly in their sources and formats, constituting a multimodal monitoring data set.

[0033] Specifically, the multimodal monitoring data may include, but is not limited to, the following categories:

[0034] 1. Structured business data: For example, database records from systems such as Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), and Supply Chain Management (SCM). This type of data is usually in tabular form and contains clearly defined fields, such as transaction amount, inventory quantity, order date, and customer rating, etc., and is highly standardized.

[0035] 2. Unstructured operational data: For example, text chat logs, email content, and project report documents from internal enterprise communication platforms (e.g., WeChat Work, DingTalk); image or video recordings from the production site; and audio recordings from conferencing systems. This type of data has no fixed format, is rich in semantic information, but requires in-depth analysis.

[0036] 3. Time-series sensor data: For example, sensor readings from Internet of Things (IoT) devices, such as temperature, vibration frequency, and energy consumption data of production line equipment; GPS trajectory and speed information of logistics vehicles; temperature, humidity, and CPU load of server rooms. This type of data is a series of observations arranged in chronological order, containing dynamic changes in equipment status and business processes.

[0037] After acquiring this raw data, necessary preprocessing operations are usually required to provide clean and well-organized input for subsequent feature extraction steps. Preprocessing operations may include:

[0038] Data cleaning: handling missing values ​​(e.g., using interpolation or padding), identifying and correcting obvious errors or outliers.

[0039] Format standardization: Converting data from different sources into a unified encoding format, timestamp, and unit of measurement.

[0040] Data alignment: For data with time-series relationships, alignment is performed based on a unified time benchmark to ensure that data from different modalities are comparable in the time dimension.

[0041] Preliminary analysis of unstructured data: word segmentation and stop word removal for text data; keyframe extraction and resolution standardization for image / video data; transcription or characterization (e.g., Mel spectrogram extraction) for audio data.

[0042] Through the technical solutions described in the above embodiments, this application constructs a heterogeneous data foundation that integrates the enterprise's digital and physical worlds and covers the entire operational chain, providing abundant raw materials for subsequent in-depth and comprehensive status analysis. It overcomes the shortcomings of traditional methods, such as single data sources and insufficient information dimensions, enabling the monitoring system to "see" a more complete picture of the enterprise.

[0043] Step S102: Extract features from the multimodal monitoring data to obtain an initial feature set corresponding to different modes.

[0044] Directly analyzing raw multimodal data is inefficient and difficult because different modalities exist in different metric spaces, such as the pixel space of images, the vocabulary space of text, and the value domain space of time-series data. The purpose of feature extraction is to map these heterogeneous raw data into a unified mathematical representation (i.e., feature vector) space that better represents their inherent patterns and facilitates subsequent processing. Existing techniques often design independent feature extractors for different modalities, but the extracted features are often at different semantic levels and distributions, leading to difficulties in subsequent fusion.

[0045] To address the aforementioned issues, one embodiment of this application introduces a self-supervised pre-training and feature alignment step to learn more general and aligned feature representations. Specifically, the self-supervised pre-training and feature alignment included in step S102 can be implemented through steps S21 to S23, as detailed below:

[0046] Step S21: Using unlabeled historical multimodal data collected from enterprises, construct a self-supervised learning task and pre-train the basic feature extraction network.

[0047] Step S21 aims to leverage the massive amounts of unlabeled data accumulated by the enterprise throughout its history, allowing the model to learn the essential features of the data itself. This reduces reliance on expensive manually labeled data and improves the generalization ability of the feature extractor. Specifically, the self-supervised learning task in step S21 can be further refined into steps S21a to S21d as follows:

[0048] Step S21a: Using unlabeled historical multimodal data collected from enterprises, construct a self-supervised learning task that includes at least a time-series comparison prediction task and a modal mask reconstruction task.

[0049] Step S21b: Based on the time series comparison prediction task, construct positive and negative sample pairs for time series data, and enable the basic feature extraction network to distinguish between normal and abnormal patterns in different time periods through comparative learning.

[0050] Specifically, step S21b can be implemented as follows: For a time-series data (e.g., a sensor reading sequence), a similar "positive sample" sequence can be generated from the original sequence through data augmentation (including slight jitter, scaling, etc.), and a "negative sample" can be randomly selected from sequences from different time periods or different devices; simultaneously, an encoder network (which can be a one-dimensional convolutional neural network or a Transformer encoder) is constructed to extract the features of the sequence. The training objective of this encoder network is to make the features of the positive sample pairs as close as possible in the latent space, while maximizing the distance between the features of the negative sample pairs. The loss function for the time-series comparison prediction task... It can be represented as:

[0051]

[0052] in, These are features of the anchor point samples. It is a positive sample feature. It is a negative sample feature. It is a similarity function (e.g., cosine similarity). Here, is the temperature hyperparameter, and N is the batch size. Through this task, the encoder network learns to capture dynamic patterns in the sequence that are related to the time context, effectively distinguishing between normal operation and potential abnormal fluctuations.

[0053] Step S21c: Based on the modal masking reconstruction task, for the random masked input fragment of unstructured data, train the basic feature extraction network to reconstruct the masked content based on the context.

[0054] Specifically, step S21c can be implemented as follows: For text data, a portion of tokens are randomly masked, and a Transformer-based encoder-decoder is trained to predict the masked tokens based on the unmasked context. For image data, image patches can be randomly masked, and the encoder-decoder can reconstruct the pixel values ​​of these patches. The loss function for the modality masking reconstruction task... Typically, this involves cross-entropy loss (for text) or mean squared error loss (for images). Through this task, the basic feature extraction network is forced to deeply understand the semantic structure and contextual dependencies of the data, thereby learning semantically rich feature representations.

[0055] Step S21d: By jointly optimizing the loss of the temporal comparison prediction task and the modal mask reconstruction task, the basic feature extraction network is pre-trained so that it learns a general feature representation that is robust to noise and rich in semantics.

[0056] The specific implementation of step S21d includes: designing a multi-task loss function, for example... ,in, and These are the hyperparameters, or weights, for balancing the temporal contrast prediction task and the modality mask reconstruction task. Through this multi-task pre-training, the basic feature extraction network can simultaneously learn the dynamic patterns of temporal data and the semantic information of unstructured data, becoming a powerful general-purpose feature extractor.

[0057] Step S22: Use the pre-trained basic feature extraction network as a shared encoder to encode new input data of different modalities to obtain primary features.

[0058] After pre-training, the encoder portion of this basic feature extraction network is fixed or fine-tuned, serving as a shared encoder. For new input data, whether temporal or textual, it undergoes forward propagation through this shared encoder to obtain the corresponding primary feature vectors. For example, for a piece of text, the output might be the vector corresponding to the [CLS] token in the last hidden layer; for temporal data, the output might be the feature vector after global average pooling. This ensures that features from different modalities are mapped to a latent space defined by the same model parameters. The latent space is an abstract, low-dimensional mathematical space where the inherent patterns and regularities learned by the model are represented in vector form.

[0059] Step S23: Design a cross-modal alignment module to align features of different modalities in the latent space by maximizing mutual information or minimizing their distribution distance, and output the aligned initial feature set.

[0060] Although shared encoders provide a unified processing framework for data from different modalities, the distribution of the generated primary features in the latent space may still differ due to the inherent differences in the data. Cross-modal alignment aims to further bridge the semantic distance between these features from different sources, including alignment based on maximizing mutual information and / or alignment based on minimizing distribution distance, detailed below:

[0061] 1) Alignment based on maximizing mutual information. Mutual information measures the degree of interdependence between two random variables. In this embodiment, alignment based on maximizing mutual information can be: constructing a discriminator, taking a pair of feature vectors from different modalities as input, and determining whether they describe different aspects of the same entity at the same time point (positive pair) or are randomly paired (negative pair). By maximizing the mutual information between positive pair features, the model is encouraged to extract information shared across different modalities that is relevant to high-level semantics. The objective can be formalized as maximizing... , where f is the feature extraction function, and X and Y are inputs of different modalities.

[0062] 2) Alignment based on minimizing distribution distance. For example, using Maximum Mean Discrepancy (MMD) or adversarial training, MMD measures the difference between two distributions in the Reproducing Kernel Hilbert Space (RKHS). By minimizing the MMD between feature distributions of different modalities, their statistical properties can be made more consistent. Adversarial training introduces a modality discriminator that attempts to distinguish which modality a feature comes from, while the feature extractor strives to generate features that the discriminator cannot distinguish, thus achieving distribution alignment.

[0063] After step S102, especially through the self-supervised pre-training and feature alignment in steps S21 to S23, a high-quality, semantically aligned initial feature set is obtained. Where M is the number of modes, each It is a dense, low-dimensional vector representation of the corresponding modal data. This lays a solid foundation for subsequent fusion and effectively alleviates the "modal divide" problem.

[0064] Step S103: Based on the initial feature set of each modality, evaluate its usability under the current monitoring task, and select the core modalities to participate in subsequent fusion accordingly.

[0065] After obtaining the initial aligned feature set, a key challenge is selecting the most valuable information for the specific monitoring task from among the numerous modalities. Traditional methods often employ fixed rules (e.g., selecting all modalities or pre-specifying a few "important" modalities) or simple weighted averages, which ignores two crucial facts: first, the data quality (e.g., completeness, noise level) of different modalities may dynamically change over time or in different scenarios; second, the importance of the same modality can vary drastically in different tasks (e.g., financial risk warning vs. production efficiency assessment). Static strategies cannot adapt to this dynamism and may result in low-quality or irrelevant information interfering with core judgments.

[0066] To address this, this application proposes a dynamic, task-aware modality availability assessment and selection mechanism. The core idea is to assess the availability of each modality under the current monitoring task based on its initial feature set, and then select the core modalities to participate in subsequent fusion. This not only filters out noise but also ensures the focus of fusion resources.

[0067] As an embodiment of this application, the usability of each modality under the current monitoring task is evaluated based on the initial feature set of each modality, which can be achieved through steps S31 to S33, as detailed below:

[0068] Step S31: Calculate the internal data quality index of the initial feature set of each modality, wherein the internal data quality index of the initial feature set of each modality includes at least one of information entropy, missing rate and noise level.

[0069] Specifically, the calculation of internal data quality metrics for the initial feature set of each modality includes the following calculations: information entropy, missing rate, and noise level:

[0070] 1) Information entropy: For a feature vector distribution of a modality, its information entropy can be calculated. ,in, This is a probability estimate of the feature value falling into the i-th interval. Higher entropy may indicate that the feature contains more information, but it may also originate from noise. In the context of the task, the relative change in entropy can be of interest.

[0071] 2) Missing rate: The proportion of valid data for this modality feature within a recent time window. For example, if a sensor signal is frequently interrupted, its missing rate will increase.

[0072] 3) Noise level: This can be assessed by calculating the smoothed residuals of the feature sequence or by estimating the energy of high-frequency components using methods such as wavelet transform. For image or text features, the stability or confidence of their embedding vectors can be evaluated.

[0073] The internal data quality indicators of each modality's initial feature set, such as information entropy, missing rate, and noise level, quantify the reliability of the modality data.

[0074] Step S32: Calculate the correlation index between the initial feature set of each modality and the current monitoring task objective.

[0075] Specifically, the correlation index for calculating the initial feature set of each modality and the current monitoring task objective can be: the monitoring task objective is typically represented by a label or target variable Y (e.g., a "normal / abnormal" category, or the value of a key performance indicator). For the features of each modality... The correlation between it and Y can be calculated. For classification tasks, mutual information can be used. For regression tasks, Pearson correlation coefficients or model-based feature importance scores can be used. For example, a simple prediction model based on single-modal features can be trained, and its performance or feature weights can be used as a measure of relevance. This step quantifies the association between modal features and the current task.

[0076] Step S33: Based on the preset decision rules, combine the correlation index with the internal features of each modality's initial feature set to generate a quantified usability score for each modality.

[0077] Specifically, the implementation of step S33 mainly includes decision rule and score generation, including: the preset decision rule can be a learnable function or a heuristic formula. For example, usability score. It can be designed as:

[0078]

[0079] in, It is the overall data quality score of the i-th modality (which can be a weighted combination of information entropy, missing rate, and noise level, after normalization). It is its relevance score to the task (normalized). and It is a trade-off factor ( This determines the relative importance of data reliability and task relevance. Another more complex approach is to train a small neural network that takes quality and relevance metrics as input, uses the predicted contribution of the modality feature to the final task as a supervision signal, and directly outputs an availability score.

[0080] After generating a quantified usability score for each modality, the core modalities participating in subsequent fusion can be selected based on this score. Specifically, this includes: based on the calculated usability scores of each modality... Apply filtering strategies. For example, set an absolute threshold. Select all The modalities can be selected based on their scores; or a relative threshold can be set to select the top K modalities by score. The selected modalities constitute a subset of the core modalities. .

[0081] In another embodiment of this application, the usability of each modality under the current monitoring task is evaluated based on the initial feature set of each modality, which can be achieved through steps S'31 to S'35, as detailed below:

[0082] Step S'31: Construct a task context-aware usability evaluation network, which includes a task encoder, a modal feature encoder, a cross-attention interaction module, and an evaluation head.

[0083] In this embodiment, the availability assessment network consists of a task encoder, a modal feature encoder, a cross-attention interaction module, and an assessment head. The task encoder maps the identifier of the current monitoring task (e.g., "supply chain risk warning" or "production efficiency assessment") to a task context vector, and the modal feature encoder extracts a compact modal feature summary vector for the initial feature set of each modality.

[0084] Step S'32: Based on the cross-attention mechanism, calculate the relevance weights of each modality feature to the current task context.

[0085] The task context vector is used as the query, and the modality feature summary vectors of each modality are used as the key and value, respectively, and input into the cross-attention interaction module. By calculating and normalizing the similarity between the query and each key, a set of attention weights is obtained, which characterize the relative importance of each modality feature summary in the current specific task context.

[0086] Step S'33: Use the evaluation head to generate availability scores and uncertainty estimates for each modality.

[0087] The value vectors of each modality, weighted by the cross-attention interaction module, are input into the evaluation head. The evaluation head of this application is designed to output more than just a scalar usability score. It also simultaneously outputs an uncertainty estimate characterizing the reliability of the score. Uncertainty estimation This is obtained by evaluating a specific branch in the head (e.g., predicting a log-variance), reflecting the low confidence of the evaluation due to data noise, feature ambiguity, or insufficient relevance to the task.

[0088] Step S'34: Integrate availability score and uncertainty estimate to perform uncertainty-based adaptive modality screening.

[0089] First, based on uncertainty estimation Original usability score Perform calibration, for example using This method attenuates the modal scores with high uncertainty. Then, based on the calibrated scores... A differentiable screening strategy (e.g., Gumbel-Softmax or Top-K smoothing approximation) is employed to select a subset of core modes from all modes. This strategy ensures that high-scoring modes are selected with a high probability, while the entire screening process is capable of gradient backpropagation, allowing the usability assessment network to be integrated with downstream and jointly optimized by analysis tasks in an end-to-end manner.

[0090] Step S'35: Utilize feedback from downstream tasks to perform online meta-updates on the availability assessment network.

[0091] The selected core modal subset is used for subsequent feature fusion and state analysis, and feedback from monitoring results (such as changes in loss value and accuracy) is obtained. Based on this feedback, the meta-gradient of the availability assessment network parameters is calculated. This meta-gradient is applied periodically or irregularly to perform small-step online updates to the assessment network, enabling its assessment criteria to adapt to changes in the enterprise operating environment and the evolution of task objectives, thus achieving continuous self-optimization of the assessment strategy.

[0092] Through step S103 of the above embodiment, the system achieves data-driven intelligent modality selection. It can automatically discard high-noise modes caused by equipment failure or redundant modes with extremely low relevance under specific tasks, thereby concentrating computing resources on high-quality, highly relevant information. This not only improves the efficiency of subsequent fusion but also ensures the purity and task relevance of the input information from the source, laying the foundation for generating reliable fusion features.

[0093] Step S104: Based on the learned intermodal relationships, the initial feature sets corresponding to the core modalities are fused to generate a unified multimodal fusion feature representation.

[0094] After identifying the core modalities, the next step is to fuse these heterogeneous feature vectors into a unified, information-rich representation. Simple feature concatenation or weighted averaging cannot capture the complex, non-linear interactions between modalities. For example, when a decline in profits in financial statements (Modal A) occurs simultaneously with a surge in negative sentiment about the company on social media (Modal B), the cumulative effect may be far greater than the simple sum of their individual effects. Therefore, this application proposes an intelligent fusion method based on learned inter-modal relationships.

[0095] Specifically, based on the learned intermodal relationships, the initial feature sets corresponding to the core modalities are fused to generate a unified multimodal fusion feature representation, which can be achieved through the following steps S41 to S43:

[0096] Step S41: Assign initial fusion weights to each core modality based on the availability score of each modality.

[0097] The importance of the selected core modalities still varies. One direct approach is to use usability scores. ( Normalized weights are used as their initial fusion weights. ,Right now This provides a reasonable initial bias for fusion based on upstream evaluation.

[0098] Step S42: Employ an attention-based fusion network to dynamically adjust the initial fusion weights based on the real-time interactions between the initial feature sets, thereby generating the final feature weighting coefficients.

[0099] Step S42 is the core of the fusion process, primarily because the attention mechanism simulates the "focusing" ability in human cognition, dynamically determining which information to focus on and the degree of focus based on the current specific input context. In this embodiment, the attention-based fusion network performs the following steps S421 to S423:

[0100] Step S421: Calculate query-key-value pairs, where the query vector of the query-key-value pair is generated by the current monitoring task identifier, and the key-value pair is obtained by linear transformation of the initial feature set of each modality.

[0101] First, a task embedding vector, i.e., a query vector q, is learned or defined for the current monitoring task (e.g., "supply chain disruption risk warning"). Then, features for each core modality are... ( ), through learnable linear transformation matrices and Project them respectively as key vectors Sum value vector In this context, the key vector is used to calculate the relevance to the query, and the value vector is the information to be aggregated.

[0102] Step S422: Calculate the attention distribution between modalities based on the similarity between the query vector q and each key vector.

[0103] Specifically, step S422 can be implemented by: calculating the query vector q and each key The similarity is calculated (usually using dot product or additive attention). Then, it is normalized using the Softmax function to obtain the attention weights. :

[0104]

[0105] This set of weights This is known as attention distribution, which quantifies the importance of each modal feature within the current task context. This process enables the modeling of dynamic collaborative relationships between modalities because the weights are calculated in real-time based on the current input features, rather than remaining fixed.

[0106] Step S423: Recalibrate the initial fusion weights based on the attention distribution to obtain the final feature weighting coefficients that reflect the dynamic cooperation relationship between modalities.

[0107] In the embodiments of this application, the final fusion weight It can be designed as a combination of initial weights and attention weights, for example... ,in, It is a learnable gating parameter used to balance prior importance (from step S103) and dynamic context importance (from the current input).

[0108] Step S43: Based on the final feature weighting coefficients, perform weighted summation and nonlinear transformation on the initial feature set to generate a unified multimodal fusion feature representation.

[0109] In this embodiment of the application, a unified multimodal fusion feature representation We obtain the following by weighted summation and passing it through a nonlinear activation function (e.g., ReLU):

[0110]

[0111] vector This is a unified multimodal fusion feature representation that integrates the essential information of all core modalities and optimizes and integrates them according to task requirements and dynamic relationships between modalities.

[0112] In another embodiment of this application, based on the learned intermodal relationships, the initial feature sets corresponding to the core modalities are fused to generate a unified multimodal fusion feature representation, which can be achieved through the following steps S'41 to S'45:

[0113] Step S'41: Construct modal feature nodes and initialize the modal relationship graph.

[0114] The initial feature vector of each core modality As a node in the graph, initialize a fully connected modal graph. , where the node set N is the number of core modes, and the initial edge set. It includes connections between all node pairs. A learnable association vector is initialized for each edge (i,j). It is used to encode the complex interaction types and strengths between modes i and j.

[0115] Step S'42: Iteratively update the node and edge representations through relation-aware message passing.

[0116] Specifically, an L-round graph neural network (GNN) iteration can be performed. In the l-th iteration, for each target node i, the following relation aggregation, node update, and edge relation update are performed:

[0117] 1. Relationship aggregation: Aggregation comes from all its neighboring nodes. The message. Based on the characteristics of neighboring nodes and the edge relationship vector connecting the two. Co-generation: ,in, This indicates a message generation multilayer perceptron. .

[0118] 2. Node Update: Merge the aggregated messages with the node's own characteristics to update the node representation. ,in, This indicates that the node updates the gated loop unit, which enables the node to selectively retain historical information and integrate new messages.

[0119] 3. Edge Relationship Update: Synchronously update the edge relationship vector to reflect the changes in the intermodal relationships after this round of interaction. ,in, Represent edge relationships to update the multilayer perceptron.

[0120] This process enables nodes (modal features) and edges (intermodal relationships) to co-evolve in the iterations, ultimately resulting in a set of node representations containing rich contextual interaction information. set of edge relations .

[0121] Step S'43: Perform hierarchical graph pooling to generate hierarchical fused features.

[0122] The updated image Input a hierarchical graph pooling module. Instead of performing a simple global average, this module generates a hierarchical compressed representation through local clustering and pooling, and global graph pooling:

[0123] 1. Local Cluster Partitioning and Pooling: Using a learnable clustering assignment matrix, nodes are softly assigned to... Within each cluster, the characteristics of nodes within the same cluster are weighted and averaged based on their affinity with the cluster center, yielding... Local cluster-level features. Simultaneously, the connection strength between clusters is obtained by aggregating the weights of edges between nodes originally belonging to different clusters. Local cluster partitioning and pooling merge N modal nodes into... A more abstract cluster node.

[0124] 2. Global Graph Pooling: This involves pooling the graphs obtained above from... A new graph composed of clusters of nodes is generated using a global graph pooling function (such as attention pooling) to produce a fixed-dimensional global graph-level feature. Global graph pooling can capture the global structure and semantics of the entire modal graph.

[0125] Step S'44: Fuse hierarchical features to generate a unified multimodal fusion feature representation.

[0126] Local cluster-level feature set With global graph-level features To integrate. Specifically, to... As a query vector, it is processed through an attention mechanism. Weighted summarization is performed to obtain a globally-aware local summary. Ultimately, and The features are concatenated and passed through a nonlinear projection layer to generate a unified multimodal fusion feature representation. ,in, This indicates the final fusion of the multilayer perceptron.

[0127] Step S'45: Based on the multi-task loss of reconstruction and prediction, jointly optimize the graph construction and fusion process.

[0128] Design a multi-task loss function to train the entire graph fusion network:

[0129] 1. Modal feature reconstruction loss: requires the loss to be derived from the final fused features. Capable of decoding and reconstructing the initial features of each modality This ensures that no key modal information is lost during the fusion process.

[0130] 2. Downstream task prediction loss: Used for downstream monitoring tasks (such as classification) to calculate predicted losses.

[0131] 3. Graph sparsity regularization loss: encourages edge relation vectors The L1 norm approaches zero, which automatically prunes irrelevant or redundant intermodal connections, resulting in a simpler and more critical relational topology.

[0132] By jointly optimizing the above losses, the entire system learns end-to-end how to construct the optimal modality graph, how to effectively propagate information on the graph, and how to generate the most discriminative hierarchical fusion features.

[0133] In addition, to further improve the robustness and processing capability of the fusion model, the following enhancement mechanisms can be introduced before or during fusion:

[0134] After step S103 and before step S104, step S3a may also be included: adversarial noise injection and robust training are performed on the initial feature set of the selected core modalities. Specifically, step S3a can be implemented through steps S3a1 to S3a3, as detailed below:

[0135] Step S3a1: Add random noise that conforms to a preset distribution to the initial feature set to generate noise perturbation features.

[0136] Step S3a2: Input the original features and the noise perturbation features into the fusion network (i.e. the attention-based fusion network in step S42) at the same time, and calculate the difference between the fusion feature representations generated by the two.

[0137] Step S3a3: With the goal of minimizing the difference calculated in step S3a2, the parameters of the fusion network are optimized in reverse to improve its robustness to data perturbation.

[0138] Specifically, step S3a3 can be implemented by adding small but perturbative noise (e.g., Gaussian-distributed noise) to the input features and forcing the fusion network to produce fused representations of the original and noisy features as similar as possible. This allows the fusion network to be trained to ignore irrelevant small perturbations and learn more stable and essential feature associations. Its loss function can be expressed as... ,in, and These are the outputs obtained from the original features and the noisy features through the same fusion network, respectively. Including this in the total training loss allows the model parameters to be updated in a more robust direction.

[0139] Step S104 of the above embodiment further includes steps S44 and S45 of performing intermodal conflict detection and mitigation during feature fusion:

[0140] Step S44: Calculate the cosine similarity or mutual information between feature vectors of different modalities in real time, and identify feature pairs with similarity below a preset conflict threshold.

[0141] Specifically, step S44 can be implemented as follows: for any two core modal feature vectors and ,

[0142] Calculate cosine similarity If the value is lower than a preset threshold... (For example, -0.5) indicates a potential information conflict between the modal pairs under the current input. The conflict could stem from data errors, noise, or genuinely contradictory signals.

[0143] Step S45: Assign lower collaborative weights to the identified conflicting features, or activate independent pathways in the fusion network to decouple and recode features, so as to reduce the adverse effects of conflicting features on the fusion results.

[0144] Specifically, the implementation of step S45 mainly includes the following: allocating lower collaborative weights and feature decoupling and re-encoding:

[0145] 1. Assign lower collaborative weights: When calculating attention weights, for feature pairs (i, j) that are identified as conflicting, a penalty term can be applied to their similarity scores before Softmax calculation, such as multiplying them by a decay factor less than 1, thereby indirectly reducing their weight in the fusion.

[0146] 2. Feature Decoupling and Re-encoding: Design an independent pathway (e.g., an additional neural network layer) within the fusion network. When a conflict is detected, the pair of conflicting features is input into this independent pathway. The goal of this pathway is to map the two features to a new subspace where their conflicting components are separated or neutralized. The processed features are then fed into the main fusion process. This is equivalent to "isolating" the conflicting information.

[0147] Through step S104 and its enhancement mechanism, this application achieves adaptive, robust, and conflict-handling deep feature fusion. The fusion network can not only dynamically adjust the contribution of each modality according to the task and input, but also resist data noise and properly handle inconsistent information between modalities, thereby generating a highly reliable and information-rich unified feature representation h, providing a powerful input for the final enterprise state analysis.

[0148] Step S105: Perform enterprise status analysis based on multimodal fusion feature representation and output monitoring results.

[0149] After obtaining a unified multimodal fusion feature representation h, the next step is to transform it into specific insights and judgments about the enterprise's operational status. Traditional centralized analysis methods require all data to be aggregated to a central server, which can lead to significant pressure on data privacy, trade secret leaks, and network bandwidth in an enterprise environment. At the same time, data from different business units (e.g., production, sales, finance) often cannot be fully integrated due to isolation, resulting in limited generalization capabilities of the analytical model.

[0150] To address the aforementioned issues, this application introduces an innovative analytical architecture: distributed enterprise state analysis based on federated learning. The core of this method lies in sending multimodal fusion feature representations to local analytical nodes deployed across various business units within the enterprise. Each node then uses its own private data to update the model locally, sharing only the update amount (gradient) of the model parameters, thereby achieving collaborative intelligence under the principle of "data not leaving the domain."

[0151] As an embodiment of this application, enterprise status analysis based on multimodal fusion feature representation, and output of monitoring results can be achieved through the following steps S51 to S54:

[0152] Step S51: Send the multimodal fusion feature representation to the local analysis nodes deployed in each business unit of the enterprise.

[0153] The central server (or fusion node) distributes the multimodal fusion feature representation h generated in step S104 to the various local analysis nodes participating in federated learning. These nodes may be deployed in branch offices in different regions, data centers in different departments, or dedicated cloud instances. The transmitted content is the fusion feature that has undergone deep abstraction and anonymization, rather than the original business data, which in itself reduces the risk of directly exposing sensitive information.

[0154] Step S52: Each local analysis node incrementally trains the shared enterprise monitoring and analysis model based on the received multimodal fusion feature representation and uses local private data, and calculates the model update gradient.

[0155] Each local node k maintains a local dataset. This includes local private labels or target values ​​that are not shared externally and correspond to the received multimodal fusion feature representation h. (For example, whether the business unit's own operations are marked as abnormal, or the true value of a certain key performance indicator). All nodes share a global initial enterprise monitoring and analysis model. The model could be a multilayer perceptron, a classifier, or a regressor.

[0156] 1. Local training: node k uses As training samples, the shared enterprise monitoring and analysis model is tested locally. (The global model in round t) undergoes several rounds of incremental training (e.g., using stochastic gradient descent SGD), with the goal of minimizing the local loss function. .

[0157] 2. Gradient Calculation: After training, node k calculates the gradient of the local model parameters with respect to that sample (or a mini-batch of samples). This gradient This reflects the direction of model improvement based on the knowledge contained in the node's private data.

[0158] Step S53: Aggregate the encrypted gradients uploaded by each local analysis node and update the parameters of the global enterprise monitoring and analysis model.

[0159] Step S53 mainly includes the following processes: secure upload, secure aggregation, and global update:

[0160] 1. Secure upload: To protect potentially hidden privacy information in the gradient, node k uploads the gradient... Previously, the gradient information could be encrypted or differential privacy noise could be added. Then, the processed gradient information was uploaded to a central aggregation server.

[0161] 2. Safe aggregation, which means that after the central server collects the gradients from all participating nodes, it performs a safe aggregation operation. A classic method is the FedAvg algorithm, which calculates a weighted average of the gradients: ,in, It is the amount of data in node k. This refers to the total amount of data. Under encrypted or differential privacy frameworks, the aggregation process can be performed in a cryptographic state, further ensuring privacy.

[0162] 3. Global update, i.e., the central server uses the aggregated gradient. Update global model parameters: ,in, It is the learning rate.

[0163] Step S54: Distribute the updated global enterprise monitoring and analysis model to each local analysis node for analysis of subsequent multimodal fusion feature representations and output monitoring results.

[0164] Updated global model It is distributed back to all local nodes. When the new multimodal fusion feature representation When generated, each node uses the latest global model. Analyze it to obtain monitoring results This result can be a classification label (e.g., "high risk", "medium risk", "low risk"), a regression value (e.g., predicted sales), or anomaly scores. Outputting the monitoring results completes the transformation from multimodal data to final decision-making knowledge.

[0165] As can be seen from steps S51 to S54 of the above embodiments, the introduction of the federated learning mechanism enables enterprises to pool the wisdom of various business units and jointly train a more powerful and general enterprise monitoring and analysis model, while strictly adhering to data privacy regulations and internal data governance policies. The local private data of each node enriches the training sample distribution of the model, improving the model's overall understanding and generalization ability of complex enterprise states.

[0166] As another embodiment of this application, enterprise status analysis based on multimodal fusion feature representation, and output of monitoring results can be achieved through the following steps S'51 to S'55:

[0167] Step S'51: Based on the meta controller, dynamically generate personalized analysis subnets for each business unit.

[0168] Specifically, a meta-controller can be deployed on the central server, which receives unified multimodal fusion feature representations from the fusion nodes. The meta-features (e.g., statistical features, spectral features, etc.) are used as the basis for the meta-controller to dynamically generate a lightweight neural network subnet structure adapted to the data characteristics of the business unit based on these meta-features and the historical performance profiles of each business unit node. and its initialization parameters This subnet is a specific branch or sparse instance of the globally shared backbone network, specifically tailored to the current data pattern of that node.

[0169] Step S'52: Perform personalized training and knowledge refinement on the local node.

[0170] Each business unit node receives the personalized subnet issued by the network. and initialization parameters. Nodes utilize local private data. Train the subnet to obtain the locally optimal parameters. After training, the nodes do not directly upload parameters. Instead, they perform knowledge refinement, which involves inputting local data into the trained subnet, collecting the feature activation distributions of the intermediate layers and the softening probability distribution of the final output, and combining them to form a local knowledge package. This process protects the privacy of the original data, exposing only abstract knowledge representations.

[0171] Step S'53: Cross-node knowledge aggregation and distillation based on knowledge graph.

[0172] The central server maintains a global knowledge graph, where nodes represent different business models or data distribution prototypes, and edges represent knowledge transferability. The server receives knowledge packages uploaded by each node. Then, the following operations are performed: knowledge graph matching and clustering, intra-class knowledge distillation, and cross-class knowledge transfer:

[0173] 1) Knowledge graph matching and clustering, i.e.: matching and clustering knowledge graphs. Match the prototype nodes in the knowledge graph and assign them to the most similar cluster.

[0174] 2) Intra-cluster knowledge distillation: Within the same cluster, a comparative distillation method is used to narrow the distance between the knowledge representations of similar nodes and widen the distance between dissimilar nodes, thereby extracting the common knowledge of the category. .

[0175] 3) Cross-category knowledge transfer: along the edges of the knowledge graph, common knowledge from high-resource, high-performance nodes is transferred to low-resource or performance-needing node categories via a relation-aware graph neural network, generating transfer-enhanced knowledge. .

[0176] Step S'54: Generate and distribute the adaptive enhanced global analysis model.

[0177] The central server will distill the common knowledge of various categories. With transfer to enhance knowledge Back-distillation back to a reinforced globally shared backbone model Specifically, a distillation loss is designed so that the global model mimics the feature activation and output distributions in the knowledge bags of each category. The updated enhanced global model... It integrates the common experiences and migration wisdom of all nodes.

[0178] Step S'55: Co-evolution of feedback-based meta-controller and knowledge graph.

[0179] Each business unit node uses the distributed enhanced global model. New fusion features Perform analysis, output monitoring results, and calculate local performance gain. This performance gain and new data pattern characteristics are fed back to the central server. The central server, based on the feedback, co-evolves with the meta-controller and the knowledge graph. Specifically, this includes: using the feedback information to update the meta-controller's policy network through reinforcement learning, enabling it to generate more effective personalized subnet structures in the future; and dynamically updating the attributes of nodes and the weights of edges in the global knowledge graph based on the new node clustering results and performance feedback, even splitting or merging nodes, so that the graph always reflects the latest enterprise data ecosystem and knowledge associations.

[0180] Furthermore, in order to establish a virtuous cycle of data value transfer within the enterprise, this application can also integrate a data contribution evaluation mechanism in step S105, that is: step S105 also includes the following steps S55 to S56, which dynamically identify and quantify the data contribution of each business unit based on a federated learning mechanism:

[0181] Step S55: Each time gradients are aggregated to update global model parameters, the marginal contribution of gradient updates provided by each local analysis node to the improvement of global model performance is evaluated using Shapley values ​​or their computational efficiency optimization algorithms.

[0182] In this embodiment, the Shapley value originates from cooperative game theory and is used to fairly distribute the total payoff of the federated learning to each participant. In this context, "federation" refers to the set of all nodes participating in the federated learning, and "total payoff" is the improvement in global model performance (e.g., an increase in accuracy on the validation set). For a node k, its Shapley value... The calculation formula is:

[0183]

[0184] Where N is the set of all nodes, and S is any subset of the set excluding node k, i.e., the alliance. It represents the model performance that can be achieved when nodes in subset S participate in training. This represents the marginal contribution of node k to joining the federation S. The fair contribution of each node can be evaluated by calculating all possible subset combinations (or using approximation algorithms such as Monte Carlo sampling).

[0185] Step S56: Based on the marginal contribution of the gradient update provided by each local analysis node to the improvement of global model performance, generate a data asset value report for each business unit and feed it back into the availability assessment basis in step S103 as a dynamic factor for assessing the long-term value of the data source.

[0186] Specifically, step S56 can be implemented by: calculating the Shapley value Normalization serves as the "long-term value score" of the data source (modality) corresponding to node k. This score can be dynamically incorporated into the usability assessment criteria in step S103. For example, the usability score is calculated in step S33. At that time, it can be As an additional bonus or weighting factor: ,in, It is an adjustment coefficient. This allows data sources that, while having low data quality or immediate relevance at certain times, but contributing significantly to model performance improvement in the long run, to still receive high availability ratings and thus not be easily discarded during modality selection. This forms a value feedback loop from final model performance to front-end data selection strategies.

[0187] Step S106: Based on the feedback information from the monitoring results, adjust the availability assessment criteria used for screening in step S103, and / or adjust the intermodal correlations used for weighted integration in step S104.

[0188] Traditional monitoring systems are mostly open-loop systems; once deployed, their internal decision-making logic (such as modal filtering rules and fusion weights) remains fixed. However, enterprise environments change dynamically, data distribution may shift (concept drift), and new business scenarios may emerge. Fixed strategies lead to system performance degradation over time. This application introduces a closed-loop optimization mechanism, enabling the system to use its own monitoring results as feedback signals to automatically optimize its core decision-making components, achieving self-evolution.

[0189] This closed-loop optimization can be carried out along two paths:

[0190] Path 1: Adjust the availability assessment criteria used for screening in step S103, which mainly includes steps S61 to S66, detailed as follows:

[0191] Step S61: Construct a simplified causal graph model that includes multimodal features as causes and monitoring results as effects.

[0192] Traditional usability assessments primarily rely on relevance and data quality; however, relevance does not equate to causation. A modal feature may be highly correlated with a monitoring result, but this correlation could be a "spurious correlation" caused by confounding factors. For example, a social media sentiment index (modality A) might be highly correlated with corporate stock price fluctuations (outcome Y), but the real driver of stock prices might be underlying macroeconomic policies (confounding factor Z), and the sentiment index merely reflects the influence of Z. If modality A is given a high weight solely based on relevance, predictions will be inaccurate when Z remains constant while A fluctuates due to other factors. Therefore, this application introduces a simplified causal graphical model. This graph contains nodes: each modal feature. The monitoring result Y, and any key confounding factors that may be identified (if observable). Edges represent possible causal relationships, such as... This causal graphical model can be a structural equation model based on domain knowledge, or a causal structure learned from data.

[0193] Step S62: When the monitoring results show deviation, intervene in the causal graph model by fixing or perturbing the values ​​of one or more modal features.

[0194] Specific operation: When the system detects an increase in recent monitoring errors (e.g., an increase in the false alarm rate or missed alarm rate of warnings), causal analysis is triggered. This involves analyzing a specific modal feature node in the causal graph. Performing a "do-calculus" involves manually setting a value without considering its original cause. For example, in a simulation, [the value is set by the operator]. The value is fixed to its historical average, or its distribution is replaced with another distribution.

[0195] Step S63: Observe the changes in the predicted results after the intervention, and use this to quantitatively estimate the causal effect of each modality feature on the results.

[0196] Under the new causal graph after intervention, the causal graph model is rerun to calculate the expected monitoring results. Compare this to the expected outcome without intervention. Compare them. The difference between the two (or its derivative) is the characteristic. Average causal effect (ACE) on outcome Y This quantifies "if the characteristics are forcibly changed". "Then how much will the result Y change on average?" This statement removes the influence of confounding factors.

[0197] Step S64: Compare the statistical correlation index and causal effect index of each modality feature.

[0198] Step S65: If a feature of a certain modality exhibits high statistical correlation but low causal effect, then in the usability assessment criteria, its correlation-based weight is reduced, and a penalty term based on the stability of causal effect is introduced.

[0199] Specifically, compare the correlation index calculated in step S32. The causal effect index estimated in step S63 If a certain mode i is found It's very tall, but If the value is very small or even close to zero, it indicates that the feature is likely an indicator of "spurious correlation." In this case, the decision rule in step S33 should be reduced. Item weight coefficient Or directly introduce a... Inversely proportional penalty items (among them) To prevent division by zero for small constants, its usability score is improved. Lower ,in, It refers to the intensity of the punishment. Simultaneously, a system based on... The stability is measured across different time windows, and further penalties are imposed if the causal effect fluctuates significantly.

[0200] Step S66: Use the updated availability assessment criteria to perform modality screening, in order to more likely select core modalities that have a stable causal contribution to the monitoring results.

[0201] Updated assessment criteria It will be used in the next round or the next batch of modality screening (step S103), thereby systematically improving the "causal purity" of the selected modality set.

[0202] Path 2: Adjusting the intermodal correlations used for weighted integration in step S104, which mainly includes steps S61' to S64', detailed below:

[0203] Step S61': Construct a simplified causal graph model that includes multimodal features as causes and monitoring results as effects.

[0204] The implementation logic of step S61' is similar to that of step S61, and can be found in the relevant descriptions of the foregoing embodiments, which will not be repeated here.

[0205] Step S62': When the monitoring results show deviation, intervene in the causal graph model by fixing or perturbing the values ​​of one or more modal features.

[0206] The implementation logic of step S621' is similar to that of step S62, and can be found in the relevant descriptions of the foregoing embodiments, which will not be repeated here.

[0207] Step S63': Observe the changes in the predicted results after the intervention, and use this to quantitatively estimate the causal effect of each modality feature on the results.

[0208] The implementation logic of step S63' is similar to that of step S63, and can be found in the relevant descriptions of the foregoing embodiments, which will not be repeated here.

[0209] Step S64': Based on the causal effect, the weight parameters that reflect the intermodal correlation in the fusion network are directionally modified to reduce the fusion bias introduced by spurious correlations.

[0210] Modified Fusion Networks: Causal Effects Directly reflects the characteristics The magnitude of the actual driving force behind the result. This can be... After normalization, this serves as prior knowledge or a regularization term, guiding the update of weight parameters in the fusion network (the attention network in step S42). For example, a causal alignment regularization term can be added to the loss function of the fusion network: ,in, This is the attention weight predicted by the network. This loss term encourages the network to learn the attention distribution. Distribution of causal effect intensity of each feature Consistent with this, during backpropagation optimization of network parameters, the model is guided to assign higher attention weights to features with strong causal effects, thereby suppressing the influence of features that are only statistically correlated but have no causal effect in the fusion.

[0211] Through the closed-loop optimization in step S106, the entire monitoring system transforms from a statically executed program into an intelligent agent capable of learning from experience, reflecting on errors, and continuously improving itself. It uses the final monitoring performance as a "teacher" to optimize the "judgment criteria" and "fusion strategies" of the two core components: front-end data selection (step S103) and feature fusion (step S104), giving the system strong environmental adaptability and long-term robustness.

[0212] When deploying this invention to a completely new enterprise or business line, a "cold start" problem is often encountered: a lack of sufficient labeled data to train the feature extractor, fusion network, and analysis model. Traditional methods require collecting large amounts of data in the new scenario and training from scratch, which is time-consuming and costly. To address this, this application introduces a rapid adaptive phase based on meta-learning, namely... Figure 1When the example method is first deployed in a new enterprise scenario, it performs a rapid adaptive phase based on meta-learning, which mainly includes steps S01 to S03, detailed below:

[0213] Step S01: Extract meta-knowledge from existing monitoring tasks in multiple enterprise scenarios and build a meta-learner.

[0214] It should be noted that meta-learning aims to "learn how to learn." It assumes that historical monitoring task data already exists from multiple different enterprises (or multiple different business units within the same enterprise). Each task... Each has its own training set and test set This is used to learn how to monitor on a specific task. The goal of meta-learning is to extract cross-task general knowledge—that is, "meta-knowledge"—from learning experiences across multiple tasks, so that when faced with a completely new task... In this case, it can adapt quickly with only a small number of samples. Meta-learners can employ algorithms such as Model-Independent Meta-Learning (MAML). The essence of a meta-learner M is an initializer of model parameters. Its training process involves: training on multiple tasks... In the previous iteration, in each task, the current meta-parameters are used. The initialized model, after several steps of gradient descent, Quickly adapt to obtain task-specific parameters Then The loss is calculated; the ultimate goal is to find a set of initial parameters. This minimizes the average loss after rapid adaptation across all tasks. This group It encodes the meta-knowledge of "how to quickly adapt to a new monitoring task".

[0215] Step S02: Using a small number of initial samples from the new enterprise scenario, drive the meta-learner to quickly generate feature extractor parameters, fusion network initialization parameters, and analysis model prior parameters adapted to the new scenario.

[0216] When facing new enterprise scenarios At that time, there were only a small number of initial samples. (For example, several weeks' worth of data). This small sample size is fed into a trained meta-learner M, which in turn guides the parameters of the model (including feature extractors, fusion networks, and analysis models). The above quickly adjusts to a suitable state through gradient updates in very few steps. This state. This process is called meta-testing or rapid adaptation.

[0217] Step S03: Use the generated parameters as the initial state of the corresponding models in steps S102, S104, and S105 to achieve rapid cold start and performance improvement under small sample conditions.

[0218] The model parameters obtained after rapid adaptation are used as the initial parameters for: the feature extraction network (e.g., a shared encoder) in step S102, the fusion network (attention mechanism, etc.) in step S104, and the enterprise monitoring and analysis model (e.g., a classifier) ​​in step S105, respectively. In this way, the entire system does not start from random initialization in new scenarios, but from a "high starting point" that already possesses strong generalization and rapid learning capabilities. This significantly shortens the convergence time and data volume required for the system to reach usable performance, solving the core pain point of "cold start" in AI application deployment.

[0219] In the joint pre-training in step S03, a multi-task learning framework can be used to further enhance the initial model, specifically including the following steps S031 to S033:

[0220] Step S031: Construct a shared feature encoding layer whose output simultaneously serves the feature fusion task and at least one auxiliary prediction task; Step S032: The auxiliary prediction task is designed to be related to the main monitoring task but easier to learn, guiding the model to learn more generalizable feature representations (e.g., while the main task is "financial fraud risk classification," the auxiliary task could be "financial statement item continuity prediction" or "industry prosperity regression prediction"); Step S033: Simultaneously optimize the main task and auxiliary task using a weighted loss function to complete the model initialization. Multi-task learning forces the shared encoding layer to learn more essential feature representations useful for multiple related tasks, thereby improving the generalization and robustness of the initial model.

[0221] Figure 1 The example method is highly versatile and scalable, and its output monitoring results can drive more advanced enterprise digital applications. A typical implementation is to use the output monitoring results to drive the state update and simulation prediction of an enterprise digital twin in real time, specifically including the following steps S51' to S53':

[0222] Step S51': Take the multimodal fusion feature representation as input and synchronously update the virtual state of the corresponding entity or process in the enterprise digital twin.

[0223] A digital twin is a dynamic mapping of an enterprise in virtual space. For example, by integrating information such as production line efficiency, inventory levels, and order backlog contained in the feature representation, the state parameters of the corresponding virtual entities such as "production line," "warehouse," and "logistics" in the twin can be updated in real time.

[0224] Step S52': In the digital twin, run the simulation model based on the updated state to predict the trend of key indicators and potential risk points in the future.

[0225] Using the updated state, preset simulation models (such as discrete event simulation and system dynamics model) can be run to simulate business operations over a future period of time (such as the next hour or the next day) and predict key indicators such as "equipment failure probability", "order delivery delay risk" and "cash flow pressure".

[0226] Step S53': Combine the prediction results with the current monitoring results to generate an enhanced monitoring report that includes real-time diagnosis and forward-looking early warning.

[0227] This enables the monitoring system to not only "perceive the present" but also "foresee the future," providing enterprise decision-makers with comprehensive decision support, from real-time alerts to trend forecasts.

[0228] Furthermore, when monitoring focuses on supply chain risks, in-depth network analysis can be conducted using the aforementioned digital twin or directly based on the monitoring results. That is, when monitoring results are used for early warning of supply chain disruption risks, Figure 1 The example method may also include the following steps S5a to S5c:

[0229] Step S5a: Establish a supply chain network topology model and map the multimodal fusion feature representation to the corresponding nodes and edges of the topology model.

[0230] Supply chain networks can be modeled as a graph. In this model, node V represents entities such as suppliers, manufacturers, and distributors, while edge E represents relationships involving logistics, capital flow, and information flow. Supplier delivery delays, port congestion indices, and geopolitical risk scores, extracted from the fused features, are mapped to the attributes of the corresponding nodes and edges.

[0231] Step S5b: Simulate the risk propagation process in the supply chain network and calculate the risk exposure of each node and the vulnerability index of the overall network.

[0232] Using models such as infectious disease models, impact maximization models, or Bayesian networks, simulate how risk propagates through edges (dependencies) in a network when a node (e.g., a critical supplier) experiences an outage (risk source). Calculate the probability of each node being affected (risk exposure) and the likelihood of the entire network collapsing due to the failure of a critical node (vulnerability index).

[0233] Step S5c: Based on risk exposure and vulnerability index, identify key risk sources and generate targeted mitigation strategy recommendations as part of the monitoring results output.

[0234] For example, identifying the nodes with the highest risk exposure and recommending finding alternative suppliers for them; or recommending increasing safety stock for the connection edges with the highest vulnerability index. This upgrades risk warning from single-point judgment to systematic and networked analysis.

[0235] From the above appendix Figure 1 As can be seen from the example of the enterprise data monitoring method using multimodal feature fusion, on the one hand, by acquiring and fusing multimodal monitoring data from at least two different sources or formats, it is possible to comprehensively utilize information from heterogeneous data sources within the enterprise, overcoming the limitations of a single modality data perspective. This provides a more comprehensive and three-dimensional data foundation for enterprise status analysis, making subsequent monitoring results more comprehensive and reliable. Furthermore, by evaluating the usability of the initial feature sets of each modality under the current monitoring task and selecting core modalities to participate in subsequent fusion, low-quality and low-relevance modal data interference can be effectively filtered out, thus avoiding the problem of "garbage data in, garbage results out." This ensures that the data input to the fusion stage is high-value information that has undergone task adaptability screening, laying the foundation for generating high-quality fused feature representations. On the other hand, by weighted integration of the core modality features based on the learned inter-modal relationships, the technical solution of this application can dynamically capture and quantify the complex and nonlinear relationships between different modal features, rather than using fixed fusion rules. This makes the fusion process more efficient and reliable. The process can adapt to different task scenarios and the inherent laws of data. The unified multimodal fusion feature representation generated can more accurately reflect the essence of the enterprise's status, thereby significantly improving the accuracy of subsequent status analysis. Thirdly, enterprise status analysis based on the aforementioned high-quality and highly adaptable fusion feature representation results in monitoring outputs derived from in-depth collaborative mining of multi-source heterogeneous information. Therefore, it is more reliable than analysis results based on a single modality or simple fusion method. Furthermore, since the fusion weight correlation is learned, it provides a traceable basis for analysis decisions to a certain extent. Fourthly, based on the feedback information from the monitoring results, the usability assessment criteria used for modality screening and / or the intermodal correlations used for feature weighting integration are dynamically adjusted, forming a closed-loop optimization loop for the entire monitoring system. This mechanism allows the system to learn from historical monitoring experience and continuously correct its data selection and fusion strategies. Thus, it can maintain and continuously improve monitoring performance when facing data distribution drift, new modalities, or task changes, possessing robustness and adaptability for long-term application. In summary, the technical solution of this application achieves adaptiveness and improved accuracy in enterprise monitoring under multimodal data fusion through a closed-loop optimization mechanism.

[0236] Please see the appendix Figure 2This application provides a multimodal feature fusion enterprise data monitoring device, which may include an acquisition module 201, an extraction module 202, an evaluation module 203, a fusion module 204, an analysis module 205, and an adjustment module 206, as detailed below:

[0237] The acquisition module 201 is used to acquire the enterprise's multimodal monitoring data, wherein the multimodal monitoring data includes data from at least two different sources or formats;

[0238] The extraction module 202 is used to extract features from the enterprise's multimodal monitoring data to obtain an initial feature set corresponding to different modalities;

[0239] Evaluation module 203 is used to evaluate the usability of each modality under the current monitoring task based on the initial feature set of each modality, and thereby select the core modalities to participate in subsequent fusion.

[0240] The fusion module 204 is used to fuse the initial feature sets corresponding to the core modalities based on the learned intermodal relationships to generate a unified multimodal fusion feature representation;

[0241] Analysis module 205 is used to perform enterprise status analysis based on a unified multimodal fusion feature representation and output monitoring results;

[0242] The adjustment module 206 is used to adjust the availability assessment criteria for selecting core modalities to participate in subsequent fusion based on the feedback information of the monitoring results, and / or adjust the intermodal correlations for fusing the initial feature sets corresponding to the core modalities.

[0243] From the above appendix Figure 2As can be seen from the example of the enterprise data monitoring device for multimodal feature fusion, on the one hand, by acquiring and fusing multimodal monitoring data from at least two different sources or formats, it is possible to comprehensively utilize information from heterogeneous data sources within the enterprise, overcoming the limitations of a single modality data perspective. This provides a more comprehensive and three-dimensional data foundation for enterprise status analysis, thereby making subsequent monitoring results more comprehensive and reliable. Furthermore, by evaluating the usability of the initial feature sets of each modality under the current monitoring task and selecting core modalities to participate in subsequent fusion, low-quality and low-relevance modal data interference can be effectively filtered out, thus avoiding the problem of "garbage data in, garbage results out." This ensures that the data input to the fusion stage is high-value information that has undergone task adaptability screening, laying the foundation for generating high-quality fused feature representations. On the other hand, by weighted integration of the core modality features based on the learned inter-modal relationships, the technical solution of this application can dynamically capture and quantify the complex and nonlinear relationships between different modal features, rather than using fixed fusion rules. This makes the fusion process more efficient and reliable. The process can adapt to different task scenarios and the inherent laws of data. The unified multimodal fusion feature representation generated can more accurately reflect the essence of the enterprise's status, thereby significantly improving the accuracy of subsequent status analysis. Thirdly, enterprise status analysis based on the aforementioned high-quality and highly adaptable fusion feature representation results in monitoring outputs derived from in-depth collaborative mining of multi-source heterogeneous information. Therefore, it is more reliable than analysis results based on a single modality or simple fusion method. Furthermore, since the fusion weight correlation is learned, it provides a traceable basis for analysis decisions to a certain extent. Fourthly, based on the feedback information from the monitoring results, the usability assessment criteria used for modality screening and / or the intermodal correlations used for feature weighting integration are dynamically adjusted, forming a closed-loop optimization loop for the entire monitoring system. This mechanism allows the system to learn from historical monitoring experience and continuously correct its data selection and fusion strategies. Thus, it can maintain and continuously improve monitoring performance when facing data distribution drift, new modalities, or task changes, possessing robustness and adaptability for long-term application. In summary, the technical solution of this application achieves adaptiveness and improved accuracy in enterprise monitoring under multimodal data fusion through a closed-loop optimization mechanism.

[0244] Figure 3 This is a schematic diagram of the structure of a device provided in one embodiment of this application. For example... Figure 3 As shown, the device 3 in this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program for a multimodal feature fusion enterprise data monitoring method. When the processor 30 executes the computer program 32, it implements the steps in the above-described multimodal feature fusion enterprise data monitoring method embodiment, for example... Figure 1The steps S101 to S106 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of the acquisition module 201, extraction module 202, evaluation module 203, fusion module 204, analysis module 205, and adjustment module 206 are shown.

[0245] For example, the computer program 32 of the enterprise data monitoring method for multimodal feature fusion mainly includes: acquiring multimodal monitoring data of the enterprise, wherein the multimodal monitoring data includes data from at least two different sources or formats; extracting features from the multimodal monitoring data to obtain initial feature sets corresponding to different modalities; evaluating the usability of each initial feature set under the current monitoring task based on the initial feature set of each modality, and thereby selecting core modalities to participate in subsequent fusion; fusing the initial feature sets corresponding to the core modalities based on the learned intermodal relationships to generate a unified multimodal fusion feature representation; performing enterprise status analysis based on the multimodal fusion feature representation and outputting monitoring results; adjusting the usability evaluation criteria used to select core modalities to participate in subsequent fusion, and / or adjusting the intermodal relationships used to fuse the initial feature sets corresponding to the core modalities, based on the feedback information of the monitoring results. The computer program 32 can be divided into one or more modules / units, which are stored in memory 31 and executed by processor 30 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 32 in device 3. For example, computer program 32 can be divided into the functions of acquisition module 201, extraction module 202, evaluation module 203, fusion module 204, analysis module 205, and adjustment module 206 (a module in the virtual device). The specific functions of each module are as follows: Acquisition module 201 is used to acquire multimodal monitoring data of the enterprise, wherein the multimodal monitoring data includes data from at least two different sources or formats; Extraction module 202 is used to extract features from the multimodal monitoring data of the enterprise to obtain initial feature sets corresponding to different modalities; Evaluation module 203 is used to evaluate the performance of each modality based on the initial feature sets of each modality. The system assesses the availability of the current monitoring task and selects core modalities to participate in subsequent fusion based on this. The fusion module 204 is used to fuse the initial feature sets corresponding to the core modalities based on the learned inter-modal relationships to generate a unified multimodal fusion feature representation. The analysis module 205 is used to perform enterprise status analysis based on the unified multimodal fusion feature representation and output monitoring results. The adjustment module 206 is used to adjust the availability assessment criteria for selecting core modalities to participate in subsequent fusion and / or adjust the inter-modal relationships for fusing the initial feature sets corresponding to the core modalities based on the feedback information of the monitoring results.

[0246] Device 3 may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of device 3 and does not constitute a limitation on device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, the device may also include input / output devices, network access devices, buses, etc.

[0247] The processor 30 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0248] The memory 31 can be an internal storage unit of the device 3, such as a hard disk or RAM of the device 3. The memory 31 can also be an external storage device of the device 3, such as a plug-in hard disk, Smart MediaCard (SMC), Secure Digital (SD) card, or Flash Card equipped on the device 3. Furthermore, the memory 31 can include both internal and external storage units of the device 3. The memory 31 is used to store computer programs and other programs and data required by the device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0249] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed. That is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0250] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0251] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0252] In the embodiments provided in this application, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0253] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0254] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0255] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, all or part of the processes in the above-described embodiments of this application can also be implemented by a computer program instructing related hardware. The computer program for the enterprise data monitoring method of multimodal feature fusion can be stored in a storage medium. When the computer program is executed by a processor, it can implement the steps of the above-described method embodiments, namely: acquiring multimodal monitoring data of the enterprise, wherein the multimodal monitoring data includes data from at least two different sources or formats; extracting features from the multimodal monitoring data to obtain initial feature sets corresponding to different modalities; evaluating the usability of each initial feature set under the current monitoring task based on the initial feature sets of each modality, and thereby selecting the core modalities to participate in subsequent fusion; fusing the initial feature sets corresponding to the core modalities based on the learned intermodal relationships to generate a unified multimodal fusion feature representation; performing enterprise status analysis based on the multimodal fusion feature representation and outputting monitoring results; adjusting the usability evaluation criteria used to select the core modalities to participate in subsequent fusion, and / or adjusting the intermodal relationships used to fuse the initial feature sets corresponding to the core modalities, based on the feedback information of the monitoring results. Computer programs include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of storage media can be appropriately added or removed according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, storage media do not include electrical carrier signals and telecommunication signals.

[0256] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application. The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the protection scope of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A multi-modal feature fusion enterprise data monitoring method, characterized in that, The method includes: Step S1: Obtain the enterprise's multimodal monitoring data, which includes data from at least two different sources or formats; Step S2: Extract features from the multimodal monitoring data to obtain an initial feature set corresponding to different modes; Step S3: Based on the initial feature set of each modality, evaluate its usability under the current monitoring task, and select the core modalities to participate in subsequent fusion accordingly; Step S4: Based on the learned intermodal relationships, the initial feature sets corresponding to the core modalities are fused to generate a unified multimodal fusion feature representation; Step S5: Perform enterprise status analysis based on the multimodal fusion feature representation and output monitoring results; Step S6: Based on the feedback information of the monitoring results, adjust the usability assessment criteria used for screening in step S3, and / or adjust the intermodal correlations used for weighted integration in step S4.

2. The multi-modal feature fusion based enterprise data monitoring method as claimed in claim 1, wherein, Step S2 further includes the following steps of self-supervised pre-training and feature alignment: S21: Using unlabeled historical multimodal data collected from enterprises, construct a self-supervised learning task and pre-train the basic feature extraction network; S22: The pre-trained basic feature extraction network is used as a shared encoder to encode new input data of different modalities to obtain primary features of different modalities; S23: Design a cross-modal alignment module to align features of different modalities in the latent space by maximizing the mutual information between the primary features of different modalities or minimizing their distribution distance, and output the aligned initial feature set.

3. The multi-modal feature fusion based enterprise data monitoring method as claimed in claim 1, wherein, The evaluation of the usability of the initial feature set based on each modality for the current monitoring task includes: S31: Calculate the internal data quality index of the initial feature set of each modality, wherein the internal data quality index includes at least one of information entropy, missing rate and noise level; S32: Calculate the correlation index between the initial feature set of each modality and the current monitoring task objective; S33: Based on the preset decision rules, the internal data quality indicators and correlation indicators are combined to generate a quantitative availability score for each modality.

4. The enterprise data monitoring method based on multimodal feature fusion according to claim 3, characterized in that, The step of fusing the initial feature sets corresponding to the core modalities based on the learned inter-modal relationships to generate a unified multimodal fusion feature representation includes: S41: Assign initial fusion weights to each core modality based on the availability score; S42: Employ an attention-based fusion network to dynamically adjust the initial fusion weights based on the real-time interactions between the initial feature sets, thereby generating the final feature weighting coefficients; S43: Based on the feature weighting coefficients, perform weighted summation and nonlinear transformation on the initial feature set to generate the unified multimodal fusion feature representation.

5. The enterprise data monitoring method based on multimodal feature fusion according to claim 4, characterized in that, The fusion network employing an attention mechanism dynamically adjusts the initial fusion weights based on the real-time interactions between the initial feature sets to generate the final feature weighting coefficients, including: S421: Calculate query-key-value pairs, wherein the query vector is generated by the current monitoring task identifier, and the key-value pairs are obtained by linear transformation of the initial feature set of each modality; S422: Calculate the attention distribution between modalities based on the similarity between the query vector and each key vector; S423: Recalibrate the initial fusion weights based on the attention distribution to obtain the final feature weighting coefficients that reflect the dynamic cooperation relationship between modalities.

6. The enterprise data monitoring method based on multimodal feature fusion according to claim 1, characterized in that, The enterprise status analysis based on the multimodal fusion feature representation outputs monitoring results including: S51: Send the multimodal fusion feature representation to the local analysis nodes deployed in each business unit of the enterprise; S52: Each of the local analysis nodes, based on the received multimodal fusion feature representation, uses local private data to incrementally train the shared enterprise monitoring and analysis model, and calculates the model update gradient; S53: Aggregate the encrypted gradients uploaded by each local analysis node and update the parameters of the global enterprise monitoring and analysis model; S54: Distribute the updated global enterprise monitoring and analysis model to each local analysis node for analysis of subsequent multimodal fusion feature representations and output monitoring results.

7. The enterprise data monitoring method based on multimodal feature fusion according to claim 3, characterized in that, In step S6, adjusting the usability assessment criteria used for screening in step S3 based on the feedback information of the monitoring results includes the following causal enhancement adjustment steps: S61: Construct a simplified causal graph model that includes multimodal features as causes and monitoring results as effects; S62: When the monitoring results deviate, the causal graph model is intervened to fix or perturb the values ​​of one or more modal features; S63: Observe the changes in the predicted results after intervention, and use this to quantitatively estimate the causal effect index of each modality feature on the results; S64: Compare the statistical correlation index of each modality feature with the causal effect index; S65: If, according to the statistical correlation index and the causal effect index, the characteristics of a certain modality show high statistical correlation but low causal effect, then in the usability assessment criteria, its correlation-based weight is reduced, and a penalty term based on the stability of causal effect is introduced. S66: Use updated availability assessment criteria for modality screening to more readily identify core modalities that make a stable causal contribution to the monitoring results.

8. A multimodal feature fusion enterprise data monitoring device, characterized in that, The device includes: The acquisition module is used to acquire multimodal monitoring data of the enterprise, wherein the multimodal monitoring data includes data from at least two different sources or formats; The extraction module is used to extract features from the multimodal monitoring data to obtain an initial feature set corresponding to different modalities; The evaluation module is used to assess the usability of each modality under the current monitoring task based on the initial feature set of each modality, and to select the core modalities to participate in subsequent fusion accordingly. The fusion module is used to fuse the initial feature sets corresponding to the core modalities based on the learned inter-modal relationships to generate a unified multimodal fusion feature representation; The analysis module is used to perform enterprise status analysis based on the multimodal fusion feature representation and output monitoring results; The adjustment module is used to adjust the availability assessment criteria used for screening in step S3 and / or adjust the intermodal correlations used for weighted integration in step S4 based on the feedback information of the monitoring results.

9. An apparatus comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images