An industrial state monitoring method and device based on multi-modal collaborative perception

Abstract

This invention discloses an industrial condition monitoring method and device based on multimodal collaborative sensing, comprising: acquiring multimodal sample data of a target industrial environment; inputting the multimodal sample data into an initial multilayer sensing model to obtain a predicted sensing report vector; fine-tuning the initial multilayer sensing model based on the predicted sensing report vector using a Prompt-Tuning strategy to obtain a fine-tuned multilayer sensing model; splitting and deploying the fine-tuned multilayer sensing model to the cloud and edge, and dynamically and collaboratively optimizing the fine-tuned multilayer sensing model through a synchronization mechanism between the cloud and edge to obtain a target multilayer sensing model; collecting multimodal data in the target industrial environment, and obtaining industrial condition monitoring results based on the multimodal data and the target multilayer sensing model. This invention achieves closed-loop control of the entire process from data acquisition and feature alignment to real-time sensing, improving the accuracy of condition monitoring results.

Description

Technical Field

[0001] This invention relates to the field of industrial intelligent sensing and artificial intelligence fusion technology, and in particular to an industrial condition monitoring method and device based on multimodal collaborative sensing. Background Technology

[0002] With the rapid development of intelligent manufacturing, the Industrial Internet, and artificial intelligence technologies, industrial systems are accelerating their transition from traditional automation to a highly intelligent stage. Among these, the Industrial Agent, a key component of the industrial intelligence system, plays a central role in the closed loop of perception, reasoning, and decision-making. Specifically, by deeply integrating multi-source sensor data and knowledge models, the Industrial Agent possesses the capabilities for autonomous environmental perception, state recognition, anomaly diagnosis, and predictive control, providing intelligent support for complex manufacturing processes, equipment maintenance, and energy management. However, industrial scenarios are inherently highly complex and dynamic, containing massive amounts of heterogeneous data from various sources, including visual images, acoustic signals, vibration waveforms, process control parameters, text logs, and temperature, humidity, and pressure sensors. This data exhibits strong time-varying, non-stationary, and high-noise characteristics. Therefore, achieving reliable industrial condition monitoring in multimodal and highly dynamic environments is one of the core technical challenges in building Industrial Agents.

[0003] In related technologies, temporal models based on convolutional neural networks (CNN), long short-term memory networks (LSTM), or Transformers are used to obtain state monitoring results. However, these single-modal algorithms struggle to provide reliable multi-dimensional cognitive capabilities, thus reducing the accuracy of state monitoring results. Furthermore, related technologies employ post-decision-level fusion or simple feature concatenation, failing to achieve deep alignment of the underlying semantics of the data. This results in fragmented feature spaces, redundancy of modal information, and insufficient utilization of complementary advantages, further reducing the accuracy of state monitoring results. Alternatively, Multimodal Large Models (MLLMs) are used to learn universal semantic representation capabilities across vision and language through comparative learning and self-supervised pre-training on massive amounts of internet image and text data. However, these universal multimodal models are trained on open-domain network data and lack embedding of the physical mechanisms of industrial equipment, fault evolution patterns, and professional knowledge systems. Their semantic space deviates from the precise requirements of industrial tasks, thus reducing the accuracy of state monitoring results. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, one objective of this invention is to propose an industrial condition monitoring method based on multimodal collaborative sensing. This method can obtain a fine-tuned multi-layer sensing model through a Prompt-Tuning strategy, and use the fine-tuned multi-layer sensing model to obtain industrial condition monitoring results through multimodal data in the target industrial environment. Thus, the fine-tuned multi-layer sensing model can be used to fuse multi-source sensor data, establish a cross-modal semantic representation space, and combine spatiotemporal consistency modeling and lightweight deployment strategies to achieve closed-loop control of the entire process from data acquisition and feature alignment to real-time sensing, thereby improving the accuracy of condition monitoring results.

[0006] Another objective of this invention is to propose an industrial condition monitoring device based on multimodal collaborative sensing.

[0007] To achieve the above objectives, one embodiment of the present invention proposes an industrial condition monitoring method based on multimodal collaborative sensing, comprising:

[0008] Acquire multimodal sample data of the target industrial environment; The multimodal sample data is input into the initial multilayer sensing model to obtain the predicted sensing report vector; The initial multilayer sensing model is fine-tuned based on the predicted sensing report vector using the Prompt-Tuning strategy to obtain the fine-tuned multilayer sensing model. The fine-tuned multilayer perception model is split and deployed to the cloud and the edge, and the fine-tuned multilayer perception model is dynamically and collaboratively optimized through the synchronization mechanism between the cloud and the edge to obtain the target multilayer perception model. Multimodal data is collected from the target industrial environment, and industrial condition monitoring results are obtained based on the multimodal data through the target multilayer sensing model.

[0009] The industrial condition monitoring method based on multimodal collaborative sensing in this invention may also have the following additional technical features: Further, the step of inputting the multimodal sample data into the initial multilayer sensing model to obtain the predicted sensing report vector includes: The multimodal sample data is sliced ​​at a time granularity and synchronized to obtain processed multimodal data; The features of the processed multimodal data are extracted and encoded to obtain a multimodal embedding matrix; The multimodal embedding matrix is ​​input into the cross-modal feature alignment network to obtain the first fusion feature with semantic alignment; The semantically aligned first fusion feature is input into the multimodal collaborative perception model to obtain the multimodal result; The multimodal results are structured and analyzed to generate a predictive perception report vector containing the state evolution path.

[0010] Further, the step of extracting and encoding the features of the processed multimodal data to obtain a multimodal embedding matrix includes: The continuous signal-type modes in the processed multimodal data are subjected to short-time Fourier transform and Mel-frequency transform to obtain feature tensors, and the feature tensors are projected to obtain a first feature vector; Spatial feature vectors of visual image modalities are extracted from the processed multimodal data using a convolutional neural network or a visual Transformer, and the spatial feature vectors are mapped to the embedding space to obtain a second feature vector. The text data in the processed multimodal data is vectorized using a semantic model to obtain a third feature vector; The first feature vector, the second feature vector, and the third feature vector are processed by linear mapping and regularization to obtain the multimodal embedding matrix.

[0011] Further, the cross-modal feature alignment network includes a feature extraction layer and a mapping layer; the step of inputting the multimodal embedding matrix into the cross-modal feature alignment network to obtain the first fused feature for semantic alignment includes: The feature extraction layer extracts preliminary features of each modality in the multimodal embedding matrix; The preliminary features of each modality are projected onto a unified semantic embedding space through the mapping layer to obtain the semantic embedding of each modality. The semantic embeddings of each modality at the prediction time step are aggregated to obtain the first semantically aligned fusion feature.

[0012] Furthermore, the multimodal cooperative sensing model includes Layer embedding transformation and scale feature extraction layer; the first fusion feature with semantic alignment is input into the multimodal collaborative perception model to obtain multimodal results, including: Through the above The layered embedding transformation performs a layered nonlinear transformation on the input features of each modality in the semantically aligned first fusion feature to generate multi-scale embedding representation features; The attention score of the embedding representation feature of each layer is calculated through the scale feature extraction layer, and the attention score is normalized to obtain the normalized attention weight. The embedding of all modalities and all layers is weighted and fused based on the attention weight to obtain the second fused feature. Perform a dimensionality compression operation on the second fused feature to obtain a multimodal result.

[0013] Further, the step of performing structured parsing on the multimodal results to generate a prediction-aware report vector containing the state evolution path includes: The multimodal results are mapped into structured result vectors using a semantic mapping function; Construct a modal importance matrix; Based on the structured result vector and the modal importance matrix, a predictive perception report vector for the state evolution path is generated.

[0014] Furthermore, the step of fine-tuning the initial multilayer sensing model based on the predicted sensing report vector using the Prompt-Tuning strategy to obtain the fine-tuned multilayer sensing model includes: Obtain the actual annotation results corresponding to the multimodal sample data; A trainable Prompt vector is introduced into the intermediate layer features of the initial multilayer perception model, and the model parameters are updated using the Prompt vector based on the real annotation results and the predicted perception report vector to obtain a fine-tuned multilayer perception model.

[0015] To achieve the above objectives, another embodiment of the present invention proposes an industrial condition monitoring device based on multimodal collaborative sensing, the device comprising: The acquisition module is used to acquire multimodal sample data of the target industrial environment; The prediction module is used to input the multimodal sample data into the initial multilayer sensing model to obtain the predicted sensing report vector; The fine-tuning module is used to fine-tune the initial multilayer perception model based on the predicted perception report vector using the Prompt-Tuning strategy to obtain the fine-tuned multilayer perception model. The deployment module is used to split and deploy the fine-tuned multi-layer perception model to the cloud and the edge, and dynamically and collaboratively optimize the fine-tuned multi-layer perception model through the synchronization mechanism between the cloud and the edge to obtain the target multi-layer perception model. The condition monitoring module is used to collect multimodal data in the target industrial environment and obtain industrial condition monitoring results based on the multimodal data through the target multilayer sensing model.

[0016] This invention proposes an industrial condition monitoring method and device based on multimodal collaborative sensing. The method involves acquiring multimodal sample data of the target industrial environment; inputting the multimodal sample data into an initial multilayer sensing model to obtain a predicted sensing report vector; fine-tuning the initial multilayer sensing model using a Prompt-Tuning strategy based on the predicted sensing report vector to obtain a fine-tuned multilayer sensing model; splitting and deploying the fine-tuned multilayer sensing model to the cloud and edge, and dynamically and collaboratively optimizing the fine-tuned multilayer sensing model through a synchronization mechanism between the cloud and edge to obtain a target multilayer sensing model; collecting multimodal data from the target industrial environment, and obtaining industrial condition monitoring results based on the multimodal data and the target multilayer sensing model. Therefore, this invention can fuse multi-source sensor data through a fine-tuned multilayer sensing model to establish a cross-modal semantic representation space, and combined with spatiotemporal consistent modeling and lightweight deployment strategies, achieve closed-loop control of the entire process from data acquisition and feature alignment to real-time sensing, thereby improving the accuracy of condition monitoring results.

[0017] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0018] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart of an industrial condition monitoring method based on multimodal collaborative sensing according to an embodiment of the present invention; Figure 2 This is a schematic diagram of an industrial condition monitoring device based on multimodal collaborative sensing according to an embodiment of the present invention. Detailed Implementation

[0019] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0020] The following description, with reference to the accompanying drawings, describes an industrial condition monitoring method and apparatus based on multimodal collaborative sensing according to embodiments of the present invention.

[0021] First, the industrial condition monitoring method based on multimodal collaborative sensing proposed according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart of an industrial condition monitoring method based on multimodal collaborative sensing according to an embodiment of the present invention.

[0023] like Figure 1 As shown, the industrial condition monitoring method based on multimodal collaborative sensing includes the following steps: Step S1: Obtain multimodal sample data of the target industrial environment.

[0024] It should be noted that the execution subject of the above-mentioned industrial condition monitoring method based on multimodal collaborative sensing is an industrial condition monitoring device based on multimodal collaborative sensing. This industrial condition monitoring device based on multimodal collaborative sensing can be implemented by software and / or hardware. In this embodiment, the industrial condition monitoring device based on multimodal collaborative sensing can be configured in an electronic device.

[0025] In one embodiment of the present invention, the aforementioned multimodal sample data may include various types of sensor data. Specifically, in one embodiment, the various types of sensor data may include visual images acquired by a high frame rate industrial camera, noise and collision signals collected by an acoustic sensor, vibration waveforms recorded by an accelerometer, time-series operating parameters acquired by temperature and pressure sensors, and textual semantic data formed from system operation logs, control commands, and manual operation records.

[0026] Step S2: Input the multimodal sample data into the initial multilayer sensing model to obtain the predicted sensing report vector.

[0027] In one embodiment of the present invention, after obtaining multimodal sample data through the above steps, the multimodal sample data can be input into the initial multilayer perception model to obtain the predicted perception report vector.

[0028] In one embodiment of the present invention, the method for inputting multimodal sample data into an initial multilayer sensing model to obtain a predicted sensing report vector may include the following steps: Step S21: Perform time-granular slicing and synchronization processing on the multimodal sample data to obtain the processed multimodal data.

[0029] In one embodiment of the present invention, in the time domain The original observation sequence corresponding to the multimodal sample data is as follows: ,

[0030] in, This represents the observation vector of the i-th mode at time t. The feature dimension is the corresponding modality.

[0031] In one embodiment of the present invention, the method for performing time-granular slicing and synchronization processing on multimodal sample data to obtain processed multimodal data may include: performing preprocessing operations on the multimodal sample data to obtain preprocessed multimodal data, and aligning the preprocessed multimodal data to a unified time axis through a synchronization operator to obtain processed multimodal data.

[0032] In one embodiment of the present invention, since different sensing links have differences in sampling rate, response delay, and acquisition noise, a unified preprocessing operation is required on the original observation sequence to obtain preprocessed multimodal data:

[0033] in, This represents the preprocessing function, which sequentially performs denoising, missing data completion, and scale normalization operations to ensure that the data from each modality are comparable in terms of numerical distribution and sampling interval.

[0034] Furthermore, in one embodiment of the present invention, considering the possible time delay or sampling drift between different modes, their relative offset is assumed to be... It can be achieved through synchronization operators The preprocessed multimodal data is aligned to a unified time axis to obtain the processed multimodal data:

[0035] in, It is a time synchronization function used to eliminate sampling delay based on timestamp difference, trigger signal or dynamic time warping (DTW) to ensure that all modes correspond consistently in the time domain.

[0036] Step S22: Extract the features of the processed multimodal data and encode them to obtain the multimodal embedding matrix.

[0037] In one embodiment of the present invention, after obtaining the processed multimodal data through the above steps, the features of the processed multimodal data can be extracted and encoded to obtain a multimodal embedding matrix.

[0038] In one embodiment of the present invention, the method for extracting and encoding the features of the processed multimodal data to obtain a multimodal embedding matrix may include the following steps: Step S221: Perform short-time Fourier transform and Mel spectrum transform on the continuous signal type mode in the processed multimodal data to obtain the feature tensor, and project the feature tensor to obtain the first feature vector; In one embodiment of the present invention, the aforementioned continuous signal modal can be acoustic and vibration data.

[0039] In one embodiment of the present invention, the window length is set to... Step size is The time spectrum is calculated using a Mel filter bank with short-time Fourier transform (STFT) and Mel spectrum transform. The frequency band dimension is obtained as The number of time frames is Feature tensor Then it is passed through a lightweight convolution or Transformer encoder. The first eigenvector is obtained by projecting the feature tensor. .

[0040] Step S222: Use a convolutional neural network or visual Transformer to extract spatial feature vectors of visual image modalities from the processed multimodal data, and map the spatial feature vectors to the embedding space to obtain the second feature vector; In one embodiment of the present invention, for visual image modalities in the processed multimodal data, distortion correction and scale normalization can be performed first, and then spatial feature vectors can be extracted using a convolutional neural network (CNN) or a visual transformer (ViT). The spatial feature vectors are then mapped to the embedding space to obtain a second feature vector. ,in, This can be either a Convolutional Neural Network (CNN) or a Visual Transformer (ViT). This is a visual image modality.

[0041] Step S223: Use a semantic model to vectorize the text data in the processed multimodal data to obtain the third feature vector; In one embodiment of the present invention, for the processed multimodal data in text, the corresponding system events or operation records can be aligned according to the timestamps, and a third feature vector can be obtained using a semantic model. ,in, For semantic model, It is text data.

[0042] In one embodiment of the present invention, the semantic model described above can be a language model trained on industrial corpora, such as BERT-Industrial.

[0043] Step S224: The first eigenvector, the second eigenvector, and the third eigenvector are processed by linear mapping and regularization to obtain the multimodal embedding matrix.

[0044] In one embodiment of the present invention, after obtaining the first feature vector, the second feature vector and the third feature vector through the above steps, the first feature vector, the second feature vector and the third feature vector can be processed by linear mapping and regularization to obtain a multimodal embedding matrix.

[0045] In one embodiment of the present invention, the first feature vector, the second feature vector, and the third feature vector are mapped through a linear mapping matrix. With normalization function The process is performed to obtain the embedding matrix for each mode. The resulting embedding matrices are then concatenated sequentially along their feature dimensions to form a multimodal embedding matrix:

[0046] In one embodiment of the present invention, the above steps are used to obtain... back, It can be used as an input sample for subsequent multimodal feature fusion and alignment calculations to achieve semantic consistency and spatiotemporal correspondence of cross-modal information.

[0047] Step S23: Input the multimodal embedding matrix into the cross-modal feature alignment network to obtain the first fusion feature with semantic alignment.

[0048] In one embodiment of the present invention, after obtaining the multimodal embedding matrix through the above steps, the multimodal embedding matrix can be input into the cross-modal feature alignment network to obtain the first fusion feature with semantic alignment.

[0049] In one embodiment of the present invention, the cross-modal feature alignment network described above may include a feature extraction layer and a mapping layer.

[0050] In one embodiment of the present invention, the method of inputting a multimodal embedding matrix into a cross-modal feature alignment network to obtain a semantically aligned first fused feature may include the following steps: Step S231: Extract preliminary features of each modality in the multimodal embedding matrix through the feature extraction layer; In one embodiment of the present invention, the feature extraction layer uses a modal feature extraction function. Extracting preliminary features of each mode from the multimodal embedding matrix. .

[0051] Step S232: Project the preliminary features of each modality onto a unified semantic embedding space through a mapping layer to obtain the semantic embedding of each modality; In one embodiment of the present invention, the mapping layer maps the preliminary features of each mode using linear or nonlinear mapping operators. Projected onto a unified semantic embedding space In this process, the semantic embeddings of each modality are obtained as follows: ,in, , .

[0052] In one embodiment of the present invention, an alignment loss function is introduced during the training process of the cross-modal feature alignment network. This is used to measure the similarity of different modalities embedded in a shared space.

[0053] Specifically, in one embodiment of the present invention, for the time domain For all observed samples, the alignment loss function is defined as:

[0054] in, is the Euclidean norm, used to measure the distance between cross-modal embeddings.

[0055] Step S233: Aggregate the semantic embeddings of each modality at the prediction time step to obtain the first fusion feature with semantic alignment.

[0056] In one embodiment of the present invention, after obtaining the semantic embeddings of each modality through the above steps, the second semantic embeddings of each modality at the prediction time step can be aggregated to obtain the first fusion feature with semantic alignment.

[0057] Specifically, in one embodiment of the invention, the semantic embedding of all modalities is time-series. Aggregation is performed at the point to obtain the first fusion feature of semantic alignment. .

[0058] Furthermore, in one embodiment of the present invention, the first fused feature can serve as subsequent input, providing a semantically consistent feature basis for downstream spatiotemporal perception and reasoning.

[0059] Step S24: Input the semantically aligned first fusion feature into the multimodal collaborative perception model to obtain the multimodal result.

[0060] In one embodiment of the present invention, after obtaining the semantically aligned first fusion feature through the above steps, the semantically aligned first fusion feature can be input into the multimodal collaborative perception model to obtain multimodal results, so as to capture the semantic association of multi-source features in the time and modality dimensions, thereby realizing dynamic prediction and anomaly identification of device operating status.

[0061] In one embodiment of the present invention, the multimodal cooperative sensing model may include Layer embedding transformation and scale feature extraction layer.

[0062] In one embodiment of the present invention, the method of inputting the semantically aligned first fusion feature into a multimodal collaborative perception model to obtain a multimodal result may include the following steps: Step S241, through Layered embedding transformation performs a layered nonlinear transformation on the input features of each modality in the semantically aligned first fusion feature to generate multi-scale embedding representation features; In one embodiment of the present invention, the layer embedding transformation corresponding to each modality applies a layer-by-layer nonlinear transformation to the input features of each modality in the semantically aligned first fusion feature, to obtain the first... Multi-scale embedding representation features of layers:

[0063] in, For element-wise nonlinear activation functions (such as...) or ), For the first Layer trainable weight matrix For bias terms, This represents the time delay term, used to capture the changing trends of multimodal data across adjacent time slices.

[0064] In one embodiment of the present invention, the dynamic change patterns of multimodal features can be captured at different time scales through the above-described hierarchical mapping operation.

[0065] Step S242: Calculate the attention score of the embedding representation feature of each layer through the scale feature extraction layer, and normalize the attention score to obtain the normalized attention weight. Based on the attention weight, perform weighted fusion of the embeddings of all modalities and all layers to obtain the second fused feature. In one embodiment of the present invention, for the first Layer embedding, calculate its attention score:

[0066] in, This is a trainable scoring vector used to measure the importance of features in this layer.

[0067] In one embodiment of the present invention, the attention score is normalized to obtain the normalized attention weight:

[0068] in, Indicates that the i-th mode is in the... The normalized attention weights of the layers are used to balance the contributions of different layers and different modal features.

[0069] Furthermore, in one embodiment of the present invention, a second fusion feature is obtained by weighted fusion of the embeddings of all modalities and all layers based on the aforementioned attention weights:

[0070] In one embodiment of the present invention, the above-mentioned fusion features It integrates semantic information from visual, acoustic, vibrational, and textual modalities at different times and scales, ensuring the consistency of multi-source features in the shared embedding space.

[0071] Step S243: Perform dimensionality compression on the second fused feature to obtain the multimodal result.

[0072] In one embodiment of the present invention, after obtaining the second fusion feature through the above steps, a dimensionality compression operation can be performed on the second fusion feature to obtain a multimodal result, which further improves the real-time reasoning capability at the edge.

[0073] In one embodiment of the present invention, a compression mapping function is defined. ,in, This is used to reduce computational load while preserving key semantic information under task requirements.

[0074] In one embodiment of the present invention, a dimensionality compression operation is performed on the second fusion feature using the aforementioned compression mapping function to obtain a multimodal result. .

[0075] In one embodiment of the present invention, when the task requires complete semantic reasoning, It can be an identity mapping; otherwise, perform linear or nonlinear dimensionality reduction operations. And, As the output of the fusion representation, it is used in subsequent state classification, anomaly detection and trend prediction modules to realize spatiotemporal perception and intelligent decision-making of industrial equipment.

[0076] Step S25: Perform structured analysis on the multimodal results to generate a prediction-aware report vector containing the state evolution path.

[0077] In one embodiment of the present invention, after obtaining the multimodal results through the above steps, the multimodal results can be structured and analyzed to generate a predictive perception report vector containing the state evolution path, so as to form a traceable and interpretable state cognition report, providing support for the diagnosis, control and decision-making of industrial intelligent agents.

[0078] In one embodiment of the present invention, the method for performing structured analysis on multimodal results to generate a predictive perception report vector containing state evolution paths may include the following steps: Step S251: Map the multimodal results into a structured result vector using a semantic mapping function; In one embodiment of the present invention, the multimodal results are mapped into a structured result vector using a semantic mapping function as follows:

[0079] in, Represents a semantic mapping function. Label the equipment with a serial number. The results of the operation status classification For the type of exception or fault category, This refers to the corresponding time window or duration period.

[0080] In one embodiment of the present invention, through semantic mapping operations, high-dimensional embeddings are projected into an understandable engineering semantic space, thereby realizing the translation from "model output" to "business information".

[0081] Step S252: Construct the modal importance matrix; In one embodiment of the present invention, a modality importance matrix can be constructed based on the aforementioned attention weights. ,in, It reflects the combined weight of each modality at the current moment, and is used to explain the relative contribution of the model's decision sources and multi-source data.

[0082] Step S253: Based on the structured result vector and modal importance matrix, generate a predicted perception report vector for the state evolution path.

[0083] In one embodiment of the present invention, after obtaining the structured result vector and modal importance matrix through the above steps, a prediction perception report vector of the state evolution path can be generated based on the structured result vector and modal importance matrix.

[0084] In one embodiment of the present invention, the predicted perception report vector is: ,in, This represents the predictive perception report vector, which contains the entire chain of information from the identification results to the interpretable weights, supporting quantitative analysis and visualization of device status.

[0085] Step S3: The initial multilayer sensing model is fine-tuned based on the predicted sensing report vector using the Prompt-Tuning strategy to obtain the fine-tuned multilayer sensing model.

[0086] In one embodiment of the present invention, after obtaining the predicted perception report vector through the above steps, the Prompt-Tuning strategy can be used to fine-tune the initial multilayer perception model based on the predicted perception report vector to obtain the fine-tuned multilayer perception model.

[0087] In one embodiment of the present invention, the method for fine-tuning the initial multilayer sensing model based on the predicted sensing report vector using the Prompt-Tuning strategy to obtain the fine-tuned multilayer sensing model may include the following steps: Step S31: Obtain the actual annotation results corresponding to the multimodal sample data; Step S32: Introduce trainable Prompt vectors into the intermediate layer features of the initial multilayer perception model, and update the model parameters based on the real annotation results and the predicted perception report vectors through the Prompt vectors to obtain the fine-tuned multilayer perception model.

[0088] In one embodiment of the present invention, the method for updating model parameters based on real annotation results and predicted perception report vectors using a Prompt vector to obtain a fine-tuned multilayer perception model may include: updating model parameters based on real annotation results and predicted perception report vectors using a task loss function based on the Prompt vector to obtain a fine-tuned multilayer perception model, wherein the task loss function is:

[0089] in, For true annotation results, To predict the perceived report vector for the model, This represents the task loss function, which can be selected as mean squared error or cross-entropy depending on the task type.

[0090] In one embodiment of the present invention, guided by the Prompt vector, the model can quickly adapt to the semantic feature distribution under different working conditions without changing the main structure.

[0091] Step S4: The fine-tuned multilayer perception model is split and deployed to the cloud and edge, and the fine-tuned multilayer perception model is dynamically and collaboratively optimized through the synchronization mechanism between the cloud and the edge to obtain the target multilayer perception model.

[0092] In one embodiment of the present invention, a cloud-edge collaborative operation architecture is constructed to achieve a collaborative perception mechanism of "cloud learning - edge execution". Specifically, in one embodiment, cloud-based global optimization and edge model updates achieve dynamic collaboration and knowledge transfer through a parameter synchronization mechanism. Furthermore, the cloud model... Responsible for global training and complex computation, edge agent model Responsible for real-time inference and local decision-making, the cloud and edge can achieve collaborative optimization through periodic parameter synchronization:

[0093] in, This is the synchronization ratio coefficient. This provides an index for edge nodes. Furthermore, by employing the aforementioned parameter synchronization strategy, while ensuring real-time performance on the edge side, it continuously absorbs updated global knowledge from the cloud, enabling dynamic collaborative optimization of the cloud-edge model.

[0094] Furthermore, in one embodiment of the present invention, during model operation, incremental learning and online feedback mechanisms can be combined to dynamically optimize feature alignment and semantic mapping accuracy based on real-time perceived data. Additionally, the semantic mapping function and embedding projection parameters can be adaptively fine-tuned using prediction errors uploaded from edge nodes and changes in attention weights to minimize dynamic errors.

[0095] in, This represents the real-time sampling window. Furthermore, through continuous iteration and feedback updates, the model can maintain robustness and high accuracy over the long term, achieving sustainable self-learning and adaptive evolution capabilities for industrial intelligent agents.

[0096] In one embodiment of the present invention, the above-described target multilayer perception model can be run in the perception and reasoning module of an industrial intelligent agent.

[0097] Furthermore, in one embodiment of the present invention, edge devices can be deployed locally to resolve transmission latency, while cloud computing power can be centralized to ensure global optimization, thereby supporting the requirements for low latency and real-time perception, and thus achieving a balance between fast response and high intelligence.

[0098] Step S5: Collect multimodal data in the target industrial environment, and obtain industrial condition monitoring results based on the multimodal data through the target multilayer sensing model.

[0099] In one embodiment of the present invention, after collecting multimodal data in the target industrial environment, the multimodal data can be input into the target multilayer perception model to obtain the target perception report vector, and a comprehensive perception report can be generated based on the target perception report vector. The comprehensive perception report includes the equipment state evolution path, main abnormal modes and key influencing factors, and is displayed in the form of charts or time series.

[0100] Furthermore, in one embodiment of the present invention, the comprehensive perception report is transmitted to the industrial operation and maintenance platform in real time through a system interface, enabling linkage with the monitoring system, early warning system, and control strategy module. Simultaneously, key features and labeling results in the report are sent back to the data processing module as new labeled samples for iterative optimization of the subsequent model training process, thereby constructing a continuously evolving, self-learning industrial perception system.

[0101] According to the industrial condition monitoring method based on multimodal collaborative sensing proposed in this invention, multimodal sample data of the target industrial environment is acquired; the multimodal sample data is input into an initial multilayer sensing model to obtain a predicted sensing report vector; a Prompt-Tuning strategy is used to fine-tune the initial multilayer sensing model based on the predicted sensing report vector to obtain a fine-tuned multilayer sensing model; the fine-tuned multilayer sensing model is split and deployed to the cloud and edge, and the fine-tuned multilayer sensing model is dynamically and collaboratively optimized through a synchronization mechanism between the cloud and the edge to obtain a target multilayer sensing model; multimodal data in the target industrial environment is collected, and industrial condition monitoring results are obtained based on the multimodal data and the target multilayer sensing model. Therefore, this invention can fuse multi-source sensor data through a fine-tuned multilayer sensing model to establish a cross-modal semantic representation space, and combined with spatiotemporal consistency modeling and lightweight deployment strategies, achieve closed-loop control of the entire process from data acquisition and feature alignment to real-time sensing, thereby improving the accuracy of condition monitoring results.

[0102] Next, referring to the accompanying drawings, an industrial condition monitoring device based on multimodal collaborative sensing proposed according to an embodiment of the present invention is described.

[0103] Figure 2 This is a schematic diagram of an industrial condition monitoring device based on multimodal collaborative sensing according to an embodiment of the present invention.

[0104] like Figure 2 As shown, the industrial condition monitoring device 10 based on multimodal collaborative sensing includes: an acquisition module 201, a prediction module 202, a fine-tuning module 203, a deployment module 204, and a condition monitoring module 205, wherein, The acquisition module 201 is used to acquire multimodal sample data of the target industrial environment; Prediction module 202 is used to input multimodal sample data into the initial multilayer sensing model to obtain the prediction sensing report vector; The fine-tuning module 203 is used to fine-tune the initial multilayer sensing model based on the predicted sensing report vector using the Prompt-Tuning strategy to obtain the fine-tuned multilayer sensing model. The deployment module 204 is used to split and deploy the fine-tuned multi-layer perception model to the cloud and the edge, and dynamically and collaboratively optimize the fine-tuned multi-layer perception model through the synchronization mechanism between the cloud and the edge to obtain the target multi-layer perception model. The condition monitoring module 205 is used to collect multimodal data in the target industrial environment and obtain industrial condition monitoring results based on the multimodal data through the target multilayer sensing model.

[0105] Furthermore, the aforementioned prediction module 202 is specifically used for: The multimodal sample data is sliced ​​at a time granularity and synchronized to obtain the processed multimodal data; Features of the processed multimodal data are extracted and encoded to obtain a multimodal embedding matrix; The multimodal embedding matrix is ​​input into the cross-modal feature alignment network to obtain the first fused feature with semantic alignment; The semantically aligned first fusion feature is input into the multimodal collaborative perception model to obtain multimodal results; The multimodal results are structured and analyzed to generate a predictive perception report vector containing the state evolution path.

[0106] Furthermore, the aforementioned prediction module 202 is also used for: The continuous signal modalities in the processed multimodal data are subjected to short-time Fourier transform and Mel spectrum transform to obtain feature tensors, and the feature tensors are projected to obtain the first eigenvector. Spatial feature vectors of visual image modalities are extracted from the processed multimodal data using convolutional neural networks or visual Transformers, and the spatial feature vectors are mapped to the embedding space to obtain the second feature vector; The text data in the processed multimodal data is vectorized using a semantic model to obtain the third feature vector; The first, second, and third eigenvectors are processed by linear mapping and regularization to obtain the multimodal embedding matrix.

[0107] Furthermore, the aforementioned cross-modal feature alignment network includes a feature extraction layer and a mapping layer; the prediction module 202 is also used for: Preliminary features of each modality in the multimodal embedding matrix are extracted through a feature extraction layer; The preliminary features of each modality are projected onto a unified semantic embedding space through a mapping layer to obtain the semantic embedding of each modality. The semantic embeddings of each modality at the prediction time step are aggregated to obtain the first semantically aligned fusion feature.

[0108] Furthermore, the aforementioned multimodal collaborative sensing model includes Layer embedding transformation and scale feature extraction layer; prediction module 402 is also used for: pass Layered embedding transformation performs a layered nonlinear transformation on the input features of each modality in the semantically aligned first fusion feature to generate multi-scale embedding representation features; The attention score of the embedding representation feature of each layer is calculated by the scale feature extraction layer, and the attention score is normalized to obtain the normalized attention weight. The embedding of all modalities and all layers is weighted and fused based on the attention weight to obtain the second fused feature. Perform dimensionality compression on the second fusion feature to obtain multimodal results.

[0109] Furthermore, the aforementioned prediction module 202 is also used for: Multimodal results are mapped into structured result vectors using a semantic mapping function; Construct a modal importance matrix; Based on the structured result vector and modal importance matrix, a predictive perception report vector for the state evolution path is generated.

[0110] Furthermore, the aforementioned fine-tuning module 203 is specifically used for: Obtain the true annotation results corresponding to multimodal sample data; A trainable Prompt vector is introduced into the intermediate layer features of the initial multilayer perception model, and the model parameters are updated using the Prompt vector based on the real annotation results and the predicted perception report vector, resulting in a fine-tuned multilayer perception model.

[0111] According to an embodiment of the present invention, an industrial condition monitoring device based on multimodal collaborative sensing acquires multimodal sample data of the target industrial environment; inputs the multimodal sample data into an initial multilayer sensing model to obtain a predicted sensing report vector; employs a Prompt-Tuning strategy to fine-tune the initial multilayer sensing model based on the predicted sensing report vector to obtain a fine-tuned multilayer sensing model; splits and deploys the fine-tuned multilayer sensing model to the cloud and edge, and dynamically and collaboratively optimizes the fine-tuned multilayer sensing model through a synchronization mechanism between the cloud and edge to obtain a target multilayer sensing model; collects multimodal data from the target industrial environment, and obtains industrial condition monitoring results based on the multimodal data and the target multilayer sensing model. Therefore, the present invention can fuse multi-source sensor data through a fine-tuned multilayer sensing model to establish a cross-modal semantic representation space, and, combined with spatiotemporal consistency modeling and lightweight deployment strategies, achieves closed-loop control of the entire process from data acquisition and feature alignment to real-time sensing, thereby improving the accuracy of condition monitoring results.

[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0113] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0114] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An industrial condition monitoring method based on multimodal collaborative sensing, characterized in that, The method includes: Acquire multimodal sample data of the target industrial environment; The multimodal sample data is input into the initial multilayer sensing model to obtain the predicted sensing report vector; The initial multilayer sensing model is fine-tuned based on the predicted sensing report vector using the Prompt-Tuning strategy to obtain the fine-tuned multilayer sensing model. The fine-tuned multilayer perception model is split and deployed to the cloud and the edge, and the fine-tuned multilayer perception model is dynamically and collaboratively optimized through the synchronization mechanism between the cloud and the edge to obtain the target multilayer perception model. Multimodal data is collected from the target industrial environment, and industrial condition monitoring results are obtained based on the multimodal data through the target multilayer sensing model.

2. The method according to claim 1, characterized in that, The step of inputting the multimodal sample data into the initial multilayer sensing model to obtain the predicted sensing report vector includes: The multimodal sample data is sliced ​​at a time granularity and synchronized to obtain processed multimodal data; The features of the processed multimodal data are extracted and encoded to obtain a multimodal embedding matrix; The multimodal embedding matrix is ​​input into the cross-modal feature alignment network to obtain the first fusion feature with semantic alignment; The semantically aligned first fusion feature is input into the multimodal collaborative perception model to obtain the multimodal result; The multimodal results are structured and analyzed to generate a predictive perception report vector containing the state evolution path.

3. The method according to claim 2, characterized in that, The step of extracting and encoding features from the processed multimodal data to obtain a multimodal embedding matrix includes: The continuous signal-type modes in the processed multimodal data are subjected to short-time Fourier transform and Mel-frequency transform to obtain feature tensors, and the feature tensors are projected to obtain a first feature vector; Spatial feature vectors of visual image modalities are extracted from the processed multimodal data using a convolutional neural network or a visual Transformer, and the spatial feature vectors are mapped to the embedding space to obtain a second feature vector. The text data in the processed multimodal data is vectorized using a semantic model to obtain a third feature vector; The first feature vector, the second feature vector, and the third feature vector are processed by linear mapping and regularization to obtain the multimodal embedding matrix.

4. The method according to claim 2, characterized in that, The cross-modal feature alignment network includes a feature extraction layer and a mapping layer; the step of inputting the multimodal embedding matrix into the cross-modal feature alignment network to obtain the first fused feature for semantic alignment includes: The feature extraction layer extracts preliminary features of each modality in the multimodal embedding matrix; The preliminary features of each modality are projected onto a unified semantic embedding space through the mapping layer to obtain the semantic embedding of each modality. The semantic embeddings of each modality at the prediction time step are aggregated to obtain the first semantically aligned fusion feature.

5. The method according to claim 2, characterized in that, The multimodal cooperative sensing model includes Layer embedding transformation and scale feature extraction layer; The step of inputting the semantically aligned first fusion feature into the multimodal collaborative perception model to obtain multimodal results includes: Through the above The layered embedding transformation performs a layered nonlinear transformation on the input features of each modality in the semantically aligned first fusion feature to generate multi-scale embedding representation features; The attention score of the embedding representation feature of each layer is calculated through the scale feature extraction layer, and the attention score is normalized to obtain the normalized attention weight. The embeddings of all modalities and all layers are weighted and fused based on the attention weight to obtain the second fused feature. Perform a dimensionality compression operation on the second fused feature to obtain a multimodal result.

6. The method according to claim 2, characterized in that, The step of performing structured analysis on the multimodal results to generate a predictive perception report vector containing the state evolution path includes: The multimodal results are mapped into structured result vectors using a semantic mapping function; Construct a modal importance matrix; Based on the structured result vector and the modal importance matrix, a predictive perception report vector for the state evolution path is generated.

7. The method according to claim 1, characterized in that, The step of fine-tuning the initial multilayer sensing model using the Prompt-Tuning strategy based on the predicted sensing report vector to obtain the fine-tuned multilayer sensing model includes: Obtain the actual annotation results corresponding to the multimodal sample data; A trainable Prompt vector is introduced into the intermediate layer features of the initial multilayer perception model, and the model parameters are updated using the Prompt vector based on the real annotation results and the predicted perception report vector to obtain a fine-tuned multilayer perception model.

8. An industrial condition monitoring device based on multimodal collaborative sensing, characterized in that, The device includes: The acquisition module is used to acquire multimodal sample data of the target industrial environment; The prediction module is used to input the multimodal sample data into the initial multilayer sensing model to obtain the predicted sensing report vector; The fine-tuning module is used to fine-tune the initial multilayer perception model based on the predicted perception report vector using the Prompt-Tuning strategy to obtain the fine-tuned multilayer perception model. The deployment module is used to split and deploy the fine-tuned multi-layer perception model to the cloud and the edge, and dynamically and collaboratively optimize the fine-tuned multi-layer perception model through the synchronization mechanism between the cloud and the edge to obtain the target multi-layer perception model. The condition monitoring module is used to collect multimodal data in the target industrial environment and obtain industrial condition monitoring results based on the multimodal data through the target multilayer sensing model.

9. An electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images