Equipment state identification method and device, equipment and storage medium

Abstract

According to the equipment state recognition method and device, the equipment and the storage medium provided by the embodiment of the invention, the multi-modal data in the operation process of the target equipment is acquired, and the multi-modal data comprises the image data, the text data and the temperature data; performing feature extraction and embedded coding on the multi-modal data through a feature extraction model to obtain embedded vectors of all modals, and aligning the embedded vectors of all modals to a unified target semantic embedding space; through a modal attention mechanism, the fusion weight of each modal is adjusted according to the state feature of the target device, and the embedding vectors of each modal are fused according to the fusion weight of each modal to construct a multi-modal fusion vector; and recognizing the state of the target equipment and / or performing abnormal risk early warning through a recognition model according to the multi-modal fusion vector. By introducing a modal attention mechanism, dynamic weighted fusion of image, text and temperature modals is realized, the accuracy and generalization ability of state recognition are remarkably improved, and the accuracy and robustness of the model under complex working conditions are improved.

Description

Technical Field

[0001] This application relates to the field of smart grid technology, and in particular to a method, apparatus, device and storage medium for identifying equipment status. Background Technology

[0002] With the continuous improvement of the intelligence and unmanned operation of industrial equipment, equipment operation status assessment based on multi-source sensing data has become a key link in intelligent operation and maintenance systems.

[0003] Traditional equipment condition monitoring methods often rely on single-modal data (such as images or sensor values). When faced with complex operating conditions, missing modal data, or abnormal interference, they are prone to problems such as inaccurate assessment, false alarms, and missed alarms, making it difficult to meet the requirements of high robustness and high accuracy in actual industrial scenarios. Summary of the Invention

[0004] This application provides a device status identification method, apparatus, device, and storage medium to improve [device status identification].

[0005] In a first aspect, embodiments of this application provide a device status identification method, including:

[0006] Acquire multimodal data during the operation of the target device, wherein the multimodal data includes image data, text data, and temperature data;

[0007] The feature extraction model is used to extract features from the multimodal data and embed them into the data to obtain the embedding vectors of each modality. Then, the embedding vectors of each modality are semantically aligned across modalities to align the embedding vectors of each modality to a unified target semantic embedding space.

[0008] The fusion weights of each modality are adjusted according to the state characteristics of the target device through the modal attention mechanism, and the embedding vectors of each modality are fused according to the fusion weights of each modality to construct a multimodal fusion vector.

[0009] Based on the multimodal fusion vector, the status of the target device is identified and / or anomaly risk warnings are issued through the recognition model.

[0010] In one possible implementation, the step of performing feature extraction and embedding encoding on the multimodal data to obtain the embedding vector for each modality includes:

[0011] The image data is divided into image blocks of a preset size using an image modality feature extraction model. Each image block is flattened into a one-dimensional vector and mapped to the semantic embedding space of the target dimension to obtain an image block vector. The position code of each image block is determined and fused with the corresponding position code by addition to obtain a fused vector of image block vector and position code.

[0012] The global semantic features are extracted from the fusion vector of the image block vector and the position encoding using a preset encoder, and then mapped to the target semantic embedding space to obtain the embedding vector of the image modality.

[0013] In one possible implementation, the temperature data is a temperature distribution image; correspondingly, the step of performing feature extraction and embedding encoding on the multimodal data to obtain the embedding vector for each modality includes:

[0014] The temperature feature map is obtained by sequentially processing the temperature distribution image with multiple convolutional blocks using a temperature modality feature extraction model. Each convolutional block includes a convolution operation and a max pooling operation.

[0015] The two-dimensional position code of the temperature distribution image is determined and added element by element to the temperature feature map to obtain a temperature feature vector containing spatial coordinate information;

[0016] The temperature feature vector containing spatial coordinate information is mapped to the target semantic embedding space to obtain the embedding vector of the temperature mode.

[0017] In one possible implementation, the step of performing feature extraction and embedding encoding on the multimodal data to obtain the embedding vector for each modality includes:

[0018] The text data is normalized and segmented using a text modality feature extraction model, and the segmentation results are converted into corresponding word identifiers.

[0019] The context representation of each word is obtained through a preset text processing model, and the vector corresponding to the context representation of each word is used as the semantic feature vector of the text data.

[0020] The semantic feature vectors of the text data are mapped to the target semantic embedding space to obtain the embedding vectors of the text modality.

[0021] In one possible implementation, the method further includes:

[0022] Multiple multimodal training data are acquired, and the feature extraction model corresponding to each modality is used to extract features and embed the data to obtain the embedding vector of each modality in the multimodal training data.

[0023] By maximizing the similarity between different modal embedding vectors belonging to the same multimodal training data and minimizing the similarity between different modal embedding vectors not belonging to the same multimodal training data, the feature extraction model corresponding to each modality is trained.

[0024] In one possible implementation, the step of training the feature extraction model corresponding to each modality by maximizing the similarity between different modality embedding vectors belonging to the same multimodal training data and minimizing the similarity between different modality embedding vectors not belonging to the same multimodal training data includes:

[0025] Construct a loss function for the similarity between different modal embedding vectors, and construct a joint contrastive learning loss function for the feature extraction models corresponding to each modality based on the loss function for the similarity between different modal embedding vectors;

[0026] The feature extraction models for each modality are trained using the joint contrastive learning loss function.

[0027] In one possible implementation, the step of adjusting the fusion weights of each modality based on the state characteristics of the target device using a modal attention mechanism, and fusing the embedding vectors of each modality based on the fusion weights of each modality to construct a multimodal fusion vector, includes:

[0028] A query vector is assigned to the embedding vector of each modality based on the semantic importance of the data in each modality;

[0029] Based on the embedding vector of each modality, and the query vector and key vector corresponding to the embedding vector of each modality, the attention score between the embedding vector of each modality and the corresponding query vector is determined. After normalizing the attention score, the fusion weight of each modality is obtained.

[0030] The embedding vectors of each modality are fused according to the fusion weights of each modality to construct a multimodal fusion vector.

[0031] In one possible implementation, the method further includes:

[0032] Determine the confidence level of each modality's data, and adjust the fusion weights of each modality based on the confidence level of each module; and / or

[0033] The system detects whether there are missing or abnormal data in each modality. If any modality has missing or abnormal data, the fusion weight of that modality is reduced.

[0034] In one possible implementation, the method further includes:

[0035] Continuously collect multimodal data during the operation of the target device, and adaptively optimize the feature extraction model and / or the recognition model using a self-supervised data augmentation strategy.

[0036] Secondly, embodiments of this application provide a device status identification apparatus, comprising:

[0037] The acquisition module is used to acquire multimodal data during the operation of the target device, wherein the multimodal data includes image data, text data and temperature data;

[0038] The feature extraction module is used to extract features and embed them into the multimodal data through a feature extraction model to obtain the embedding vectors of each modality, and to perform cross-modal semantic alignment on the embedding vectors of each modality to align the embedding vectors of each modality to a unified target semantic embedding space.

[0039] The fusion module is used to adjust the fusion weights of each modality according to the state characteristics of the target device through a modal attention mechanism, and to fuse the embedding vectors of each modality according to the fusion weights of each modality to construct a multimodal fusion vector.

[0040] The identification module is used to identify the status of the target device and / or provide anomaly risk warnings based on the multimodal fusion vector and an identification model.

[0041] Thirdly, embodiments of this application provide an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0042] The memory stores computer-executed instructions;

[0043] The processor executes computer execution instructions stored in the memory to implement the method as described in the first aspect or any of the above methods.

[0044] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect above.

[0045] Fifthly, embodiments of this application provide a computer program, the computer program product including a computer program stored in a computer-readable storage medium, at least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, it can implement the method described in the first aspect or the method described in the second aspect.

[0046] The device state recognition method, apparatus, device, and storage medium provided in this application acquire multimodal data during the operation of a target device, including image data, text data, and temperature data. A feature extraction model is used to extract features from the multimodal data and embed them into an embedding vector for each modality. Cross-modal semantic alignment is then performed on the embedding vectors to align them to a unified target semantic embedding space. A modal attention mechanism is used to adjust the fusion weights of each modality based on the state characteristics of the target device. The embedding vectors of each modality are then fused according to these fusion weights to construct a multimodal fusion vector. Based on the multimodal fusion vector, a recognition model identifies the state of the target device and / or provides an anomaly risk warning. By introducing a modal attention mechanism, dynamic weighted fusion of image, text, and temperature modalities is achieved, significantly improving the accuracy and generalization ability of state recognition and enhancing the accuracy and robustness of the model under complex operating conditions. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0048] Figure 1 A schematic diagram of a scenario for a device status identification method provided in an embodiment of this application;

[0049] Figure 2 A flowchart illustrating the device status identification method provided in this application embodiment;

[0050] Figure 3 A flowchart illustrating the device status identification method provided in this application embodiment;

[0051] Figure 4 A flowchart illustrating the device status identification method provided in this application embodiment;

[0052] Figure 5 A flowchart illustrating the device status identification method provided in this application embodiment;

[0053] Figure 6 This is a schematic diagram of the device status identification device provided in the embodiments of this application;

[0054] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0055] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] Before introducing the embodiments of this application, the application background of the embodiments of this application will be explained first:

[0058] With the continuous improvement of the intelligence and unmanned operation of industrial equipment, equipment operation status assessment based on multi-source sensing data has become a key link in intelligent operation and maintenance systems. Traditional equipment status monitoring methods often rely on single-modal data (such as images or sensor values). When faced with complex operating conditions, missing modalities, or abnormal interference, they are prone to problems such as inaccurate assessment, false alarms, and missed alarms, making it difficult to meet the requirements of high robustness and high accuracy in actual industrial scenarios.

[0059] This application takes into account the recent surge in research interest in multimodal learning. By fusing data from different sources and modalities (such as images, text, and temperature), a more comprehensive characterization of equipment operation can be achieved, effectively improving the model's generalization ability and robustness. In practical applications, unmanned aerial vehicle (UAV) inspection systems have been widely deployed in industries such as power, wind power, and petrochemicals. These systems can simultaneously acquire visible light images, infrared thermal images, and structured or unstructured text records (such as maintenance logs and on-site descriptions) of equipment, providing a data foundation for multimodal perception.

[0060] However, current multimodal state assessment methods still suffer from the following technical bottlenecks:

[0061] (1) Multimodal information has semantic differences, making it difficult to achieve high-quality semantic alignment and fusion;

[0062] (2) Different modalities have significant differences in data quality and information density, which may lead to modal redundancy or modal conflict problems;

[0063] (3) Industrial scenarios are often accompanied by scarce samples, insufficient labels or missing modalities, which affect the stability and reliability of the model.

[0064] (4) The distribution of equipment status may change dynamically with time, environment or task scenario, and existing models generally lack online adaptive capabilities.

[0065] To address the aforementioned issues, advanced methods such as cross-modal contrastive learning, modal attention mechanisms, pseudo-label generation, and self-supervised learning have been gradually introduced in recent years to improve the performance of multimodal systems. However, these technologies still lack targeted integrated designs for industrial applications, especially in scenarios with incomplete data and frequent state changes. There is still an urgent need for a multimodal state recognition technology system with unified semantic space modeling, dynamic modal fusion, and self-iterative capabilities.

[0066] To address at least one technical problem in the existing technology, this application provides a device state recognition method. By introducing a modal attention mechanism, it achieves dynamic weighted fusion of image, text, and temperature modalities, significantly improving the accuracy and generalization ability of state recognition, and enhancing the accuracy and robustness of the model under complex working conditions.

[0067] The scenario for the device status identification method in this application is as follows: Figure 1 As shown, multimodal data during the operation of the target device is acquired, including image data, text data, and temperature data. Feature extraction and embedding encoding are performed on the multimodal data using a feature extraction model to obtain embedding vectors for each modality. Cross-modal semantic alignment is then performed on the embedding vectors to align them to a unified target semantic embedding space. A modal attention mechanism is used to adjust the fusion weights of each modality based on the state characteristics of the target device. The embedding vectors of each modality are then fused according to these fusion weights to construct a multimodal fusion vector. Based on the multimodal fusion vector, a recognition model is used to identify the state of the target device and / or provide anomaly risk warnings.

[0068] The technical solution of this application will now be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0069] Figure 2 This is a flowchart illustrating the device status identification method provided in the embodiments of this application, as shown below. Figure 2 As shown, the method may include the following steps:

[0070] S201. Acquire multimodal data during the operation of the target device, wherein the multimodal data includes image data, text data, and temperature data.

[0071] In this embodiment, in industries such as power, wind power, and petrochemicals, image data, text data, and temperature data can be collected for the target device to form multimodal data of the target device.

[0072] Optionally, an automated drone inspection system deployed at the industrial site can be used to acquire environmental images of the target equipment during operation, serving as image data, and to collect infrared thermal imaging temperature data, serving as temperature data. The drone is equipped with a high-resolution visible light camera and an infrared thermal imager, and periodically collects images of the equipment's appearance from different angles and its surface temperature distribution along a preset flight path.

[0073] Secondly, text data for the corresponding time period is synchronously extracted from the equipment maintenance management system, including but not limited to maintenance records, fault logs, and manual inspection notes. This type of text data can be retrieved in a structured manner through the system API interface, or the inspection voice information can be converted into text by the speech-to-text module.

[0074] Of course, this embodiment is not limited to the above-mentioned methods for obtaining multimodal data during the operation of the target device; other methods may also be used.

[0075] Optionally, the raw data of the above multimodal data may be formatted and preprocessed. The preprocessing operations include, but are not limited to, the following:

[0076] The acquired image data is subjected to size unification, brightness normalization, and image enhancement processing.

[0077] Spatial interpolation and temperature range standardization are performed on the temperature data to extract temperature channels;

[0078] Text data is segmented, denoised, and encoded to form a context-dependent text embedding representation (such as BERT encoding).

[0079] The above methods can automatically complete the collaborative acquisition and preliminary processing of images, temperature, and text without interfering with the normal operation of the equipment, laying a data foundation for subsequent modal fusion and state assessment.

[0080] S202. The feature extraction model is used to extract features from the multimodal data and embedding and encoding them to obtain the embedding vectors of each modality. The embedding vectors of each modality are then semantically aligned across modalities to align the embedding vectors of each modality to a unified target semantic embedding space.

[0081] In this embodiment, feature extraction and embedding encoding can be performed on the multimodal data to obtain embedding vectors for each modality. Each modality can be configured with a feature extraction model, which can perform feature extraction and embedding encoding on the multimodal data separately. The specific feature extraction and embedding encoding methods are not limited in this embodiment and can be any feasible method. After obtaining the embedding vectors for each modality, cross-modal semantic alignment can be performed on the embedding vectors of each modality to align them to a unified target semantic embedding space. The unified target semantic embedding space has the same dimension, which can be used for tasks such as alignment, fusion, and classification of the embedding vectors of each modality.

[0082] S203. Using a modal attention mechanism, adjust the fusion weights of each modality according to the state characteristics of the target device, and fuse the embedding vectors of each modality according to the fusion weights of each modality to construct a multimodal fusion vector.

[0083] In this embodiment, a modality-wise attention mechanism is introduced. By combining the current device state features, the fusion weights of each modality in the feature fusion process can be dynamically adjusted. Then, based on the fusion weights of each modality, vector fusion is performed to generate a multimodal fusion vector with consistent structure and unified semantics, which alleviates the redundancy and conflict between modalities and improves the adaptability and robustness of the device state assessment model under complex working conditions.

[0084] Optionally, by using a modal attention mechanism, adjusting the fusion weights of each modality based on the state characteristics of the target device, and fusing the embedding vectors of each modality according to the fusion weights to construct a multimodal fusion vector, the following steps are taken:

[0085] A query vector is assigned to the embedding vector of each modality based on the semantic importance of the data in each modality;

[0086] Based on the embedding vector of each modality, and the query vector and key vector corresponding to the embedding vector of each modality, the attention score between the embedding vector of each modality and the corresponding query vector is determined. After normalizing the attention score, the fusion weight of each modality is obtained.

[0087] The embedding vectors of each modality are fused according to the fusion weights of each modality to construct a multimodal fusion vector.

[0088] In this embodiment, based on the scaled dot product attention mechanism, a query vector (Modality Querying) can first be assigned to the embedding vector of each modality based on the semantic importance of each modality's data. The query vector Q m Q img Q temp Q txtIt is used to model the semantic importance of each modality in specific samples and tasks.

[0089] For each modality's embedding vector, calculate its attention weight with the corresponding query vector. A scaled dot product attention mechanism is used, as shown in the following formula:

[0090]

[0091] in Let d be the key vector of the modal features, α be the vector dimension, and d be the vector dimension. m This represents the attention score for modality m. The query vector represents the question vector for the current modality, used to calculate the degree of attention to other modalities, and the key vector represents the index vectors of other modalities, used to interact with the query vector to generate attention weights.

[0092] Attention score α for all modalities m Perform Softmax normalization to obtain the weight w for each mode. m Then, a weighted summation method is used to merge all modalities:

[0093]

[0094] Where F m F is the characteristic representation of mode m. fused This is the final fused device state semantic representation vector, also known as the multimodal fusion vector.

[0095] Optionally, to prevent the model from over-relying on a particular modality during training or inference, the following two enhancement strategies are proposed:

[0096] Determine the confidence level of each modality's data, and adjust the fusion weights of each modality based on the confidence level of each module; and / or

[0097] The system detects whether there are missing or abnormal data in each modality. If any modality has missing or abnormal data, the fusion weight of that modality is reduced.

[0098] In this embodiment, a modal confidence estimator can be used to model the noise level of each modality based on indicators such as image blurriness, heatmap abrupt changes, and text integrity. The confidence results are used to dynamically adjust the fusion weights of each modality to suppress interference from poor-quality modalities.

[0099] In addition, a modal gating device can be designed to automatically reduce the fusion participation of a modality and reduce the fusion weight of that modality when a missing or abnormal modality is detected (such as missing frames in drone images or sensor disconnection), thus ensuring the overall robustness and stability of the model.

[0100] S204. Based on the multimodal fusion vector, identify the status of the target device and / or issue an abnormal risk warning through the identification model.

[0101] In this embodiment, the multimodal fusion vector obtained from the above process is input into the recognition model. The recognition model can employ a multilayer perceptron (MLP), Transformer, or graph neural network (GNN) to adapt to complex state pattern recognition requirements. Of course, any other feasible model can also be used.

[0102] The identification model can be pre-trained under supervision using labeled historical equipment status data (including normal, abnormal, and fault levels) to optimize classifier weights and enable the model to accurately determine the current equipment status category. Furthermore, class imbalance handling strategies (such as weighted cross-entropy and focus loss) can be introduced to improve the identification capability of a few abnormal states.

[0103] Furthermore, the device status can be judged based on the status recognition results. Threshold rules or learning early warning strategies can be set to assess the risk level of identified abnormal states, outputting early warning signals and risk level prompts. Optionally, early warning information can be combined with contextual data such as the device's geographical location and operating environment to form a comprehensive risk assessment report, assisting maintenance personnel in decision-making. Optionally, user-defined early warning thresholds and alarm strategies can be supported, flexibly adapting to different equipment types and operating conditions.

[0104] Based on any of the above embodiments, the recognition model can be integrated into the equipment monitoring platform, which can receive multimodal data and model inference results in real time to achieve online status monitoring.

[0105] Based on any of the above embodiments, the model performance is continuously corrected and optimized through a feedback mechanism of historical early warning data and subsequent maintenance records, thereby improving the accuracy of early warning and the speed of response.

[0106] The device status recognition method provided in this application acquires multimodal data during the operation of a target device, including image data, text data, and temperature data. It extracts features and embeds them into the multimodal data using a feature extraction model to obtain embedding vectors for each modality. The embedding vectors are then semantically aligned across modalities to a unified target semantic embedding space. A modal attention mechanism is used to adjust the fusion weights of each modality based on the device's status characteristics. The embedding vectors of each modality are then fused according to these weights to construct a multimodal fusion vector. Based on this fusion vector, a recognition model identifies the status of the target device and / or provides an anomaly risk warning. By introducing a modal attention mechanism, dynamic weighted fusion of image, text, and temperature modalities is achieved, significantly improving the accuracy and generalization ability of status recognition and enhancing the model's accuracy and robustness under complex operating conditions.

[0107] Based on any of the above embodiments, for image data, feature extraction and embedding encoding are performed on the multimodal data to obtain embedding vectors for each modality, such as... Figure 3 Specifically, it may include:

[0108] S301. Using the feature extraction model of image modality, the image data is divided into image blocks of a preset size. Each image block is flattened into a one-dimensional vector. The one-dimensional vector is mapped to the semantic embedding space of the target dimension to obtain the image block vector. The position code of each image block is determined. Each image block vector is fused with the corresponding position code by addition to obtain the fused vector of image block vector and position code.

[0109] S302. Use a preset encoder to extract global semantic features from the fusion vector of image block vector and position encoding, and map the global semantic features to the target semantic embedding space to obtain the embedding vector of image modality.

[0110] In this embodiment, the image data can be standardized first, and then image enhancement can be performed; of course, standardization and image enhancement can also be omitted.

[0111] In this embodiment, image data can be divided into fixed-size image blocks, each block is flattened into a one-dimensional vector, and then mapped to a target-dimensional semantic embedding space through a linear transformation layer. The target-dimensional semantic embedding space can be a cross-modal unified spatial dimension. In addition, a learnable positional code is added to each image block, and the image block vector and the positional code are fused in an additive manner, enabling the model to distinguish the location of the block.

[0112] Furthermore, a pre-defined encoder is used to extract global semantics. This pre-defined encoder can be a Transformer encoder, or optionally, a MobileViT encoder can be used instead of a standard ViT encoder to reduce the number of parameters and computational complexity. The MobileViT encoder or ViT encoder consists of multiple stacked Transformer layers, each containing a multi-head self-attention module, residual connections + LayerNorm, and a feedforward network module.

[0113] A special token (classification label) can be input to a pre-defined encoder; after model training, the output vector of this token can serve as a global semantic representation of the image. This token is a specially designed input in the Transformer architecture, used to represent the image throughout the semantic extraction process. During training, the token is added to the image's input sequence, and as the model trains, it gradually learns how to aggregate global information about the image. The semantic representation of the entire image can be obtained through the output vector of this token.

[0114] Finally, the global semantic features output by the preset encoder are mapped to a cross-modal unified target semantic embedding space to obtain the image modality embedding vector. This can be mapped to a cross-modal unified spatial dimension through a single MLP (Multilayer Perceptron); thus obtaining the final semantic feature representation f of the image modality. img This refers to the embedding vector of the image modality, which is used for subsequent tasks such as modality alignment, fusion, and classification.

[0115] Based on any of the above embodiments, the temperature data is a temperature distribution image; correspondingly, for the temperature data, feature extraction and embedding encoding are performed on the multimodal data to obtain the embedding vectors of each modality, such as... Figure 4 Specifically, it may include:

[0116] S401. Using the feature extraction model of temperature mode, the temperature distribution image is processed sequentially using multiple convolutional blocks to obtain a temperature feature map, wherein each convolutional block includes a convolution operation and a max pooling operation.

[0117] S402. Determine the two-dimensional position code of the temperature distribution image and add it element by element to the temperature feature map to obtain a temperature feature vector containing spatial coordinate information;

[0118] S403. Map the temperature feature vector containing spatial coordinate information to the target semantic embedding space to obtain the embedding vector of the temperature mode.

[0119] In this embodiment, for temperature distribution images such as infrared thermal imaging images, temperature distribution features are extracted using a dedicated CNN model for temperature distribution images, while retaining spatial location sensitivity, to generate embedding vectors for temperature modes.

[0120] Specifically, a dedicated CNN network for temperature distribution images can be pre-built: a dedicated temperature feature extraction neural network is built for infrared thermal imaging images, using a lightweight convolutional neural network (CNN) structure, consisting of 4 convolutional blocks, each containing a convolution operation and a max pooling operation.

[0121] First, the input single-channel temperature distribution image is fed into the first convolutional layer, which uses a 3×3 kernel, outputs 16 channels, and employs the ReLU activation function. Then, spatial compression is performed using a 2×2 max pooling operation to extract low-level temperature edge features.

[0122] Next, the second and third convolutional blocks continue to use 3×3 convolutional kernels and max pooling layers, respectively, increasing the output channels layer by layer to 32 and 64, in order to capture temperature distribution patterns over a larger area.

[0123] The fourth convolutional block continues to extract deep features in the same way, outputting a spatially sensitive temperature embedding feature map that preserves the spatial structure information of the device's temperature distribution.

[0124] It should be noted that the size of the convolution kernel in each convolution block is not limited to the examples above; other sizes are also possible.

[0125] After extracting the temperature feature map, a two-dimensional positional encoding is introduced, which is added element-wise to the temperature feature map to encode spatial coordinate information, resulting in a temperature feature vector containing spatial coordinate information. Further, this temperature feature vector containing spatial coordinate information is mapped to a cross-modal unified spatial dimension through a single MLP layer, i.e., mapped to a cross-modal unified target semantic embedding space; thus, the final semantic feature representation f of the temperature modality is obtained. temp That is, the embedding vector of the temperature mode.

[0126] Based on any of the above embodiments, for text data, feature extraction and embedding encoding are performed on the multimodal data to obtain embedding vectors for each modality, such as... Figure 4 The following are included:

[0127] S501. The text data is normalized and segmented using a text modality feature extraction model, and the segmentation results are converted into corresponding word identifiers.

[0128] S502. Obtain the context representation of each word through a preset text processing model, and use the vector corresponding to the context representation of each word as the semantic feature vector of the text data.

[0129] S503. Map the semantic feature vector of the text data to the target semantic embedding space to obtain the embedding vector of the text modality.

[0130] In this embodiment, text preprocessing may include two steps: text cleaning and normalization, and word segmentation.

[0131] Text standardization includes, but is not limited to: removing invalid symbols and using a unified encoding format (UTF-8); removing redundant content such as timestamps and numbers (if not critical information); and establishing custom thesaurus or terminology mappings for technical terms.

[0132] Sentence segmentation and word segmentation can specifically involve segmenting each piece of text data, such as maintenance or inspection text, into sentences; and using a tokenizer supported by pre-defined text processing models such as RoBERTa (that is, converting text into token IDs).

[0133] Furthermore, a pre-defined text processing model such as RoBERTa is selected as the backbone model; the token ID is input into the RoBERTa model to obtain the context representation of each token; the output vector corresponding to the context representation of the token is taken as the semantic representation f of the entire text. text That is, the semantic feature vector of text data.

[0134] By normalizing the semantic feature vectors of the text data using LayerNorm, and adding a fully connected layer Linear(d,D), the semantic feature vector dimension of the text data is projected to a spatial dimension D that is consistent with the embedding vectors of images and temperatures, that is, mapped to a cross-modal unified target semantic embedding space.

[0135] Based on any of the above embodiments, the method further includes constructing positive and negative sample pairs using cross-modal paired samples, and training the model using the InfoNCE (Information Noise Contrastive Estimation) loss function, specifically including:

[0136] Multiple multimodal training data are acquired, and the feature extraction model corresponding to each modality is used to extract features and embed the data to obtain the embedding vector of each modality in the multimodal training data.

[0137] By maximizing the similarity between different modal embedding vectors belonging to the same multimodal training data and minimizing the similarity between different modal embedding vectors not belonging to the same multimodal training data, the feature extraction model corresponding to each modality is trained.

[0138] In this embodiment, a cross-modal contrastive learning framework is constructed based on the embedding vectors of the three modalities extracted in the above process.

[0139] Image-text comparison: Positive sample pairs are formed by pairing image modalities with corresponding text modalities, and negative sample pairs are formed by pairing images with other unpaired (asynchronously acquired) text.

[0140] Using InfoNCE loss, the loss function formula is as follows:

[0141]

[0142] Where sim(·,·) represents the similarity between embedding vectors, τ is the temperature coefficient, and N is the number of sample pairs. The loss function representing image-text comparison.

[0143] Image-Temperature Contrast: Image modalities and synchronously acquired temperature modalities (infrared thermogram features) are used as positive sample pairs, while images are paired with other unpaired (asynchronously acquired) temperature modalities to form negative sample pairs, reinforcing the correspondence between images and temperature distribution features. A contrastive learning framework is also used for training.

[0144] Text-Temperature Contrast: Text modalities and synchronously acquired temperature modalities (infrared thermogram features) are used as positive sample pairs, while text is paired with other unpaired (asynchronously acquired) temperature modalities to form negative sample pairs, reinforcing the correspondence between text and temperature distribution features. A contrastive learning framework is also used for training.

[0145] Based on the above embodiments, by training the feature extraction models corresponding to each modality, the method may further include:

[0146] Construct a loss function for the similarity between different modal embedding vectors, and construct a joint contrastive learning loss function for the feature extraction models corresponding to each modality based on the loss function for the similarity between different modal embedding vectors;

[0147] The feature extraction models for each modality are trained using the joint contrastive learning loss function.

[0148] In this embodiment, during training, the multimodal pairing relationships between image-text, image-temperature, and text-temperature are comprehensively considered. The feature extraction model corresponding to each modality is optimized through a joint contrastive learning loss function, as follows:

[0149]

[0150] in The loss function representing image-text comparison, The loss function representing the image-temperature contrast is... The loss function represents the text-temperature comparison, where λ1, λ2, and λ3 are weighting coefficients.

[0151] Through the above loss function optimization, the feature extraction model corresponding to each modality can automatically learn the semantic mapping relationship between each modality, align the features of different modalities to a unified semantic embedding space, thereby improving the accuracy and robustness of subsequent multimodal fusion and state evaluation.

[0152] Based on any of the above embodiments, the method further includes:

[0153] Continuously collect multimodal data during the operation of the target device, and adaptively optimize the feature extraction model and / or the recognition model using a self-supervised data augmentation strategy.

[0154] In this embodiment, the parameters of the device status recognition model can be updated in real time using the multimodal data continuously collected during device operation, thereby achieving dynamic adaptation of the model.

[0155] Optionally, incremental learning algorithms can be designed to prevent the model from forgetting historical knowledge and to quickly adapt to changes in the distribution of new samples.

[0156] Optionally, a data cleaning and anomaly detection module can be integrated to address potential noise and anomalous samples in the online data, ensuring the stability and effectiveness of online learning.

[0157] By incorporating online learning mechanisms, the model gains continuous optimization and adaptive capabilities, thereby enhancing the intelligence level and practical application value of the equipment condition assessment system.

[0158] Optionally, a multimodal data pseudo-label generation module can be constructed to automatically generate supplementary information for missing modalities, expanding the diversity and richness of training samples. Through cross-modal pseudo-label generation and self-supervised enhancement strategies, the problems of insufficient labeled samples and missing modalities in industrial scenarios are effectively alleviated, and the robustness of the model under varying data distributions and complex environments is enhanced.

[0159] Optionally, self-supervised tasks based on intermodal mutual prediction can be designed, such as predicting text descriptions from image content and inferring temperature trends from text, to promote the model to learn more robust and generalized representations.

[0160] Optionally, contrastive learning methods can be used to enhance multimodal consistency and improve the learning performance of the model in unlabeled or poorly labeled environments.

[0161] Based on any of the above embodiments, an online learning and data augmentation module is embedded into the device monitoring system to achieve automatic model updates and maintenance. It supports model performance monitoring and automatic alerts, promptly detecting model degradation and triggering retraining processes. In conjunction with business needs, it regularly exports model update reports to assist operations and maintenance personnel in developing optimization strategies.

[0162] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0163] Figure 6 This is a schematic diagram of the device status identification device provided in an embodiment of this application. Figure 6 As shown, the device status identification device 600 includes:

[0164] The acquisition module 601 is used to acquire multimodal data during the operation of the target device, wherein the multimodal data includes image data, text data and temperature data;

[0165] The feature extraction module 602 is used to extract features and embed code the multimodal data through the feature extraction model to obtain the embedding vector of each modality, and to perform cross-modal semantic alignment on the embedding vector of each modality to align the embedding vector of each modality to a unified target semantic embedding space.

[0166] The fusion module 603 is used to adjust the fusion weights of each modality according to the state characteristics of the target device through a modal attention mechanism, and to fuse the embedding vectors of each modality according to the fusion weights of each modality to construct a multimodal fusion vector.

[0167] The identification module 604 is used to identify the status of the target device and / or provide anomaly risk warnings based on the multimodal fusion vector and the identification model.

[0168] In an optional embodiment, when the feature extraction module 602 performs feature extraction and embedding encoding on the multimodal data to obtain the embedding vectors of each modality, it is used to:

[0169] The image data is divided into image blocks of a preset size using an image modality feature extraction model. Each image block is flattened into a one-dimensional vector and mapped to the semantic embedding space of the target dimension to obtain an image block vector. The position code of each image block is determined and fused with the corresponding position code by addition to obtain a fused vector of image block vector and position code.

[0170] The global semantic features are extracted from the fusion vector of the image block vector and the position encoding using a preset encoder, and then mapped to the target semantic embedding space to obtain the embedding vector of the image modality.

[0171] In an optional embodiment, the temperature data is a temperature distribution image; correspondingly, when the feature extraction module 602 performs feature extraction and embedding encoding on the multimodal data to obtain the embedding vectors of each modality, it is used to:

[0172] The temperature feature map is obtained by sequentially processing the temperature distribution image with multiple convolutional blocks using a temperature modality feature extraction model. Each convolutional block includes a convolution operation and a max pooling operation.

[0173] The two-dimensional position code of the temperature distribution image is determined and added element by element to the temperature feature map to obtain a temperature feature vector containing spatial coordinate information;

[0174] The temperature feature vector containing spatial coordinate information is mapped to the target semantic embedding space to obtain the embedding vector of the temperature mode.

[0175] In an optional embodiment, when the feature extraction module 602 performs feature extraction and embedding encoding on the multimodal data to obtain the embedding vectors of each modality, it is used to:

[0176] The text data is normalized and segmented using a text modality feature extraction model, and the segmentation results are converted into corresponding word identifiers.

[0177] The context representation of each word is obtained through a preset text processing model, and the vector corresponding to the context representation of each word is used as the semantic feature vector of the text data.

[0178] The semantic feature vectors of the text data are mapped to the target semantic embedding space to obtain the embedding vectors of the text modality.

[0179] In an optional embodiment, the apparatus further includes a training module for:

[0180] Multiple multimodal training data are acquired, and the feature extraction model corresponding to each modality is used to extract features and embed the data to obtain the embedding vector of each modality in the multimodal training data.

[0181] By maximizing the similarity between different modal embedding vectors belonging to the same multimodal training data and minimizing the similarity between different modal embedding vectors not belonging to the same multimodal training data, the feature extraction model corresponding to each modality is trained.

[0182] In an optional embodiment, the training module trains the feature extraction model corresponding to each modality, including:

[0183] Construct a loss function for the similarity between different modal embedding vectors, and construct a joint contrastive learning loss function for the feature extraction models corresponding to each modality based on the loss function for the similarity between different modal embedding vectors;

[0184] The feature extraction models for each modality are trained using the joint contrastive learning loss function.

[0185] In an optional embodiment, when the fusion module 603 adjusts the fusion weights of each modality according to the state characteristics of the target device through the modal attention mechanism, and fuses the embedding vectors of each modality according to the fusion weights of each modality to construct a multimodal fusion vector, it is used to:

[0186] A query vector is assigned to the embedding vector of each modality based on the semantic importance of the data in each modality;

[0187] Based on the embedding vector of each modality, and the query vector and key vector corresponding to the embedding vector of each modality, the attention score between the embedding vector of each modality and the corresponding query vector is determined. After normalizing the attention score, the fusion weight of each modality is obtained.

[0188] The embedding vectors of each modality are fused according to the fusion weights of each modality to construct a multimodal fusion vector.

[0189] In an optional embodiment, the fusion module 603 is further configured to:

[0190] Determine the confidence level of each modality's data, and adjust the fusion weights of each modality based on the confidence level of each module; and / or

[0191] The system detects whether there are missing or abnormal data in each modality. If any modality has missing or abnormal data, the fusion weight of that modality is reduced.

[0192] In an optional embodiment, the training module is further configured to:

[0193] Continuously collect multimodal data during the operation of the target device, and adaptively optimize the feature extraction model and / or the recognition model using a self-supervised data augmentation strategy.

[0194] The apparatus provided in this application embodiment can be used to execute the determination method in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0195] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. Additionally, these modules can be fully or partially integrated together, or implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed through the integrated logic circuits in the hardware of the processor element or through software instructions.

[0196] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 7 As shown, the electronic device may include: a processor 701, a memory 702, and computer program instructions stored in the memory 702 and executable on the processor 701. When the processor 701 executes the computer program instructions, it implements the method provided in any of the foregoing embodiments.

[0197] Optionally, the various components of the electronic device can be connected via a system bus.

[0198] The memory 702 can be a separate memory unit or a memory unit integrated into the processor 701. The number of processors 701 can be one or more.

[0199] It should be understood that the processor 701 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0200] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Memory 702 may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0201] All or part of the steps in the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above-described method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0202] The electronic device provided in this application embodiment can be used to execute the method provided in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0203] This application provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the above-described method.

[0204] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory, electrically erasable programmable read-only memory, erasable programmable read-only memory, programmable read-only memory, read-only memory, magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0205] Optionally, a readable storage medium can be coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Alternatively, the readable storage medium can be an integral part of the processor. Both the processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components within the device.

[0206] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and the at least one processor can implement the above-described method when executing the computer program.

[0207] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for identifying device status, characterized in that, The method includes: Acquire multimodal data during the operation of the target device, wherein the multimodal data includes image data, text data, and temperature data; The feature extraction model is used to extract features from the multimodal data and embed them into the data to obtain the embedding vectors of each modality. Then, the embedding vectors of each modality are semantically aligned across modalities to align the embedding vectors of each modality to a unified target semantic embedding space. The fusion weights of each modality are adjusted according to the state characteristics of the target device through the modal attention mechanism, and the embedding vectors of each modality are fused according to the fusion weights of each modality to construct a multimodal fusion vector. Based on the multimodal fusion vector, the status of the target device is identified and / or anomaly risk warnings are issued through the recognition model.

2. The method according to claim 1, characterized in that, The step of extracting features and embedding them into the multimodal data to obtain the embedding vectors for each modality includes: The image data is divided into image blocks of a preset size using an image modality feature extraction model. Each image block is flattened into a one-dimensional vector and mapped to the semantic embedding space of the target dimension to obtain an image block vector. The position code of each image block is determined and fused with the corresponding position code by addition to obtain a fused vector of image block vector and position code. The global semantic features are extracted from the fusion vector of the image block vector and the position encoding using a preset encoder, and then mapped to the target semantic embedding space to obtain the embedding vector of the image modality.

3. The method according to claim 1, characterized in that, The temperature data is a temperature distribution image; correspondingly, the step of performing feature extraction and embedding encoding on the multimodal data to obtain the embedding vector for each modality includes: The temperature feature map is obtained by sequentially processing the temperature distribution image with multiple convolutional blocks using a temperature modality feature extraction model. Each convolutional block includes a convolution operation and a max pooling operation. The two-dimensional position code of the temperature distribution image is determined and added element by element to the temperature feature map to obtain a temperature feature vector containing spatial coordinate information; The temperature feature vector containing spatial coordinate information is mapped to the target semantic embedding space to obtain the embedding vector of the temperature mode.

4. The method according to claim 1, characterized in that, The step of extracting features and embedding them into the multimodal data to obtain the embedding vectors for each modality includes: The text data is normalized and segmented using a text modality feature extraction model, and the segmentation results are converted into corresponding word identifiers. The context representation of each word is obtained through a preset text processing model, and the vector corresponding to the context representation of each word is used as the semantic feature vector of the text data. The semantic feature vectors of the text data are mapped to the target semantic embedding space to obtain the embedding vectors of the text modality.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Multiple multimodal training data are acquired, and the feature extraction model corresponding to each modality is used to extract features and embed the data to obtain the embedding vector of each modality in the multimodal training data. By maximizing the similarity between different modal embedding vectors belonging to the same multimodal training data and minimizing the similarity between different modal embedding vectors not belonging to the same multimodal training data, the feature extraction model corresponding to each modality is trained.

6. The method according to claim 5, characterized in that, The training of the feature extraction model corresponding to each modality includes: Construct a loss function for the similarity between different modal embedding vectors, and construct a joint contrastive learning loss function for the feature extraction models corresponding to each modality based on the loss function for the similarity between different modal embedding vectors; The feature extraction models for each modality are trained using the joint contrastive learning loss function.

7. The method according to any one of claims 1-4, characterized in that, The process of adjusting the fusion weights of each modality based on the state characteristics of the target device through a modal attention mechanism, and fusing the embedding vectors of each modality according to the fusion weights to construct a multimodal fusion vector, includes: A query vector is assigned to the embedding vector of each modality based on the semantic importance of the data in each modality; Based on the embedding vector of each modality, and the query vector and key vector corresponding to the embedding vector of each modality, the attention score between the embedding vector of each modality and the corresponding query vector is determined. After normalizing the attention score, the fusion weight of each modality is obtained. The embedding vectors of each modality are fused according to the fusion weights of each modality to construct a multimodal fusion vector.

8. A device for identifying equipment status, characterized in that, The device includes: The acquisition module is used to acquire multimodal data during the operation of the target device, wherein the multimodal data includes image data, text data and temperature data; The feature extraction module is used to extract features and embed them into the multimodal data through a feature extraction model to obtain the embedding vectors of each modality, and to perform cross-modal semantic alignment on the embedding vectors of each modality to align the embedding vectors of each modality to a unified target semantic embedding space. The fusion module is used to adjust the fusion weights of each modality according to the state characteristics of the target device through a modal attention mechanism, and to fuse the embedding vectors of each modality according to the fusion weights of each modality to construct a multimodal fusion vector. The identification module is used to identify the status of the target device and / or provide anomaly risk warnings based on the multimodal fusion vector and an identification model.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images