Power equipment intelligent operation and maintenance method based on multi-modal collaborative attention mechanism and related device
By employing a multimodal collaborative attention mechanism, the problem of poor multimodal data fusion in power equipment operation and maintenance is solved, enabling a comprehensive and accurate understanding of equipment status and improving operation and maintenance efficiency and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUANENG JINGMEN THERMAL POWER CO LTD
- Filing Date
- 2026-01-05
- Publication Date
- 2026-04-24
AI Technical Summary
In traditional power equipment operation and maintenance, the multimodal data fusion effect is poor, the depth of scenario understanding is insufficient, and there is a lack of multimodal collaborative attention design, which makes it impossible to achieve a comprehensive understanding of equipment status.
A multimodal collaborative attention mechanism is adopted. By collecting and preprocessing multimodal data, visual, semantic and temporal features are extracted, and a three-layer collaborative attention mechanism is constructed to perform feature fusion and scene understanding to generate a comprehensive report.
It enables a three-dimensional and comprehensive understanding of power equipment operation and maintenance scenarios, precise feature association, and adaptive weight adjustment, making it applicable to different types of equipment and scenarios, and improving operation and maintenance efficiency and accuracy.
Smart Images
Figure CN121920993A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of industrial intelligent operation and maintenance and multimodal artificial intelligence technology, specifically involving a method and related device for intelligent operation and maintenance of power equipment based on a multimodal collaborative attention mechanism. Background Technology
[0002] In the operation and maintenance of equipment in the power industry, traditional operation and maintenance methods face multiple challenges: (1) The problem of information silos: Equipment status information is scattered across different modalities of data, including visual information, text information, and real-time data. Visual information consists of photos of equipment taken by manual inspections, including appearance status, indicator lights, and instrument readings. Text information includes unstructured descriptions such as maintenance logs, repair reports, and equipment ledgers. Real-time data includes time-series data such as vibration, temperature, and pressure collected by the DCS system. Currently, these data are usually processed independently by different systems, lacking effective means of fusion and analysis.
[0003] (2) Limitations of traditional methods: When using single-modal data analysis, images only focus on visual features, text analysis only processes language descriptions, and data monitoring only focuses on numerical trends, failing to form a complete understanding of equipment status. Some methods rely on simple feature concatenation based on multimodal data: Some existing multimodal methods simply concatenate feature vectors from different modalities, ignoring the complex relationships and semantic complementarity between modalities. They also neglect the role of cross-modal attention mechanisms. Existing attention mechanisms are mostly used within a single modality, such as self-attention in Transformers, but the model lacks a multimodal collaborative attention design specifically for power operation and maintenance scenarios.
[0004] (3) Actual operation and maintenance needs require a comprehensive assessment of the health status of equipment from multi-source heterogeneous data; it is necessary to understand the behavior patterns of equipment in specific operation and maintenance scenarios and to provide comprehensive scenario understanding support for different operation and maintenance tasks (such as fault diagnosis, status assessment, and maintenance decision-making). Summary of the Invention
[0005] The purpose of this invention is to provide a method and related device for intelligent operation and maintenance of power equipment based on a multimodal collaborative attention mechanism, so as to solve the problems of poor multimodal data fusion effect and insufficient depth of scenario understanding in the prior art, and to achieve a comprehensive, accurate and intelligent understanding of the operation and maintenance scenario of power equipment.
[0006] To achieve the above objectives, the present invention adopts the following technical solution: The intelligent operation and maintenance method for power equipment based on a multimodal collaborative attention mechanism includes the following steps: S1: Collect multimodal data and perform preprocessing; S2: Extract multimodal features from the preprocessed multimodal data. Multimodal features include visual features, semantic features, and temporal features. S3: Based on the extracted multimodal features, perform multimodal collaborative attention calculation and construct a three-layer collaborative attention mechanism; S4: Based on a three-layer collaborative attention mechanism, multimodal feature fusion and scene understanding are performed. The multimodal features enhanced by collaborative attention are deeply fused and input into the scene understanding module, and output equipment status assessment, fault warning and maintenance suggestion information. S5: Output and application of scenario understanding results, generating comprehensive scenario understanding reports to provide visual support for operation and maintenance decisions.
[0007] A further improvement of this invention is that, S1: acquiring multimodal data and performing preprocessing, including: Collect equipment inspection images, maintenance text data, and real-time equipment operation data, and standardize and normalize all types of data.
[0008] A further improvement of this invention is that, S2: multimodal feature extraction is performed on the preprocessed multimodal data, including: Visual features of inspection images are extracted using a visual encoder. Use a text encoder to extract semantic features from the operation and maintenance text. Use a time encoder to extract the temporal features of real-time data. .
[0009] A further improvement of this invention is that, S3: based on the extracted multimodal features, multimodal collaborative attention calculation is performed to construct a three-layer collaborative attention mechanism, including: S3.1: Intramodal self-attention layer, calculating self-attention within each modality to enhance the weights of important features. (Formula omitted) ,in , , These are the Query, Key, and Value matrices, respectively, and the Softmax function is a mapping function. S3.2: Cross-modal collaborative attention layer, designing bidirectional cross-modal attention interaction; from visual to textual attention, allowing semantic features to focus on relevant visual regions, the formula is as follows. From text to visual attention, enabling visual features to focus on relevant textual descriptions, the formula is as follows: From real-time data to multimodal attention, it establishes a correlation between real-time data features and image and semantic features, as shown in the formula. ; S3.3: Modal weight adaptive layer, which dynamically adjusts the contribution weight of each modality according to the current operation and maintenance task and scenario complexity. The weight factor is calculated based on the confidence level of each modality feature and the task relevance.
[0010] A smart operation and maintenance system for power equipment based on a multimodal collaborative attention mechanism includes: Data acquisition and preprocessing unit: Acquires multimodal data and performs preprocessing; Multimodal feature extraction unit: Extracts multimodal features from preprocessed multimodal data. Multimodal features include visual features, semantic features, and temporal features. Three-layer collaborative attention mechanism construction unit: Based on the extracted multimodal features, multimodal collaborative attention calculation is performed to construct a three-layer collaborative attention mechanism; Multimodal feature fusion and scene understanding unit: Based on a three-layer collaborative attention mechanism, it performs multimodal feature fusion and scene understanding, deeply fuses the multimodal features enhanced by collaborative attention, inputs them into the scene understanding module, and outputs equipment status assessment, fault warning and maintenance suggestion information; Scenario Understanding Report Generation Unit: Outputs and applies scenario understanding results, generates comprehensive scenario understanding reports, and provides visual support for operation and maintenance decisions.
[0011] A further improvement of this invention is that, in the data acquisition and preprocessing unit: acquiring multimodal data and performing preprocessing includes: Collect equipment inspection images, maintenance text data, and real-time equipment operation data, and standardize and normalize all types of data.
[0012] A further improvement of this invention is that, in the multimodal feature extraction unit: multimodal feature extraction is performed on the preprocessed multimodal data, including: Visual features of inspection images are extracted using a visual encoder. Use a text encoder to extract semantic features from the operation and maintenance text. Use a time encoder to extract the temporal features of real-time data. .
[0013] A further improvement of this invention lies in the fact that, in the three-layer collaborative attention mechanism construction unit: based on the extracted multimodal features, multimodal collaborative attention calculation is performed to construct a three-layer collaborative attention mechanism, including: S3.1: Intramodal self-attention layer, calculating self-attention within each modality to enhance the weights of important features. (Formula omitted) ,in , , These are the Query, Key, and Value matrices, respectively, and the Softmax function is a mapping function. S3.2: Cross-modal collaborative attention layer, designing bidirectional cross-modal attention interaction; from visual to textual attention, allowing semantic features to focus on relevant visual regions, the formula is as follows. From text to visual attention, enabling visual features to focus on relevant textual descriptions, the formula is as follows: From real-time data to multimodal attention, it establishes a correlation between real-time data features and image and semantic features, as shown in the formula. ; S3.3: Modal weight adaptive layer, which dynamically adjusts the contribution weight of each modality according to the current operation and maintenance task and scenario complexity. The weight factor is calculated based on the confidence level of each modality feature and the task relevance.
[0014] An electronic device includes: a processor and a memory coupled to the processor, the memory storing a computer program that, when executed by the processor, implements the steps of the intelligent operation and maintenance method for power equipment based on a multimodal collaborative attention mechanism.
[0015] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the intelligent operation and maintenance method for power equipment based on a multimodal collaborative attention mechanism.
[0016] Compared with the prior art, the present invention has at least the following beneficial technical effects: (1) Deep scene understanding capability: Through the multimodal collaborative attention mechanism, a three-dimensional and all-round understanding of equipment operation and maintenance scenarios is realized, which surpasses the limitations of single-modal analysis; (2) Precise feature association: Cross-modal collaborative attention can automatically discover and strengthen the semantic association between different modalities, such as accurately corresponding the fault phenomena described in the text with abnormal areas in the image and abnormal fluctuations in the data; (3) Adaptive weight adjustment: The importance of each mode is dynamically adjusted according to the specific operation and maintenance task. For example, real-time data is emphasized when diagnosing faults, and all modes are considered in a balanced manner when assessing the status. (4) Strong generalization ability: The collaborative attention mechanism can learn general multimodal association patterns, which are applicable to different types of main devices and operation and maintenance scenarios; (5) High engineering practicality: It can be directly integrated into the existing intelligent operation and maintenance platform to improve operation and maintenance efficiency and accuracy. Attached Figure Description
[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0018] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a structural block diagram of the system of the present invention. Detailed Implementation
[0019] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.
[0020] In the description of this invention, it should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0021] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0022] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0023] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.
[0024] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0025] Example 1 like Figure 1 As shown, the intelligent operation and maintenance method for power equipment based on a multimodal collaborative attention mechanism provided by this invention includes the following steps: S1: Collect multimodal data and perform preprocessing; S2: Extract multimodal features from the preprocessed multimodal data. Multimodal features include visual features, semantic features, and temporal features. S3: Based on the extracted multimodal features, perform multimodal collaborative attention calculation and construct a three-layer collaborative attention mechanism; S4: Based on a three-layer collaborative attention mechanism, multimodal feature fusion and scene understanding are performed. The multimodal features enhanced by collaborative attention are deeply fused and input into the scene understanding module, and output equipment status assessment, fault warning and maintenance suggestion information. S5: Output and application of scenario understanding results, generating comprehensive scenario understanding reports to provide visual support for operation and maintenance decisions.
[0026] In this embodiment, S1: Collecting multimodal data and performing preprocessing includes: Collect equipment inspection images, maintenance text data, and real-time equipment operation data, and standardize and normalize all types of data.
[0027] In this embodiment, S2: Multimodal feature extraction is performed on the preprocessed multimodal data, including: Visual features of inspection images are extracted using a visual encoder. Use a text encoder to extract semantic features from the operation and maintenance text. Use a time encoder to extract the temporal features of real-time data. .
[0028] In this embodiment, S3: Based on the extracted multimodal features, multimodal collaborative attention calculation is performed to construct a three-layer collaborative attention mechanism, including: S3.1: Intramodal self-attention layer, calculating self-attention within each modality to enhance the weights of important features. (Formula omitted) ,in , , These are the Query, Key, and Value matrices, respectively, and the Softmax function is a mapping function. S3.2: Cross-modal collaborative attention layer, designing bidirectional cross-modal attention interaction; from visual to textual attention, allowing semantic features to focus on relevant visual regions, the formula is as follows. From text to visual attention, enabling visual features to focus on relevant textual descriptions, the formula is as follows: From real-time data to multimodal attention, it establishes a correlation between real-time data features and image and semantic features, as shown in the formula. ; S3.3: Modal weight adaptive layer, which dynamically adjusts the contribution weight of each modality according to the current operation and maintenance task and scenario complexity. The weight factor is calculated based on the confidence level of each modality feature and the task relevance.
[0029] Example 2 like Figure 1 As shown, the intelligent operation and maintenance method for power equipment based on a multimodal collaborative attention mechanism provided by this invention includes the following steps: S1: Multimodal data acquisition and preprocessing, acquiring equipment inspection images, maintenance text data and real-time equipment operation data, and standardizing and normalizing various types of data; S2: Multimodal feature extraction, using a visual encoder to extract visual feature vectors from the point inspection images. Use a text encoder to extract semantic feature vectors from the operation and maintenance text. Use a time encoder to extract time-series feature vectors from real-time data. ; S3: Multimodal collaborative attention computation, constructing a three-layer collaborative attention mechanism: S3.1: Intramodal self-attention layer, calculating self-attention within each modality to enhance the weights of important features. (Formula omitted) ,in , , These are the Query, Key, and Value matrices, respectively. The Softmax function is a mapping function, and the same applies to other modalities. S3.2: Cross-modal collaborative attention layer, designing bidirectional cross-modal attention interaction. From visual to textual attention, semantic features focus on relevant visual regions, as shown in the formula. From text to visual attention, enabling visual features to focus on relevant textual descriptions, the formula is as follows: From real-time data to multimodal attention, it establishes a correlation between real-time data features and image and semantic features, as shown in the formula. ; S3.3: Modal weight adaptive layer, which dynamically adjusts the contribution weight of each modality according to the current operation and maintenance task and scenario complexity. The weight factor is calculated based on the confidence of each modality feature and the task relevance. S4: Multimodal Feature Fusion and Scene Understanding, deeply fuses multimodal features enhanced by collaborative attention. The input is fed into the scene understanding module, which outputs information such as device status assessment, fault warnings, and maintenance suggestions. S5: Output and application of scenario understanding results, generating comprehensive scenario understanding reports to provide visual support for operation and maintenance decisions.
[0030] Example 3 The following uses the intelligent operation and maintenance of a gas turbine in a power plant as an example to illustrate the specific implementation of this invention, including: S1: Data acquisition includes image data (photos of the gas turbine exterior, control panel indicator lights, and pressure gauge readings taken by inspection personnel), text data (the maintenance log entry for "#3 bearing vibration increased, accompanied by abnormal noise"), and fault descriptions in the maintenance report), and real-time data (time-series signals such as vibration acceleration, bearing temperature, speed, and power collected by the DCS system).
[0031] S2: Feature extraction. Visual features are extracted using ResNet to extract device status features from inspection images. Semantic features are extracted using a BERT encoder to extract semantic features from maintenance text. Temporal features are extracted using an LSTM network to extract temporal features from vibration and temperature data.
[0032] S3: Collaborative attention calculation. Intramodal self-attention strengthens the weight of abnormal regions in visual features, such as equipment oil stains and wear. Using cross-modal collaborative attention, a strong correlation is established between "increased vibration" in the text and vibration spectrum features in real-time data; "abnormal sound" in the text is associated with loose parts in the image; and real-time temperature data is associated with the radiator status in the image. Finally, modal weight adaptation is introduced. The current task is "bearing fault diagnosis," so the weight of real-time vibration data is the highest at 0.5, followed by text description at 0.3, and visual features at 0.2.
[0033] S4: Feature fusion and scene understanding. The fused multimodal features are input into the classifier, which outputs the diagnostic result of "early bearing wear" and generates confidence scores and main judgment criteria.
[0034] S5: The system outputs: "Early wear signs detected in bearing #3. It is recommended to check and replace it during the next shutdown." It also provides a visual interface that highlights the relevant image area, text description, and data trend.
[0035] Example 4 like Figure 2 As shown, the intelligent operation and maintenance system for power equipment based on a multimodal collaborative attention mechanism provided by the present invention includes: Data acquisition and preprocessing unit: Acquires multimodal data and performs preprocessing; Multimodal feature extraction unit: Extracts multimodal features from preprocessed multimodal data. Multimodal features include visual features, semantic features, and temporal features. Three-layer collaborative attention mechanism construction unit: Based on the extracted multimodal features, multimodal collaborative attention calculation is performed to construct a three-layer collaborative attention mechanism; Multimodal feature fusion and scene understanding unit: Based on a three-layer collaborative attention mechanism, it performs multimodal feature fusion and scene understanding, deeply fuses the multimodal features enhanced by collaborative attention, inputs them into the scene understanding module, and outputs equipment status assessment, fault warning and maintenance suggestion information; Scenario Understanding Report Generation Unit: Outputs and applies scenario understanding results, generates comprehensive scenario understanding reports, and provides visual support for operation and maintenance decisions.
[0036] In the data acquisition and preprocessing unit of this embodiment: acquiring multimodal data and performing preprocessing includes: Collect equipment inspection images, maintenance text data, and real-time equipment operation data, and standardize and normalize all types of data.
[0037] In the multimodal feature extraction unit of this embodiment: multimodal feature extraction is performed on the preprocessed multimodal data, including: Visual features of inspection images are extracted using a visual encoder. Use a text encoder to extract semantic features from the operation and maintenance text. Use a time encoder to extract the temporal features of real-time data. .
[0038] In the three-layer collaborative attention mechanism construction unit of this embodiment: based on the extracted multimodal features, multimodal collaborative attention calculation is performed to construct a three-layer collaborative attention mechanism, including: S3.1: Intramodal self-attention layer, calculating self-attention within each modality to enhance the weights of important features. (Formula omitted) ,in , , These are the Query, Key, and Value matrices, respectively, and the Softmax function is a mapping function. S3.2: Cross-modal collaborative attention layer, designing bidirectional cross-modal attention interaction; from visual to textual attention, allowing semantic features to focus on relevant visual regions, the formula is as follows. From text to visual attention, enabling visual features to focus on relevant textual descriptions, the formula is as follows: From real-time data to multimodal attention, it establishes a correlation between real-time data features and image and semantic features, as shown in the formula. ; S3.3: Modal weight adaptive layer, which dynamically adjusts the contribution weight of each modality according to the current operation and maintenance task and scenario complexity. The weight factor is calculated based on the confidence level of each modality feature and the task relevance.
[0039] Example 5 The present invention provides an electronic device comprising: a processor and a memory coupled to the processor, the memory storing a computer program, wherein when the computer program is executed by the processor, it implements the steps of the intelligent operation and maintenance method for power equipment based on a multimodal collaborative attention mechanism.
[0040] The electronic device may also include one or more of a multimedia component, an input / output (I / O) interface, and a communication component.
[0041] The processor controls the overall operation of the electronic device to complete all or part of the steps in the storage medium sharing method. The memory stores various types of data to support the operation of the electronic device. This data may include, for example, instructions for any application or method operating on the electronic device, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia components may include a screen and audio components. The screen may be, for example, a touchscreen, and the audio components are used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory or transmitted via a communication component. The audio component also includes at least one speaker for outputting audio signals. The I / O interface provides an interface between the processor and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical. The communication component is used for wired or wireless communication between the electronic device and other devices. Wireless communication includes Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof; therefore, the corresponding communication component may include a Wi-Fi module, a Bluetooth module, or an NFC module.
[0042] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing a storage medium sharing method.
[0043] Example 6 The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the intelligent operation and maintenance method for power equipment based on a multimodal collaborative attention mechanism.
[0044] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0045] This application is described with reference to flowchart illustrations and / or block diagrams of methods, systems, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.
[0046] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0047] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0048] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the scope of the invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
[0049] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that can be understood by those skilled in the art. The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.
Claims
1. A method for intelligent operation and maintenance of power equipment based on a multimodal collaborative attention mechanism, characterized in that, Includes the following steps: S1: Collect multimodal data and perform preprocessing; S2: Extract multimodal features from the preprocessed multimodal data. Multimodal features include visual features, semantic features, and temporal features. S3: Based on the extracted multimodal features, perform multimodal collaborative attention calculation and construct a three-layer collaborative attention mechanism; S4: Based on a three-layer collaborative attention mechanism, multimodal feature fusion and scene understanding are performed. The multimodal features enhanced by collaborative attention are deeply fused and input into the scene understanding module, and output equipment status assessment, fault warning and maintenance suggestion information. S5: Output and application of scenario understanding results, generating comprehensive scenario understanding reports to provide visual support for operation and maintenance decisions.
2. The intelligent operation and maintenance method for power equipment based on a multimodal collaborative attention mechanism according to claim 1, characterized in that, S1: Acquire multimodal data and perform preprocessing, including: Collect equipment inspection images, maintenance text data, and real-time equipment operation data, and standardize and normalize all types of data.
3. The intelligent operation and maintenance method for power equipment based on a multimodal collaborative attention mechanism according to claim 1, characterized in that, S2: Perform multimodal feature extraction on the preprocessed multimodal data, including: Visual features of inspection images are extracted using a visual encoder. Use a text encoder to extract semantic features from the operation and maintenance text. Use a time encoder to extract the temporal features of real-time data. .
4. The intelligent operation and maintenance method for power equipment based on a multimodal collaborative attention mechanism according to claim 1, characterized in that, S3: Based on the extracted multimodal features, perform multimodal collaborative attention calculation and construct a three-layer collaborative attention mechanism, including: S3.1: Intramodal self-attention layer, calculating self-attention within each modality to enhance the weights of important features. (Formula omitted) ,in , , These are the Query, Key, and Value matrices, respectively, and the Softmax function is a mapping function. S3.2: Cross-modal collaborative attention layer, designing bidirectional cross-modal attention interaction; from visual to textual attention, allowing semantic features to focus on relevant visual regions, the formula is as follows. From text to visual attention, enabling visual features to focus on relevant textual descriptions, the formula is as follows: From real-time data to multimodal attention, it establishes a correlation between real-time data features and image and semantic features, as shown in the formula. ; S3.3: Modal weight adaptive layer, which dynamically adjusts the contribution weight of each modality according to the current operation and maintenance task and scenario complexity. The weight factor is calculated based on the confidence level of each modality feature and the task relevance.
5. A power equipment intelligent operation and maintenance system based on a multimodal collaborative attention mechanism, characterized in that, include: Data acquisition and preprocessing unit: Acquires multimodal data and performs preprocessing; Multimodal feature extraction unit: Extracts multimodal features from preprocessed multimodal data. Multimodal features include visual features, semantic features, and temporal features. Three-layer collaborative attention mechanism construction unit: Based on the extracted multimodal features, multimodal collaborative attention calculation is performed to construct a three-layer collaborative attention mechanism; Multimodal feature fusion and scene understanding unit: Based on a three-layer collaborative attention mechanism, it performs multimodal feature fusion and scene understanding, deeply fuses the multimodal features enhanced by collaborative attention, inputs them into the scene understanding module, and outputs equipment status assessment, fault warning and maintenance suggestion information; Scenario Understanding Report Generation Unit: Outputs and applies scenario understanding results, generates comprehensive scenario understanding reports, and provides visual support for operation and maintenance decisions.
6. The intelligent operation and maintenance system for power equipment based on a multimodal collaborative attention mechanism according to claim 5, characterized in that, In the data acquisition and preprocessing unit: multimodal data is acquired and preprocessed, including: Collect equipment inspection images, maintenance text data, and real-time equipment operation data, and standardize and normalize all types of data.
7. The intelligent operation and maintenance system for power equipment based on a multimodal collaborative attention mechanism according to claim 5, characterized in that, In the multimodal feature extraction unit: multimodal feature extraction is performed on the preprocessed multimodal data, including: Visual features of inspection images are extracted using a visual encoder. Use a text encoder to extract semantic features from the operation and maintenance text. Use a time encoder to extract the temporal features of real-time data. .
8. The intelligent operation and maintenance system for power equipment based on a multimodal collaborative attention mechanism according to claim 5, characterized in that, In the three-layer collaborative attention mechanism construction unit: based on the extracted multimodal features, multimodal collaborative attention calculation is performed to construct a three-layer collaborative attention mechanism, including: S3.1: Intramodal self-attention layer, calculating self-attention within each modality to enhance the weights of important features. (Formula omitted) ,in , , These are the Query, Key, and Value matrices, respectively, and the Softmax function is a mapping function. S3.2: Cross-modal collaborative attention layer, designing bidirectional cross-modal attention interaction; from visual to textual attention, allowing semantic features to focus on relevant visual regions, the formula is as follows. From text to visual attention, enabling visual features to focus on relevant textual descriptions, the formula is as follows: From real-time data to multimodal attention, it establishes a correlation between real-time data features and image and semantic features, as shown in the formula. ; S3.3: Modal weight adaptive layer, which dynamically adjusts the contribution weight of each modality according to the current operation and maintenance task and scenario complexity. The weight factor is calculated based on the confidence level of each modality feature and the task relevance.
9. An electronic device, characterized in that, include: A processor and a memory coupled to the processor, the memory storing a computer program that, when executed by the processor, implements the steps of the intelligent operation and maintenance method for power equipment based on a multimodal collaborative attention mechanism as described in any one of claims 1-4.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the intelligent operation and maintenance method for power equipment based on a multimodal collaborative attention mechanism as described in any one of claims 1-4.