Integrated identification method and equipment for power generation equipment anomalies based on multimodal deep learning

Abstract

This invention proposes a multimodal deep learning integrated power generation equipment anomaly identification method and device. This method constructs a power generation equipment anomaly detection neural network, extracts features from unimodal data using a feature extraction network, maps the extracted features to the same semantic space, and employs a multimodal fusion strategy to fuse the semantic features of the unimodal data to generate multimodal fusion features. Based on the generated multimodal fusion features, power generation equipment anomaly prediction is performed. This invention can avoid omissions, false alarms, and misreporting during power generation equipment anomaly detection, improving the accuracy of power generation equipment anomaly detection.

Description

Technical Field

[0001] The present invention relates to the technical fields of deep learning, artificial intelligence, neural networks and new energy, and in particular to a multimodal deep learning integrated abnormality identification method for power generation equipment, a device, a computer device and a storage medium. Background Art

[0002] The integration of new energy sources, such as clean energy, with core technologies like artificial intelligence and big data has become a key integrated technology. Consequently, an increasing amount of multimodal data, including voice, images, video, and text, is being collected and manually collected in applications such as photovoltaics, wind turbines, and offshore power generation. How can we maximize or optimize the value of this massive amount of data? How can we extract effective features? How can we apply rich combined feature representations to various business systems in these applications? How can we optimize the combined feature representation capabilities? These dilemmas have become challenging and thorny issues for industry, academia, and industry. Currently, existing technologies do not address these issues by integrating multimodal data with models.

[0003] At present, on the one hand, new energy fields such as wind turbines and photovoltaic scenarios only use single-modal data and multi-modal data to obtain effective feature representation, which is insufficient, such as redundant feature representation information, inaccurate feature representation, insufficient feature representation accuracy, etc., and then perform equipment anomaly detection and fault diagnosis, system defect early warning alarm, system operation and maintenance and other functions cannot meet the needs or requirements of various businesses. In particular, when a certain equipment fails, the output result of the model constructed based on the single-modal data will be judged, which will result in misreporting, omissions, late reporting, etc., which will bring immeasurable losses to the users of the new energy field, including major accidents such as casualties, individual equipment operation status monitoring, equipment health inspection, etc. On the other hand, traditional machine learning methods such as GMM, SVM, Bayesian, or Traditional machine learning methods such as joint learning cannot meet the reliability and robustness requirements of traditional machine learning models constructed with single-modal data and multi-modal data. With the rapid development of artificial intelligence, especially deep learning, making full use of multi-modal data and model fusion methods obtained from photovoltaic, wind turbines and offshore power generation equipment in the new energy field has become a major trend in various application scenarios and functional requirements. In particular, deep learning has powerful feature extraction and characterization capabilities, which can bring various benefits such as cost reduction and efficiency improvement, quality improvement and efficiency improvement to the new energy field. For example, it can reduce the dangers of manual inspection operations, intelligent health and monitoring of the system, intelligent analysis of various operating indicators, operational analysis, etc., saving manpower and time, reducing investment costs, etc. Therefore, a new technical solution is needed to facilitate the use of computers, GPUs, data, etc. Summary of the Invention

[0004] The present invention provides a multimodal deep learning integrated power generation equipment anomaly identification method, device, computer equipment and storage medium, aiming to avoid omissions, false alarms and misreporting in the power generation equipment anomaly detection process and improve the accuracy of predicted power generation equipment anomaly detection.

[0005] To this end, the first object of the present invention is to propose a multimodal deep learning integrated method for identifying abnormalities in power generation equipment, comprising:

[0006] Acquire multimodal historical data and perform data preprocessing, and use the preprocessed multimodal historical data as a training set;

[0007] Constructing a power generation equipment anomaly detection network model and training the constructed power generation equipment anomaly detection network model using a training set; wherein the power generation equipment anomaly detection network model includes a feature extraction module, a spatial mapping module, a feature fusion module, and a result prediction module connected in sequence;

[0008] The real-time multimodal data is preprocessed and input into the trained power generation equipment anomaly detection network model, and the output result is used as the detection result of whether the power generation equipment is abnormal.

[0009] The steps of obtaining multimodal historical data and performing data preprocessing include the following steps:

[0010] Multimodal historical data collection: Collects automatically uploaded multimodal historical data obtained from high-definition cameras, microphones, and sensors installed around power generation equipment, as well as manually uploaded multimodal historical data obtained by taking photos with mobile phones;

[0011] Data cleaning: Perform data cleaning on the collected multimodal historical data; the cleaning methods include at least: data removal and data completion;

[0012] Data separation: Data separation is performed on the bimodal or multimodal mixed data in the collected multimodal historical data. After data separation, the multimodal historical data is divided into voice data, text data, image data and video data, and the detection results are marked.

[0013] Among them, the power generation equipment anomaly detection network model includes a feature extraction module, a space mapping module, a feature fusion module and a result prediction module; among them,

[0014] The feature extraction module is a feature extraction neural network that is used to extract features of a single modality from the speech data, text data, image data, and video data obtained after data separation.

[0015] The spatial mapping module is used to map the features of a single modality to the same semantic space to obtain the semantic structure information within the features of the single modality data;

[0016] The feature fusion module is used to perform feature fusion and splicing of multimodal historical data on the features of a single modality to obtain multimodal feature fusion information;

[0017] The result prediction module is used to calculate the prediction results based on the multimodal feature fusion information and complete the abnormality detection of the power generation equipment.

[0018] The feature extraction neural network is a BiLSTM network, a convolutional neural network or a deep neural network; wherein, the BiLSTM network is applied to obtain contextual information of single modal data while extracting features, that is, semantic information between adjacent single modal data.

[0019] Among them, the feature fusion module includes a bidirectional attention mechanism unit, a self-attention mechanism unit and a first fully connected layer unit; the voice data, text data, image data and video data mapped and processed by the spatial mapping module are sequentially input into the bidirectional attention mechanism unit, the self-attention mechanism unit and the first fully connected layer unit, and the output data is transmitted to the fusion module for fusion.

[0020] The steps of training the constructed power generation equipment anomaly detection network model using the training set include:

[0021] The preprocessed training set data is input into the feature extraction neural network of the feature extraction module, and a BiLSTM network is used to extract single-modal features from the speech data, text data, image data, and video data in the training set data.

[0022] After extracting single modal features from the speech data, text data, image data, and video data in the training set data, semantic space mapping is performed on the single modal features of the speech data, text data, image data, and video data;

[0023] The single modal features of the speech data, text data, image data and video data output after the semantic space mapping are respectively input into the set multimodal feature fusion mechanism; the multimodal feature fusion mechanism includes a first-fourth feature fusion module and a first fusion module and a second fusion module; the single modal features of the speech data, text data, image data and video data output after the semantic space mapping are respectively input into the first-fourth feature fusion module, the output results of the first and second feature fusion modules are input into the first fusion module, and the output results of the first fusion module, the third feature fusion module and the fourth feature fusion module are input into the second fusion module;

[0024] The output result of the second fusion module is output to the result prediction module, and the prediction result is calculated using the Softmax function and compared with the marked detection result. The network training is completed by continuously adjusting the network function and parameters until the prediction result is consistent with the marked detection result.

[0025] Among them, if the abnormal detection result of the power generation equipment is obtained, the abnormal detection result will be displayed; the display method includes at least: text display, voice broadcast, outbound call terminal, email, SMS reminder, and smart speaker.

[0026] A second object of the present invention is to provide a multimodal deep learning integrated power generation equipment anomaly identification device, comprising:

[0027] The data acquisition module is used to acquire multimodal historical data and perform data preprocessing, and use the preprocessed multimodal historical data as a training set;

[0028] A network construction module is used to construct a power generation equipment anomaly detection network model and train the constructed power generation equipment anomaly detection network model using a training set; wherein the power generation equipment anomaly detection network model includes a feature extraction module, a spatial mapping module, a feature fusion module, and a result prediction module connected in sequence;

[0029] The anomaly detection module is used to pre-process the real-time multimodal data and input it into the trained power generation equipment anomaly detection network model, and output the result as the detection result of whether the power generation equipment is abnormal.

[0030] The third object of the present invention is to provide a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method of the aforementioned technical solution is implemented.

[0031] A fourth object of the present invention is to provide a non-temporary computer-readable storage medium having a computer program stored thereon, which implements the method of the aforementioned technical solution when the computer program is executed by a processor.

[0032] Differentiating from existing technologies, the present invention provides a multimodal deep learning-based integrated power generation equipment anomaly identification method. This method constructs a power generation equipment anomaly detection neural network, extracts features from unimodal data using a feature extraction network, maps the extracted features to a common semantic space, and employs a multimodal fusion strategy to fuse the semantic features of the unimodal data to generate multimodal fusion features. This multimodal fusion feature is then used to predict power generation equipment anomalies. This method can mitigate the omissions, false positives, and misreporting of power generation equipment anomaly detection, improving the accuracy of power generation equipment anomaly prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0034] Figure 1 This is a flow chart of a multimodal deep learning integrated abnormality identification method for power generation equipment provided by the present invention.

[0035] Figure 2 This is a structural schematic diagram of a power generation equipment anomaly detection network model of a multimodal deep learning integrated power generation equipment anomaly identification method provided by the present invention.

[0036] Figure 3 This is a structural schematic diagram of a multimodal deep learning integrated power generation equipment abnormality identification device provided by the present invention.

[0037] Figure 4 It is a structural schematic diagram of a non-transitory computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0038] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention, but are not to be construed as limiting the present invention.

[0039] Figure 1 A flowchart of a multimodal deep learning integrated power generation equipment anomaly identification method provided by an embodiment of the present invention. The method includes the following steps:

[0040] Step 101: Acquire multimodal historical data and perform data preprocessing, and use the preprocessed multimodal historical data as a training set.

[0041] The multimodal data mentioned in the present invention mainly refers to the four modes of text, image, voice and video. At present, because cameras, sensors, microphones and other equipment are installed in large quantities around power generation equipment for monitoring, the data they generate is often still subject to manual inspection to determine whether there is a fault. A large amount of data may be directly deleted after generation. In addition, manual inspection not only causes misjudgment, but also makes it impossible for the monitoring equipment to effectively complete the monitoring task, which easily leads to data redundancy or data waste. The present invention collects massive multimodal historical data to train the constructed detection network, improves detection accuracy, and can also effectively utilize data. The power generation equipment involved in the present invention includes but is not limited to thermal power, hydropower, clean energy power generation equipment or generator sets, wherein clean energy power generation equipment or generator sets include but are not limited to photovoltaic, offshore wind power, nuclear power generation equipment or generator sets. The abnormal integration of power generation equipment includes but is not limited to abnormal detection, fault diagnosis and fault warning of power generation equipment.

[0042] Specifically, acquiring multimodal historical data involves obtaining various monitoring devices installed to monitor the corresponding power generation equipment, such as high-definition cameras, microphones, and sensors, from the management unit to which the power generation equipment belongs. These devices automatically upload monitoring data from the power generation equipment to the monitoring center, which is referred to as automatically uploaded multimodal historical data in this invention. In addition, there is also image, video, or voice data manually uploaded via mobile phone, which is referred to as manually uploaded multimodal historical data in this invention. The acquisition of multimodal historical data involves acquiring a large amount of automatically uploaded and manually uploaded multimodal historical data.

[0043] After acquiring the data, the collected multimodal historical data needs to be cleaned. During the data cleaning process, data screening is performed to remove data irrelevant to power generation equipment anomaly detection. The quality of the multimodal historical data is then assessed. For low-quality data, data cleaning is performed using scripts or tools, such as manual noise addition or sentence completion.

[0044] After cleaning is completed, data belonging to two or more modes are selected from the multimodal data for data separation. If there is bimodal data belonging to both voice and video, it is separated with the help of tools to obtain single-modal data. Figure 2 As shown in 101.

[0045] After the data preprocessing is completed, the process proceeds to step 102 .

[0046] Step 102: construct a power generation equipment anomaly detection network model, and train the constructed power generation equipment anomaly detection network model using a training set.

[0047] The network structure of the power generation equipment abnormality detection network model constructed by the present invention is as follows: Figure 2As shown, it includes a feature extraction module 102, a space mapping module 103, a feature fusion module 104 and a result prediction module 105; wherein,

[0048] The feature extraction module 102 is a feature extraction neural network, which is used to perform single-modality feature extraction on the voice data, text data, image data, and video data obtained after data separation.

[0049] The feature extraction neural network is a BiLSTM network, a convolutional neural network or a deep neural network; wherein, the BiLSTM network is applied to obtain contextual information of single modal data, that is, semantic information between adjacent single modal data, while extracting features.

[0050] The spatial mapping module 103 is used to map the features of a single modality to the same semantic space to obtain the semantic structure information within the features of the single modality data. Through spatial mapping, the correlation within the features of the single modality data can be effectively characterized and the internal semantic structure information can be determined.

[0051] The feature fusion module 104 is used to perform feature fusion and splicing of multimodal historical data on the features of a single modality to obtain multimodal feature fusion information.

[0052] The feature fusion module 104 includes a bidirectional attention mechanism unit 10401, a self-attention mechanism unit 10402 and a first fully connected layer unit 10403; the speech data, text data, image data and video data mapped and processed by the spatial mapping module are sequentially input into the bidirectional attention mechanism unit, the self-attention mechanism unit and the first fully connected layer unit, and the output data is transmitted to the fusion module for fusion.

[0053] The result prediction module 105 is used to calculate the prediction result based on the multimodal feature fusion information and complete the abnormality detection of the power generation equipment. The result prediction module 105 includes a second fully connected unit 1051 and a classifier unit 1052. Figure 2 As shown in , the classifier unit 1052 adopts a Softmax classifier.

[0054] like Figure 2In the present invention, the network comprises four sequentially connected feature extraction modules 102 and spatial mapping modules 103 for data from four different modalities. The four preprocessed unimodal data are fed into a feature extraction BiLSTM network, which outputs the features of the four unimodal data. The features of the four unimodal data are then fed into a spatial mapping module 103 for spatial mapping. After the spatial mapping features of the four unimodal data are output, the four features are simultaneously fed into four feature fusion modules 104. In other words, each feature fusion module 104 receives the spatial mapping features of the four unimodal data. Because the collected multimodal data contributes differently to the prediction of power generation equipment anomaly detection, to enhance the weight of a particular modality in the prediction results, the spatial mapping features of text and speech are first fused in time series, and then merged with features of images and videos at different scales to obtain the final fused features. The feature fusion module 104 employs two attention mechanisms: a bidirectional attention method that exploits the dependency of interactive features between the two modalities; and a self-attention method that exploits the correlation between the prediction results and a single modality itself.

[0055] The steps of training the constructed power generation equipment anomaly detection network model using the training set include:

[0056] The preprocessed training set data is input into the feature extraction neural network of the feature extraction module 102, and single-modal feature extraction is performed on the speech data, text data, image data and video data in the training set data through a BiLSTM network.

[0057] After single modality feature extraction is performed on the speech data, text data, image data and video data in the training set data, semantic space mapping is performed on the single modality features of the speech data, text data, image data and video data.

[0058] The single modal features of the speech data, text data, image data and video data output after the semantic space mapping are respectively input into the set multimodal feature fusion mechanism; the multimodal feature fusion mechanism includes the first to fourth feature fusion modules 1041-1044, and the first fusion module 1045 and the second fusion module 1046; the single modal features of the speech data, text data, image data and video data output after the semantic space mapping are respectively input into the first to fourth feature fusion modules 1041-1044, the output results of the first feature fusion module 1041 and the second feature fusion module 1042 are input into the first fusion module 1045, and the output results of the first fusion module 1045, the third feature fusion module 1043 and the fourth feature fusion module 1044 are input into the second fusion module 1046.

[0059] The output result of the second fusion module 1046 is output to the result prediction module 105, and the prediction result is calculated using the Softmax function and compared with the marked detection result. The network training is completed by continuously adjusting the network function and parameters until the prediction result is consistent with the marked detection result.

[0060] S103: pre-processing the real-time multimodal data and inputting it into the trained power generation equipment anomaly detection network model, and outputting the result as the detection result of whether the power generation equipment is abnormal.

[0061] If the abnormality detection result of the power generation equipment is obtained, the abnormality detection result will be displayed; the display method includes at least: text display, voice broadcast, outbound call terminal, email, SMS reminder, and smart speaker. Figure 2 As shown in Figure 107.

[0062] In order to realize the above embodiment, the present invention also proposes a multi-modal deep learning integrated recognition device for abnormality of power generation equipment, such as Figure 3 Shown, including:

[0063] A data acquisition module 310 is used to acquire multimodal historical data and perform data preprocessing, and use the preprocessed multimodal historical data as a training set;

[0064] A network construction module 320 is used to construct a power generation equipment anomaly detection network model and train the constructed power generation equipment anomaly detection network model using a training set; wherein the power generation equipment anomaly detection network model includes a feature extraction module, a spatial mapping module, a feature fusion module, and a result prediction module connected in sequence;

[0065] The anomaly detection module 330 is used to pre-process the real-time multimodal data and input it into the trained power generation equipment anomaly detection network model, and output the result as the detection result of whether the power generation equipment is abnormal.

[0066] In order to implement the above embodiment, the present invention also proposes another computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, abnormality detection of the power generation equipment according to the embodiment of the present invention is implemented.

[0067] like Figure 4As shown, the non-transitory computer-readable storage medium includes an instruction memory 810 and an interface 830. The instructions can be executed by a processor 820 of the coal mining equipment travel speed estimation device to perform the above method. Alternatively, the storage medium can be a non-transitory computer-readable storage medium, for example, a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0068] In order to implement the above embodiment, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the abnormality detection of the power generation equipment according to the embodiment of the present invention is implemented.

[0069] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0070] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0071] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0072] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0073] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0074] Those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment may be accomplished by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0075] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0076] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and are not to be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A multimodal deep learning integrated recognition method for power generation equipment anomalies, characterized in that: include: Acquire and preprocess multimodal historical data, using the preprocessed data as a training set. The multimodal historical data includes automatically uploaded multimodal historical data collected from high-definition cameras, microphones, and sensors installed around the power generation equipment, as well as manually uploaded multimodal historical data collected by taking photos with a mobile phone. Constructing a power generation equipment anomaly detection network model, and training the constructed power generation equipment anomaly detection network model using the training set; wherein the power generation equipment anomaly detection network model includes a feature extraction module, a spatial mapping module, a feature fusion module, and a result prediction module connected in sequence, wherein the feature extraction module is a feature extraction neural network for performing single-modal feature extraction on the voice data, text data, image data, and video data obtained after data separation, and applying a BiLSTM network to obtain contextual information of the single-modal data, i.e., semantic information between adjacent single-modal data, while extracting features; The spatial mapping module is used to map the features of a single modality to the same semantic space to obtain the semantic structure information within the features of the single modality data; The feature fusion module is used to perform feature fusion and splicing of multimodal historical data on the features of a single modality to obtain multimodal feature fusion information; The result prediction module is used to calculate the prediction result based on the multimodal feature fusion information to complete the abnormality detection of the power generation equipment; The real-time multimodal data is pre-processed and input into the trained power generation equipment anomaly detection network model, and the output result is used as the detection result of whether the power generation equipment is abnormal.

2. The method for integrated recognition of power generation equipment anomalies based on multimodal deep learning according to claim 1 is characterized in that: The steps of obtaining multimodal historical data and performing data preprocessing include the following steps: Data cleaning: Perform data cleaning on the collected multimodal historical data; the cleaning methods include at least: data removal and data completion; Data separation: Data separation is performed on the bimodal or multimodal mixed data in the collected multimodal historical data. After data separation, the multimodal historical data is divided into voice data, text data, image data and video data, and the detection results are marked.

3. The method for integrated recognition of power generation equipment anomalies based on multimodal deep learning according to claim 1 is characterized in that: The feature extraction neural network is a BiLSTM network, a convolutional neural network or a deep neural network.

4. The method for integrated recognition of power generation equipment anomalies based on multimodal deep learning according to claim 1 is characterized in that: The feature fusion module includes a bidirectional attention mechanism unit, a self-attention mechanism unit and a first fully connected layer unit; The speech data, text data, image data and video data mapped and processed by the spatial mapping module are sequentially input into the bidirectional attention mechanism unit, the self-attention mechanism unit and the first fully connected layer unit, and the output data is transmitted to the fusion module for fusion.

5. The method for integrated recognition of power generation equipment anomalies based on multimodal deep learning according to claim 3 is characterized in that: The step of training the constructed power generation equipment anomaly detection network model using the training set includes: The preprocessed training set data is input into the feature extraction neural network of the feature extraction module, and a BiLSTM network is used to extract single-modal features from the speech data, text data, image data, and video data in the training set data. After extracting single modal features from the speech data, text data, image data, and video data in the training set data, semantic space mapping is performed on the single modal features of the speech data, text data, image data, and video data; The single modal features of the speech data, text data, image data and video data output after the semantic space mapping are respectively input into the set multimodal feature fusion mechanism; the multimodal feature fusion mechanism includes a first-fourth feature fusion module and a first fusion module and a second fusion module; the single modal features of the speech data, text data, image data and video data output after the semantic space mapping are respectively input into the first-fourth feature fusion modules, the output results of the first and second feature fusion modules are input into the first fusion module, and the output results of the first fusion module, the third feature fusion module and the fourth feature fusion module are input into the second fusion module; The output result of the second fusion module is output to the result prediction module, and the prediction result is calculated using the Softmax function and compared with the marked detection result. The network training is completed by continuously adjusting the network function and parameters until the prediction result is consistent with the marked detection result.

6. The method for integrated recognition of power generation equipment anomalies based on multimodal deep learning according to claim 1 is characterized in that: If abnormal detection results of power generation equipment are obtained, the abnormal detection results will be displayed; The display methods include at least: text display, voice broadcast, outbound call terminal, email, SMS reminder, and smart speaker.

7. A multimodal deep learning integrated power generation equipment abnormality identification device, characterized in that: include: The data acquisition module is used to acquire and preprocess multimodal historical data, using the preprocessed multimodal historical data as a training set. The multimodal historical data includes: automatically uploaded multimodal historical data collected by high-definition cameras, microphones, and sensors installed around the power generation equipment, and manually uploaded multimodal historical data collected by taking photos with a mobile phone; A network construction module for constructing a power generation equipment anomaly detection network model and training the constructed power generation equipment anomaly detection network model using the training set; wherein the power generation equipment anomaly detection network model includes a feature extraction module, a spatial mapping module, a feature fusion module, and a result prediction module connected in sequence; wherein the feature extraction module is a feature extraction neural network for performing single-modal feature extraction on the voice data, text data, image data, and video data obtained after data separation, and applying a BiLSTM network to obtain contextual information of the single-modal data, i.e., semantic information between adjacent single-modal data, while extracting features; The spatial mapping module is used to map the features of a single modality to the same semantic space to obtain the semantic structure information within the features of the single modality data; The feature fusion module is used to perform feature fusion and splicing of multimodal historical data on the features of a single modality to obtain multimodal feature fusion information; The result prediction module is used to calculate the prediction result based on the multimodal feature fusion information to complete the abnormality detection of the power generation equipment; The anomaly detection module is used to pre-process the real-time multimodal data and input it into the trained power generation equipment anomaly detection network model, and output the result as the detection result of whether the power generation equipment is abnormal.

8. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images