Photovoltaic station monitoring method and device based on multi-modal fusion, medium and equipment

Abstract

The invention discloses a photovoltaic station monitoring method and device based on multi-modal fusion, a storage medium and electronic equipment. The method comprises the following steps: collecting multi-modal monitoring data of a photovoltaic station, wherein the multi-modal monitoring data comprises image data, audio signals and sensor data; inputting the image data into a trained target detection model for target detection to obtain an image recognition result; inputting the audio signal into a recursive convolutional neural network for feature extraction to obtain a sound recognition result; inputting the sensor data into a mathematical statistical analysis model for classification detection, and outputting an abnormal equipment signal; and fusing the image recognition result, the sound recognition result and the abnormal equipment signal to obtain a fusion result, and when the fusion result meets a preset early warning rule, sending out an early warning signal. According to the invention, the potential safety hazard of the photovoltaic station can be comprehensively and accurately identified.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a photovoltaic station monitoring method, device, storage medium and electronic device based on multi-modal fusion. Background Art

[0002] In the context of global climate change, countries around the world are generally committed to formulating and implementing energy-saving and emission-reduction policies to promote low-carbon living and the development of sustainable energy industries. Faced with the gradual depletion of energy resources and environmental pollution, the development and utilization of renewable energy has become the focus of global attention. Among them, solar photovoltaic power generation technology, as a clean and sustainable energy solution, is gaining attention from more and more countries and is rapidly expanding around the world.

[0003] In the safety management of modern photovoltaic sites, traditional methods of identifying hidden dangers mainly rely on manual inspections and single sensor monitoring. These methods are not only inefficient, but also prone to omissions and cannot fully and accurately identify various potential safety hazards. Summary of the invention

[0004] The embodiments of the present application provide a photovoltaic station monitoring method, device, storage medium and electronic device based on multi-modal fusion, which can comprehensively and accurately identify various potential safety hazards in photovoltaic stations.

[0005] The present application embodiment provides a photovoltaic station monitoring method based on multi-modal fusion, including: Collecting multimodal monitoring data of a photovoltaic station, wherein the multimodal monitoring data includes image data, audio signals and sensor data; Inputting the image data into a trained target detection model to perform target detection and obtain an image recognition result; Inputting the audio signal into a recursive convolutional neural network for feature extraction to obtain a sound recognition result; Inputting the sensor data into a mathematical statistical analysis model for classification detection and outputting abnormal device signals; The image recognition result, the sound recognition result and the abnormal device signal are fused to obtain a fusion result, and when the fusion result meets a preset warning rule, a warning signal is issued.

[0006] Furthermore, in the photovoltaic station monitoring method based on multimodal fusion, the target detection model includes a backbone network, a multi-scale attention module and a target detection module; The step of inputting the image data into a trained target detection model to perform target detection and obtain an image recognition result comprises: Inputting the image data into a backbone network for feature extraction to obtain an initial feature map; Inputting the initial feature map into the multi-scale attention module to perform multi-scale feature extraction to obtain a plurality of intermediate feature maps of different scales, and fusing the plurality of intermediate feature maps of different scales to obtain a multi-scale feature map; The multi-scale feature map is input into the target detection module for prediction to obtain an image recognition result.

[0007] Furthermore, in the photovoltaic station monitoring method based on multimodal fusion, the recursive convolutional neural network includes a sound preprocessing module, a feature extraction module, an attention mechanism module, and a classification and recognition module; wherein the feature processing module includes a convolutional layer and a recursive layer; The step of inputting the audio signal into a recursive convolutional neural network for feature extraction to obtain a sound recognition result includes: Inputting the audio signal into the sound preprocessing module, and performing signal emphasis processing and noise elimination processing on the audio signal respectively; The audio signal after passing through the sound preprocessing module is input into the feature extraction module, a convolution operation is performed on the audio signal through a convolution layer to obtain local features, and a time series capture of the local features is performed through a recursive layer to obtain a feature vector; Inputting the feature vector into the attention mechanism module for weight adjustment; The feature vector after weight adjustment is input into the classification and recognition module to obtain the sound recognition result.

[0008] Furthermore, in the photovoltaic station monitoring method based on multimodal fusion, the mathematical statistical analysis model includes a data preprocessing module, an analysis module and an early warning module; The step of inputting the sensor data into a mathematical statistical analysis model for classification detection and outputting an abnormal device signal comprises: Inputting the sensor data into the data preprocessing module for preprocessing; Inputting the preprocessed sensor data into an analysis module for mathematical analysis to obtain a classification result of the sensor data; The classification result is input into the early warning module, the classification result is analyzed according to the preset early warning rules, the early warning level is determined, and an abnormal equipment signal is issued according to the early warning level.

[0009] Furthermore, in the photovoltaic station monitoring method based on multimodal fusion, the image recognition result, the sound recognition result and the abnormal device signal are fused to obtain a fusion result, and when the fusion result meets a preset warning rule, issuing a warning signal includes: respectively setting weights of the image recognition result, the sound recognition result and the abnormal device signal, and calculating a fusion result through a logical AND operation; The fusion result is input into the fully connected network layer, and the fusion result is evaluated by using softmax to obtain an evaluation result; According to the evaluation result and the preset warning rules, a warning signal is issued.

[0010] Furthermore, in the above-mentioned photovoltaic station monitoring method based on multimodal fusion, the mathematical statistical analysis model is a random forest model, a logistic regression model, or a decision tree model.

[0011] The embodiment of the present application also provides a photovoltaic station monitoring device based on multi-modal fusion, including: A collection module, used to collect multimodal monitoring data of a photovoltaic station, wherein the multimodal monitoring data includes image data, audio signals and sensor data; An image recognition module is used to input the image data into a trained target detection model to perform target detection and obtain an image recognition result; A sound recognition module, used to input the audio signal into a recursive convolutional neural network for feature extraction to obtain a sound recognition result; A data analysis module, used to input the sensor data into a mathematical statistical analysis model for classification detection and output abnormal device signals; The fusion and output module is used to fuse the image recognition result, the sound recognition result and the abnormal device signal to obtain a fusion result, and when the fusion result meets the preset warning rule, a warning signal is issued.

[0012] An embodiment of the present application further provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are suitable for being loaded by a processor to execute any of the above-mentioned photovoltaic station monitoring methods based on multi-modal fusion.

[0013] An embodiment of the present application also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used for the steps in the photovoltaic station monitoring method based on multimodal fusion as described in any of the above items.

[0014] This application collects multimodal monitoring data separately and analyzes them separately, then fuses the analysis results, and issues a warning signal when the fusion results meet the preset warning rules. This application solves the data island problem and achieves efficient use of data by analyzing and fusing multimodal monitoring data separately. In addition, this application analyzes and identifies multimodal monitoring data through target detection models, recursive convolutional neural networks, and mathematical statistical analysis models, and can comprehensively and accurately identify safety hazards of photovoltaic stations. This application can automatically complete data collection, processing, analysis, and warning, reducing dependence on manual labor and improving the efficiency and effectiveness of safety production management. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The technical solution and other beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0016] Figure 1 A flowchart of a photovoltaic station monitoring method based on multimodal fusion provided in an embodiment of the present application.

[0017] Figure 2 A schematic diagram of the structure of a photovoltaic station monitoring device based on multimodal fusion provided in an embodiment of the present application.

[0018] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0019] Figure 4 Another schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0021] The existing problems in the monitoring process of photovoltaic stations in the power industry include: 1. Most of the safety inspections at the stations are still done manually on the spot and rely on a single type of sensor for monitoring. This traditional method is inefficient in actual operation and is prone to omissions. Due to human subjective judgment and the singleness of the sensor, it is impossible to comprehensively and accurately identify potential safety hazards, which undoubtedly brings certain risks to the safety of the station.

[0022] 2. In existing smart stations, the identification of safety anomalies mainly relies on image recognition systems and single sensor monitoring systems. However, these two systems have their own obvious limitations. The image recognition system may not be able to accurately identify some non-visual safety hazards, while the single sensor monitoring system may not be able to cover all safety hazards. Therefore, these two systems cannot achieve the fusion of multimodal data, which in turn affects the accuracy of safety hazard identification.

[0023] 3. Although the AI ​​big data analysis technology used in existing smart stations has been improved in some aspects, it still cannot achieve a significant improvement in real-time performance and a deeper degree of data fusion. This makes it impossible for stations to meet real-time requirements and to achieve full automation from data collection, processing, analysis to early warning, which to some extent limits the safety management efficiency of smart stations.

[0024] To solve the above problems, the embodiments of the present application provide a photovoltaic station monitoring method, device, storage medium and electronic device based on multimodal fusion. The photovoltaic station monitoring device based on multimodal fusion provided in the embodiments of the present application can be integrated in an electronic device, which can be a terminal, a server and other devices, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other devices.

[0025] See also Figure 1 , Figure 1 A flow chart of a photovoltaic station monitoring method based on multimodal fusion provided in an embodiment of the present application, which is applied to an electronic device, and the photovoltaic station monitoring method based on multimodal fusion includes the following steps: S1, collects multimodal monitoring data of photovoltaic stations, and the multimodal monitoring data includes image data, audio signals and sensor data.

[0026] The monitoring equipment of the photovoltaic station is used to obtain video data, convert the video data into image data, and extract the audio signal from the video. The sensors of the photovoltaic station are used to obtain sensor data, including temperature sensors, humidity sensors, pressure sensors of some transformers, etc.

[0027] S2, input the image data into the trained target detection model for target detection to obtain the image recognition result.

[0028] Specifically, we obtain video stream data of monitoring equipment in different areas of photovoltaic stations, perform frame extraction, denoising, and enhancement on the video streams, and establish a photovoltaic station dataset covering various environments (including day and night and different weather conditions); among them, dangerous scenes are used as labels, including but not limited to transformer equipment oil leakage, wild fire, photovoltaic module obstruction or damage, construction workers not wearing protective measures as required, outsiders breaking in, extreme weather, etc. The labeled dataset is saved, and the dataset is divided into training set, validation set, and test set in a ratio of 7:2:1.

[0029] The data set is input into the target detection model for processing to obtain the prediction result. The loss value is calculated based on the difference between the prediction result and the true label. Through optimization algorithms such as gradient descent, the loss value is back-propagated to various parts of the model to update the target detection model parameters. Finally, a trained target detection model is obtained.

[0030] After the target detection model is trained, the acquired image data is input into the EMA-RT-DETR target detection model. The EMA-RT-DETR target detection model is obtained by integrating the multi-scale attention module (EMA) into the backbone network of RT-DETR. The EMA-RT-DETR model can quickly and accurately locate the dangerous target area in the image and give the target bounding box.

[0031] In one embodiment, the target detection model includes a backbone network, a multi-scale attention module and a target detection module, and step S2 includes the following steps: S21, input the image data into the backbone network for feature extraction to obtain an initial feature map.

[0032] First, the backbone network of the EMA-RT-DETR model performs preliminary feature extraction on the input image data to generate an initial feature map. This module extracts the basic feature information of the image data and provides the necessary feature representation for subsequent multi-scale attention processing and object detection.

[0033] S22, input the initial feature map into the multi-scale attention module for multi-scale feature extraction, obtain multiple intermediate feature maps of different scales, and fuse the multiple intermediate feature maps of different scales to obtain a multi-scale feature map.

[0034] Specifically, multi-scale feature extraction is performed on the initial feature map output by the backbone network to generate multiple intermediate feature maps of different scales. Intermediate feature maps of different scales are fused, and the attention mechanism is used to strengthen key information. The attention mechanism can automatically learn the correlation between different regions and features, further improving the model's ability to capture important information. This module enhances the model's ability to understand complex scenes through multi-scale feature extraction and fusion, and can improve the accuracy and robustness of target detection.

[0035] S23, input the multi-scale feature map into the target detection module for prediction to obtain an image recognition result.

[0036] The target detection module includes a decoder, which predicts the target on the multi-scale feature map, including the category, location and bounding box of the predicted target. Based on the enhanced feature map (i.e., multi-scale feature map), the module can quickly and accurately detect dangerous targets in the photovoltaic station, such as transformer oil leakage, photovoltaic module occlusion, etc., and give the target bounding box.

[0037] S3, input the audio signal into the recursive convolutional neural network for feature extraction to obtain the sound recognition result.

[0038] In one embodiment, the recursive convolutional neural network includes a sound preprocessing module, a feature extraction module, an attention mechanism module, and a classification and recognition module; wherein the feature processing module includes a convolutional layer and a recursive layer.

[0039] Step S3 includes the following steps: S31, input the audio signal into the sound preprocessing module, and perform signal emphasis processing and noise elimination processing on the audio signal respectively.

[0040] Specifically, the audio signal is first processed with emphasis to enhance the energy of high-frequency signals, making subsequent processing more sensitive to subtle sound changes. Then noise elimination is performed, using filtering technology or machine learning methods to remove background noise in the audio signal and improve the signal-to-noise ratio. This module can improve the purity and recognizability of audio signals, providing high-quality input data for subsequent feature extraction and recognition.

[0041] S32, input the audio signal after passing through the sound preprocessing module into the feature extraction module, perform convolution operation on the audio signal through the convolution layer to obtain local features, and perform time-series capture on the local features through the recursive layer to obtain feature vectors.

[0042] The convolution layer performs convolution operations on the preprocessed audio signal through multiple convolution kernels to extract the local features of the audio signal. The regression layer introduces structures such as recursive neural networks (RNN) or long short-term memory networks (LSTM) to capture the temporal characteristics of the audio signal and achieve overall modeling of the audio signal. This module automatically extracts representative feature vectors from the audio signal, which can reflect the key information in the audio signal, such as pitch, rhythm, timbre, etc.

[0043] S33, input the feature vector into the attention mechanism module for weight adjustment.

[0044] The attention mechanism is used to dynamically adjust the feature weights at different time points to highlight the key features associated with abnormal sounds. This module improves the model's ability to capture key information, allowing the model to more accurately identify dangerous characteristic sounds in complex environments.

[0045] S34, inputting the weight-adjusted feature vector into the classification and recognition module to obtain the sound recognition result.

[0046] Specifically, feature mapping is first performed to map the extracted feature vectors into the classification space to form a classification decision boundary. Then, sound recognition is performed to classify and identify the input audio signal according to the classification decision boundary to determine whether it belongs to the dangerous characteristic sound category. This module realizes the classification and recognition of audio signals and outputs the sound recognition results, including the sound category and confidence level.

[0047] S4, inputs the sensor data into the mathematical statistical analysis model for classification detection and outputs abnormal equipment signals.

[0048] Obtain sensor environmental monitoring data, including temperature sensors, humidity sensors, and some transformer pressure sensors, etc., and input the sensor data into the mathematical statistical analysis model for classification detection to determine whether the current environment, buildings, and transformers need to be warned.

[0049] In one embodiment, the mathematical statistical analysis model includes a data preprocessing module, an analysis module and an early warning module, and step S4 includes the following steps: S41, inputting the sensor data into a data preprocessing module for preprocessing.

[0050] First, real-time data is obtained from each sensor through the communication protocol, and then the data is preliminarily processed, such as denoising, filtering, and missing value filling, to ensure the quality of the data. The processed data will be sent to the mathematical statistical analysis model for further analysis and prediction. This module is responsible for real-time or regular data collection from various sensors widely distributed in the environmental monitoring network, including temperature, humidity, and pressure readings of key equipment such as transformers. These raw data are the basis for subsequent analysis.

[0051] S42, inputting the preprocessed sensor data into an analysis module for mathematical analysis to obtain a classification result of the sensor data.

[0052] According to the specific needs of the photovoltaic station, combined with the data type, problem characteristics and expected goals, select the appropriate mathematical statistical analysis model. For example, logistic regression is used for binary classification problems, while decision trees and random forests can be used for multivariate analysis due to their intuitive decision paths and high accuracy.

[0053] Specifically, the collected sensor data is further cleaned and formatted to meet the data input requirements of the selected model. The selected mathematical statistical analysis model is trained with the preprocessed data to learn the mapping relationship between feature variables and warning states. The sensor data is classified and predicted through the trained mathematical statistical analysis model, and the model parameters are adjusted according to the evaluation indicators (such as accuracy, recall, etc.) to optimize the prediction performance.

[0054] The mathematical statistical analysis models that can be selected include: Logistic regression, which is suitable for handling binary classification problems. The result of linear regression is mapped to the interval [0,1] through the logistic function to indicate the probability of predicting the positive class. Decision tree, which recursively selects the optimal features for partitioning and builds a tree structure to intuitively display the decision process. Random forest, as an integrated method of decision trees, improves the stability and accuracy of the model by building multiple decision trees and combining their prediction results.

[0055] S43, inputting the classification result into the early warning module, analyzing the classification result according to the preset early warning rules, determining the early warning level, and issuing an abnormal equipment signal according to the early warning level.

[0056] Receive the prediction results from the processing module, including the warning level, warning type, etc. Analyze the prediction results according to the preset warning rules (such as warning threshold, warning level division, etc.) to determine whether a warning is needed. Send the warning information to relevant personnel or systems in various forms (such as text messages, emails, alarm sounds, etc.) to ensure timely communication and effective reception of the information. This module makes a decision on whether to issue a warning based on the prediction results of the mathematical statistical analysis model, and conveys the warning information to relevant personnel or systems in an appropriate manner so that timely response measures can be taken.

[0057] S5, fusing the image recognition result, the sound recognition result and the abnormal device signal to obtain a fusion result, and when the fusion result meets the preset warning rule, issuing a warning signal.

[0058] In one embodiment, step S5 includes: S51, respectively setting weights of the image recognition result, the sound recognition result and the abnormal device signal, and calculating the fusion result through a logical AND operation; S52, inputting the fusion result into the fully connected network layer, and using softmax to evaluate the fusion result to obtain an evaluation result; S53, issuing a warning signal according to the evaluation results and the preset warning rules.

[0059] Specifically, the image recognition results, sound recognition results and abnormal equipment signals are integrated, and after logical AND operation and weighted weighting, they are input into the FC fully connected network layer, and softmax is used to perform comprehensive evaluation and scoring. When any one or more indicators reach the warning threshold, a warning signal will be triggered. The warning signal may include sound and light alarms, SMS notifications, real-time monitoring screen pop-ups, and alarm information sent to on-site managers to remind them to take timely response measures to ensure on-site safety.

[0060] This application collects multimodal monitoring data separately and analyzes them separately, then fuses the analysis results, and issues a warning signal when the fusion results meet the preset warning rules. First, this application solves the data island problem and realizes efficient use of data by analyzing and fusing multimodal monitoring data separately. Secondly, the multimodal monitoring data is analyzed and identified through the target detection model, recursive convolutional neural network and mathematical statistical analysis model, which can comprehensively and accurately identify the safety hazards of photovoltaic stations. Moreover, in the target detection model, by grouping the channel dimensions, the spatial semantic information of each channel can be more effectively retained. Compared with the ordinary convolutional neural network, EMA-RT-DETR helps to maintain rich spatial information when multi-scale features are fused, thereby improving the detection accuracy of the model for targets of different scales. The EMA module reduces computational redundancy and reduces the computational cost of the model by grouping sub-features. This optimization enables the model to achieve faster reasoning speed while maintaining performance, meet real-time requirements, and is particularly suitable for scenarios that require rapid response. Finally, this application integrates multi-dimensional data such as sound, image, and sensor, and uses multimodal fusion technology to improve the accuracy of safety hazard identification. It can automatically complete data collection, processing, analysis and early warning, reducing dependence on manual labor and improving the efficiency and effectiveness of safety production management.

[0061] According to the method described in the above embodiment, this embodiment will be further described from the perspective of a photovoltaic field station monitoring device based on multimodal fusion. The photovoltaic field station monitoring device based on multimodal fusion can be implemented as an independent entity or integrated in an electronic device. The electronic device can be a terminal, a server, etc., wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other devices.

[0062] See also Figure 2 , Figure 2 The photovoltaic station monitoring device based on multimodal fusion provided in the embodiment of the present application is specifically described, and is applied to electronic equipment. The photovoltaic station monitoring device based on multimodal fusion may include: Collecting multimodal monitoring data of a photovoltaic station, wherein the multimodal monitoring data includes image data, audio signals and sensor data; Inputting the image data into a trained target detection model to perform target detection and obtain an image recognition result; Inputting the audio signal into a recursive convolutional neural network for feature extraction to obtain a sound recognition result; Inputting the sensor data into a mathematical statistical analysis model for classification detection and outputting abnormal device signals; The image recognition result, the sound recognition result and the abnormal device signal are fused to obtain a fusion result, and when the fusion result meets a preset warning rule, a warning signal is issued.

[0063] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0064] In addition, the embodiment of the present application also provides an electronic device, which may be a computer, a tablet computer, or other device. Figure 3 As shown, the electronic device 400 includes a processor 401 and a memory 402. The processor 401 is electrically connected to the memory 402.

[0065] The processor 401 is the control center of the electronic device 400. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or loading applications stored in the memory 402 and calling data stored in the memory 402, thereby monitoring the electronic device as a whole.

[0066] In this embodiment, the processor 401 in the electronic device 400 loads instructions corresponding to the processes of one or more application programs into the memory 402 according to the following steps, and the processor 401 runs the application programs stored in the memory 402 to implement various functions: Collecting multimodal monitoring data of a photovoltaic station, wherein the multimodal monitoring data includes image data, audio signals and sensor data; Inputting the image data into a trained target detection model to perform target detection and obtain an image recognition result; Inputting the audio signal into a recursive convolutional neural network for feature extraction to obtain a sound recognition result; Inputting the sensor data into a mathematical statistical analysis model for classification detection and outputting abnormal device signals; The image recognition result, the sound recognition result and the abnormal device signal are fused to obtain a fusion result, and when the fusion result meets a preset warning rule, a warning signal is issued.

[0067] The electronic device can implement the steps in any embodiment of the photovoltaic station monitoring method based on multimodal fusion provided in the embodiments of the present application. Therefore, it can achieve the beneficial effects that can be achieved by any photovoltaic station monitoring method based on multimodal fusion provided in the embodiments of the present invention. For details, please refer to the previous embodiments and will not be repeated here.

[0068] Figure 4 The specific structural block diagram of the electronic device provided in the embodiment of the present invention is shown, and the electronic device can be used to implement the photovoltaic station monitoring method based on multimodal fusion provided in the above embodiment. The electronic device 500 can be a terminal, a server and other devices, wherein the terminal can include a tablet computer, a laptop computer, a personal computer (PC, Personal Computer), a micro processing box, or other devices.

[0069] The RF circuit 510 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 510 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks may use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0070] The memory 520 can be used to store software programs and modules, such as the corresponding program instructions / modules in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520, that is, realizing the functions of taking pictures with the front camera, processing the captured images, and switching the display color of the display content on the display screen. The memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 520 may further include a memory remotely arranged relative to the processor 580, and these remote memories may be connected to the electronic device 500 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0071] The input unit 530 may be used to receive input digital or character information, and generate a keyboard and a mouse related to user settings and function control. The display unit 540 may be used to display information input by the user or information provided to the user and various graphical user interfaces, which may be composed of graphics, text, icons, videos, and any combination thereof. The display unit 540 may include a display panel 541, which may be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like.

[0072] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can transmit the electrical signal converted from the received audio data to the speaker 561, which is converted into a sound signal for output; on the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and converted into audio data, and then the audio data is output to the processor 580 for processing, and then sent to another terminal through the RF circuit 510, or the audio data is output to the memory 520 for further processing. The audio circuit 560 may also include an earplug jack to provide communication between an external headset and the electronic device 500.

[0073] The electronic device 500 can help the user receive requests, send information, etc. through the transmission module 570 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 570 is shown in the figure, it can be understood that it is not a necessary component of the electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0074] The processor 580 is the control center of the electronic device 500. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 520, and calling data stored in the memory 520, it executes various functions of the electronic device 500 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 580.

[0075] The electronic device 500 also includes a power supply 590 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 590 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0076] Although not shown, the electronic device 500 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors. One or more programs include instructions for performing the following operations: A collection module, used to collect multimodal monitoring data of a photovoltaic station, wherein the multimodal monitoring data includes image data, audio signals and sensor data; An image recognition module is used to input the image data into a trained target detection model to perform target detection and obtain an image recognition result; A sound recognition module, used to input the audio signal into a recursive convolutional neural network for feature extraction to obtain a sound recognition result; A data analysis module, used to input the sensor data into a mathematical statistical analysis model for classification detection and output abnormal device signals; The fusion and output module is used to fuse the image recognition result, the sound recognition result and the abnormal device signal to obtain a fusion result, and when the fusion result meets the preset warning rule, a warning signal is issued.

[0077] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0078] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores multiple instructions, and the instructions can be loaded by a processor to execute the steps of any embodiment of the photovoltaic field station monitoring method based on multimodal fusion provided by the embodiment of the present invention.

[0079] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0080] Since the instructions stored in the storage medium can execute the steps in any embodiment of the photovoltaic station monitoring method based on multimodal fusion provided in the embodiments of the present invention, the beneficial effects that can be achieved by any photovoltaic station monitoring method based on multimodal fusion provided in the embodiments of the present invention can be achieved. For details, please refer to the previous embodiments and will not be repeated here.

[0081] The above is a detailed introduction to a photovoltaic station monitoring method, device, storage medium and electronic device based on multimodal fusion provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A photovoltaic station monitoring method based on multi-modal fusion, characterized in that: The method comprises: Collecting multimodal monitoring data of a photovoltaic station, wherein the multimodal monitoring data includes image data, audio signals and sensor data; Inputting the image data into a trained target detection model to perform target detection and obtain an image recognition result; Inputting the audio signal into a recursive convolutional neural network for feature extraction to obtain a sound recognition result; Inputting the sensor data into a mathematical statistical analysis model for classification detection and outputting abnormal device signals; The image recognition result, the sound recognition result and the abnormal device signal are fused to obtain a fusion result, and when the fusion result meets a preset warning rule, a warning signal is issued.

2. The photovoltaic station monitoring method based on multimodal fusion according to claim 1 is characterized in that: The target detection model includes a backbone network, a multi-scale attention module and a target detection module; The inputting the image data into a trained target detection model to perform target detection and obtain an image recognition result comprises: Inputting the image data into a backbone network for feature extraction to obtain an initial feature map; Inputting the initial feature map into the multi-scale attention module to perform multi-scale feature extraction to obtain a plurality of intermediate feature maps of different scales, and fusing the plurality of intermediate feature maps of different scales to obtain a multi-scale feature map; The multi-scale feature map is input into the target detection module for prediction to obtain an image recognition result.

3. The photovoltaic station monitoring method based on multimodal fusion according to claim 1 is characterized in that: The recursive convolutional neural network includes a sound preprocessing module, a feature extraction module, an attention mechanism module, and a classification and recognition module; wherein the feature processing module includes a convolutional layer and a recursive layer; The step of inputting the audio signal into a recursive convolutional neural network for feature extraction to obtain a sound recognition result includes: Inputting the audio signal into the sound preprocessing module, and performing signal emphasis processing and noise elimination processing on the audio signal respectively; The audio signal after passing through the sound preprocessing module is input into the feature extraction module, a convolution operation is performed on the audio signal through a convolution layer to obtain local features, and a time series capture of the local features is performed through a recursive layer to obtain a feature vector; Inputting the feature vector into the attention mechanism module for weight adjustment; The feature vector after weight adjustment is input into the classification and recognition module to obtain the sound recognition result.

4. The photovoltaic station monitoring method based on multimodal fusion according to claim 1 is characterized in that: The mathematical statistical analysis model includes a data preprocessing module, an analysis module and an early warning module; The step of inputting the sensor data into a mathematical statistical analysis model for classification detection and outputting an abnormal device signal comprises: Inputting the sensor data into the data preprocessing module for preprocessing; Inputting the preprocessed sensor data into an analysis module for mathematical analysis to obtain a classification result of the sensor data; The classification result is input into the early warning module, the classification result is analyzed according to the preset early warning rules, the early warning level is determined, and an abnormal equipment signal is issued according to the early warning level.

5. The photovoltaic station monitoring method based on multimodal fusion according to claim 1 is characterized in that: The step of fusing the image recognition result, the sound recognition result and the abnormal device signal to obtain a fusion result, and issuing a warning signal when the fusion result meets a preset warning rule includes: respectively setting weights of the image recognition result, the sound recognition result and the abnormal device signal, and calculating a fusion result through a logical AND operation; The fusion result is input into the fully connected network layer, and the fusion result is evaluated by using softmax to obtain an evaluation result; According to the evaluation result and the preset warning rules, a warning signal is issued.

6. The photovoltaic station monitoring method based on multimodal fusion according to claim 1 or 4, characterized in that: The mathematical statistical analysis model is a random forest model, a logistic regression model or a decision tree model.

7. A photovoltaic station monitoring device based on multi-modal fusion, characterized in that: include: A collection module, used to collect multimodal monitoring data of a photovoltaic station, wherein the multimodal monitoring data includes image data, audio signals and sensor data; An image recognition module is used to input the image data into a trained target detection model to perform target detection and obtain an image recognition result; A sound recognition module, used to input the audio signal into a recursive convolutional neural network for feature extraction to obtain a sound recognition result; A data analysis module, used to input the sensor data into a mathematical statistical analysis model for classification detection and output abnormal device signals; The fusion and output module is used to fuse the image recognition result, the sound recognition result and the abnormal device signal to obtain a fusion result, and when the fusion result meets the preset warning rule, a warning signal is issued.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the photovoltaic station monitoring method based on multimodal fusion according to any one of claims 1 to 6.

9. An electronic device, characterized in that: It includes a processor and a memory, the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used to execute the steps in the photovoltaic station monitoring method based on multimodal fusion as described in any one of claims 1 to 6.

Images