Linkage control method and system based on multi-modal data

Abstract

The invention relates to a linkage control method and system based on multi-modal data, and belongs to the field of energy equipment monitoring. The method comprises the following steps: inputting data streams of operation data and surrounding environment data of energy equipment, which are acquired in real time, into a pre-trained machine learning model, judging whether the data streams are abnormal or not, and classifying abnormal results of the data streams; and inputting the video stream of the monitoring picture of the defence area where the energy equipment is located, which is acquired in real time, into a pre-trained deep learning model, judging whether the video stream is abnormal or not, and classifying the abnormal results of the video stream. And then the data stream and the video stream are comprehensively judged at the same time, and a corresponding control strategy is executed according to a comprehensive judgment result. According to the method, whether the equipment is abnormal or not is comprehensively judged based on the data stream and the video stream at the same time, and the safety of equipment abnormality judgment is effectively improved.

Description

Technical Field

[0001] The invention relates to a linkage control method and system based on multi-modal data, belonging to the field of energy equipment monitoring. Background Art

[0002] Currently, data acquisition technology and video analysis technology are widely used in the energy field, but the two technologies are in a state of independent application and cannot meet users' requirements for multi-functional application of acquisition terminals. It is necessary to further develop business functions that integrate and link data acquisition and video analysis to enrich the use functions of acquisition terminals and make up for application deficiencies.

[0003] First, the data collection terminal analyzes the operating status of the energy equipment through the data sent by various sensors and the operating parameters reported by the energy equipment. However, it is unable to promptly check whether the energy equipment has a false alarm through video monitoring, nor is it able to obtain alarm images or videos that can be used for evidence in real time.

[0004] For example, the Chinese patent application publication document with publication number CN118626998A discloses a full-process monitoring method and system for key equipment driven by data flow. By monitoring the data flow of key equipment, checking the real-time data of each indicator data flow of the equipment, combining the comparison of real-time collected data with historical baselines and the working mechanism of the monitored equipment, it is determined whether the real-time data flow is abnormal.

[0005] The second type is that the video access terminal calls the control strategy by monitoring the video screen and automatically mobilizes the AI ​​algorithm for intelligent recognition application, but it can only identify the results of abnormal events and cannot obtain the real causes of the abnormal events.

[0006] For example, the Chinese patent application publication document with publication number CN110198471A discloses an abnormality identification method, apparatus, intelligent device and storage medium, which inputs the video stream collected by the intelligent device in real time into the abnormality judgment model to identify whether there is any abnormality in the real-time video screen.

[0007] When a single solution is used to judge the abnormality of the equipment, it is easy to miss the judgment. For example, a fire occurs around the equipment but the temperature in the data stream has not been detected to reach an abnormal state. At this time, control cannot be made in time; the temperature of the equipment itself is too high, and the video stream cannot detect that there is an abnormality in the equipment itself. At this time, control cannot be made. As a result, it is impossible to respond to the equipment abnormality in time, affecting the safe operation of the equipment. Summary of the invention

[0008] The purpose of the present invention is to provide a linkage control method and system based on multimodal data, so as to solve the problem that abnormal status of equipment is easily missed when a single monitoring scheme is adopted.

[0009] To achieve the above object, the solution of the present invention includes: A linkage control method based on multimodal data of the present invention comprises the following steps: The data stream including the operation data of the energy equipment and the surrounding environment data obtained in real time is input into the pre-trained machine learning model to determine whether the data stream is abnormal and classify the abnormal results of the data stream; the video stream including the monitoring screen of the defense zone where the energy equipment is located obtained in real time is input into the pre-trained deep learning model to determine whether the video stream is abnormal and classify the abnormal results of the video stream; when the data stream is abnormal but the video stream is normal at the same time, the corresponding energy equipment is controlled according to the data stream abnormal category; when the video stream is abnormal but the data stream is normal at the same time, the corresponding energy equipment is controlled or an alarm signal is generated according to the video stream abnormal category; when the video stream is abnormal and the data stream is abnormal at the same time, an alarm signal is generated and the power supply of the corresponding energy equipment is cut off; the machine learning model used is trained by the operation data and surrounding environment data of the energy equipment that have been marked with the corresponding category that are collected in advance; the deep learning model used is trained by the monitoring screen of the defense zone where the energy equipment is located that has been marked with the corresponding category that is collected in advance.

[0010] Furthermore, after the pre-trained deep learning model detects the target object in the video stream, the kernel correlation filter tracking algorithm is used to track the movement of the target object.

[0011] Furthermore, when comprehensively analyzing the data stream obtained by the pre-trained machine learning model and the video stream obtained by the pre-trained deep learning model, a dynamic time warping algorithm is used to align the data stream and the video stream on the time axis.

[0012] Furthermore, when the power supply of the corresponding energy device is cut off, a video image of a preset time length before and after this time is also recorded.

[0013] Furthermore, the corresponding control strategies are developed using IOT edge services.

[0014] A linkage control system based on multimodal data of the present invention includes a processor, and the processor executes a computer program to implement the steps of the following method: The data stream including the operation data of the energy equipment and the surrounding environment data obtained in real time is input into the pre-trained machine learning model to determine whether the data stream is abnormal and classify the abnormal results of the data stream; the video stream including the monitoring screen of the defense zone where the energy equipment is located obtained in real time is input into the pre-trained deep learning model to determine whether the video stream is abnormal and classify the abnormal results of the video stream; when the data stream is abnormal but the video stream is normal at the same time, the corresponding energy equipment is controlled according to the data stream abnormal category; when the video stream is abnormal but the data stream is normal at the same time, the corresponding energy equipment is controlled or an alarm signal is generated according to the video stream abnormal category; when the video stream is abnormal and the data stream is abnormal at the same time, an alarm signal is generated and the power supply of the corresponding energy equipment is cut off; the machine learning model used is trained by the operation data and surrounding environment data of the energy equipment that have been marked with the corresponding category that are collected in advance; the deep learning model used is trained by the monitoring screen of the defense zone where the energy equipment is located that has been marked with the corresponding category that is collected in advance.

[0015] Furthermore, after the pre-trained deep learning model detects the target object in the video stream, the kernel correlation filter tracking algorithm is used to track the movement of the target object.

[0016] Furthermore, when comprehensively analyzing the data stream obtained by the pre-trained machine learning model and the video stream obtained by the pre-trained deep learning model, a dynamic time warping algorithm is used to align the data stream and the video stream on the time axis.

[0017] Furthermore, when the power supply of the corresponding energy device is cut off, a video image of a preset time length before and after this time is also recorded.

[0018] Furthermore, the corresponding control strategies are developed using IOT edge services.

[0019] The beneficial effects of the present invention are as follows: the present invention is a pioneering invention, which inputs the data stream of the operating data of the energy equipment and the surrounding environment data collected in real time into a pre-trained machine learning model to determine whether the data stream is abnormal and classify the abnormal results of the data stream; the video stream of the monitoring screen of the defense zone where the energy equipment is located obtained in real time is input into a pre-trained deep learning model to determine whether the video stream is abnormal and classify the abnormal results of the video stream. Then, a comprehensive judgment is made on the data stream and the video stream at the same time, and a corresponding control strategy is executed according to the comprehensive judgment result. The present invention makes a comprehensive judgment on whether the equipment is abnormal based on the data stream and the video stream at the same time, which effectively improves the security of the judgment of equipment abnormality. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of a system architecture of the present invention; Figure 2 It is a schematic diagram of a comprehensive analysis method of the present invention; Figure 3 It is a comprehensive judgment logic schematic diagram of the present invention; Figure 4 It is a schematic diagram of a linkage scene control strategy of the present invention; Figure 5 It is a schematic diagram of an emergency linkage control strategy of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0022] The idea of ​​the present invention is to process the real-time operating data of the energy equipment and the surrounding environment data to determine whether an abnormality occurs, process the video data obtained by real-time monitoring of the defense zone where the energy equipment is located to determine whether an abnormality occurs in the defense zone, analyze the corresponding abnormal data, and implement corresponding linkage control strategies.

[0023] Method Example: This embodiment provides a linkage control method based on multi-modal data, such as Figure 1 As shown, the present invention includes an end-level acquisition layer, a data processing module, a video processing module, an AI analysis module, a data service module, an IOT edge service module, a storage module and a network routing module.

[0024] The end-level acquisition layer includes data acquisition devices such as acquisition terminals, sensors, and video terminals. The acquisition terminals and sensors are used to collect the operating data of energy equipment and the surrounding environment data, and provide data stream information to the data processing module in real time in the form of data sources. The video terminal is used to collect the video data of the defense zone where the energy equipment is located, and provide video stream information to the video processing module in real time in the form of video code stream. Among them, the data stream information and the video stream information are in the same period.

[0025] The data processing module is used to process the obtained data flow information of the energy equipment and forward the obtained data flow information to the storage module; the data processing module includes a machine learning model, which is used to process and analyze the data flow, thereby completing the extraction of data features such as time, category, and attribute in the data flow information, and uploading the obtained data flow features to the data service module. The present invention does not limit the machine learning model used, as long as it can ensure that the machine learning model has the ability to continuously learn and adapt as the new data is continuously updated, thereby ensuring the accuracy of the analysis and prediction results.

[0026] The video processing module is used to process the video stream information of the energy equipment, send the obtained video stream information to the storage module, pre-process the video stream information, and then send the processed data to the AI ​​analysis module; the video processing module supports video surveillance devices connected through standard protocols such as GB / T 28181, ONVIF, and RTSP.

[0027] The AI ​​analysis module processes the video frames based on the obtained video stream information through the built-in deep learning model, and uploads the processing results to the data service module; the algorithms in the deep learning model include deep convolutional neural network algorithms, MOOSE, LUCAS-KANADE, RNN, LSTM and other algorithms, so as to complete the behavior analysis and event analysis in the video frames.

[0028] The data service module makes a comprehensive judgment based on the obtained data stream information and video stream information, and links the IOT edge service module according to the judgment results to jointly implement the edge linkage control strategy.

[0029] The IOT edge service module is used to customize the design of scenario linkage rules and emergency linkage strategies, and supports operations such as energy equipment parameter configuration management, online alarm, and generation of operation and maintenance reports. At the same time, it supports the distribution of parameter configuration and linkage strategies to terminal devices.

[0030] The storage module is used to store the collected data stream historical data, video stream historical data, processed data stream data, processed video stream data, recorded video clips, and data after comprehensive processing of data streams and video streams.

[0031] The network routing module is used to provide a secure data encryption channel to ensure the security of energy data. It supports VPN protocols such as L2TP, PPTP, Ipsec, GRE, and multiple tunnels concurrently and supports DDNS and NAT penetration functions.

[0032] The monitoring platform is used to display the data change trends, analysis results and alarm display of data streams and video streams, so that operation and maintenance personnel can quickly understand the system operation status, discover potential problems and take corresponding measures in time.

[0033] Specifically, Figure 2 As shown, the surrounding environment data and operation data of the energy equipment, such as temperature, humidity and power, are collected through sensors, data acquisition terminals, etc., and the collected data are uploaded to the data processing module in the form of data streams; at the same time, the video data of the defense zone where the energy equipment is located is collected through monitoring equipment, video terminals, etc., and the collected video is uploaded to the video processing module in the form of video streams.

[0034] The data processing module uses the built-in machine learning model to analyze and extract features from the obtained data stream to identify anomalies in the data stream, such as excessively high ambient temperature and too low operating power of energy equipment. The analysis results are then sent to the data service module. The training set used by the machine learning model during training is the pre-collected operating data of the energy equipment and the surrounding environment data of the energy equipment.

[0035] The video processing module performs pre-processing operations such as encoding and decoding, video compression, format conversion, image denoising, contrast adjustment, scaling, cropping, and image enhancement on the obtained video stream data, converts the video stream data into a format suitable for deep learning model processing, and then uploads the processed video stream data to the AI ​​analysis module.

[0036] The AI ​​analysis module uses the built-in deep learning model to monitor the target objects in the video stream in real time and classify the video frames to complete the abnormal identification of the video stream data, such as smoke fire, crowd gathering, border intrusion and other abnormal situations. Among them, after the target object is detected, the KCF (Kernelized Correlation Filters) tracking algorithm is also used to track the movement of the target object in the video sequence, and the tracked object is further classified in the AI ​​analysis module and the specific behavior pattern in the video is identified, such as the spread trend of the flame. The analysis results are then sent to the data service module. Among them, the training set used by the deep learning model during training is the pre-collected video images of various situations in the defense zone where the energy equipment is located. The various situations include normal conditions, fire, smoke, crowd gathering, border intrusion, etc., and the corresponding labels are made for each situation before training.

[0037] The data service module makes a comprehensive judgment based on the analysis results sent by the data processing module and the analysis results sent by the video processing module. By comparing historical data, changing trends, setting thresholds, etc., the final judgment result is formed, and different levels of alarm parameters are formed according to the judged abnormal types, such as data abnormality and video abnormality, data abnormality but normal video, and data normal but video abnormality. Then, different alarm parameters are sent to the IOT edge service module; at the same time, the abnormal data, abnormal images, and abnormal results after comprehensive judgment are uploaded to the storage module. When comprehensively processing data stream information and video stream information, although the two data are in the same period, they are not at the same time. Therefore, in the comprehensive judgment process, the DTW (Dynamic Time Warp) algorithm is also used to solve the alignment problem of data stream information and video stream information on the time axis, so as to ensure that data streams and video streams with different flow rates can effectively correspond. By twisting or bending the time series corresponding to the data stream information and video stream information on the time axis, the best matching path between the two time series is found, so as to calculate the similarity between them, and then improve the similarity measurement and comprehensive judgment effect between different data sources during comprehensive judgment.

[0038] The IoT edge service module has linkage control strategies set according to the level of alarm parameters, such as Figure 3 As shown in the figure, when the data is abnormal and the video is abnormal, it indicates that the abnormal situation is caused by the energy equipment itself or the abnormal situation causes the energy equipment to malfunction, for example, the energy equipment is running at high load and catching fire or the surrounding fire causes the energy equipment to run abnormally. At this time, this situation is an emergency event and an emergency linkage operation must be responded to immediately; when the data is abnormal but the video is normal, it indicates that only the energy equipment is abnormal or has an operational failure, for example, the energy equipment is running at high load, but it does not cause emergency situations such as equipment fire. This state is preset as a non-emergency state, and only the scene linkage strategy needs to be started to complete a series of operations such as restarting the energy equipment, adjusting the parameters of the energy equipment, or multi-device linkage control; when the data is normal but the video is abnormal, it indicates that there is an abnormal situation in the defense zone, and the abnormal situation here includes non-emergency events such as personnel intrusion and boundary intrusion, as well as emergency events such as fire and smoke. For non-emergency events, the equipment parameters and operating status of the energy equipment will not be affected in a short period of time, but the long-term existence of this abnormal situation will inevitably have an adverse effect on the energy equipment, so alarm operation and maintenance operations are required. For emergency events, emergency linkage operations must be responded to immediately to avoid abnormal situations causing irreversible consequences to energy equipment; when the data and video are normal, it indicates that the energy equipment is in normal condition and no operation is required.

[0039] Specifically, the scenario linkage strategy preset in the IOT edge service module is as follows: Figure 4As shown, when the data service module detects that the energy equipment is abnormal, according to the level of the corresponding alarm parameter, the linkage control strategy is started to adjust the parameters of the energy equipment, for example, restart the energy equipment, or adjust the operating frequency of the inverter equipment under the premise of meeting the frequency stability of the power system and system safety, and monitor in real time whether the energy equipment can operate stably within the set range. In addition to restarting the energy equipment or adjusting the parameters of the energy equipment, the linkage strategy between multiple devices can also be implemented. At this time, the temperature sensor detects that the ambient temperature of the energy equipment is higher or lower than the preset temperature threshold, calculates the difference with the temperature threshold, and performs closed-loop control, so as to link the fan to adjust the fan speed, thereby reducing or increasing the ambient temperature of the energy equipment. When the ambient temperature is close to the preset temperature threshold, the fan speed will remain stable. When the corresponding alarm parameters are generated, the IOT edge service module generates an operation and maintenance report with the abnormal data, parameter adjustment or linkage results, uploads it to the monitoring platform through the network routing module, and reports it to the operation and maintenance personnel.

[0040] Specifically, the emergency linkage strategy preset in the IOT edge service module is as follows: Figure 5 As shown in the figure, when the data service module detects an abnormality in the video screen, the emergency linkage strategy is activated according to the level of the corresponding alarm parameter. For example, when monitoring non-emergency situations such as human intrusion and foreign object obstruction, the monitoring platform is reported and an alarm is issued in real time. When emergency situations such as smoke and fire are detected, emergency alarm parameters are immediately generated, and remote control commands are quickly issued to control the power supply of energy equipment to disconnect to prevent the spread of fire; and the spherical network camera is linked to implement 30s video screen recording before and after the emergency occurs and horizontal 270° cruise function. At the same time, the historical data cached by the storage module is called to perform real-time fault diagnosis on energy equipment and surrounding environment, quickly locate the source of the fault, and form an emergency alarm report with the recorded video screen, emergency alarm parameters and fault causes, which is uploaded to the monitoring platform through the network routing module and reported to the operation and maintenance personnel to timely discover and solve emergencies. For example, when a fire is detected, it is also monitored that the energy equipment is running at high load or the surrounding environment temperature is too high. At this time, the operation and maintenance personnel will know that the fire is caused by high load operation or high surrounding environment temperature, which is convenient for the operation and maintenance personnel to deal with the situation.

[0041] System Example: This embodiment provides a linkage control system based on multimodal data. When monitoring energy equipment, the processor of the system adopts a linkage control method based on multimodal data as introduced in the method embodiment. Since the introduction of this method is clear enough, it will not be repeated here.

[0042] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A linkage control method based on multimodal data, characterized in that: The steps include: The data stream including the operation data of the energy equipment and the surrounding environment data obtained in real time is input into the pre-trained machine learning model to determine whether the data stream is abnormal and classify the abnormal results of the data stream; the video stream including the monitoring screen of the defense zone where the energy equipment is located obtained in real time is input into the pre-trained deep learning model to determine whether the video stream is abnormal and classify the abnormal results of the video stream; when the data stream is abnormal but the video stream is normal at the same time, the corresponding energy equipment is controlled according to the data stream abnormal category; when the video stream is abnormal but the data stream is normal at the same time, the corresponding energy equipment is controlled or an alarm signal is generated according to the video stream abnormal category; when the video stream is abnormal and the data stream is abnormal at the same time, an alarm signal is generated and the power supply of the corresponding energy equipment is cut off; the machine learning model used is trained by the operation data and surrounding environment data of the energy equipment that have been marked with the corresponding category that are collected in advance; the deep learning model used is trained by the monitoring screen of the defense zone where the energy equipment is located that has been marked with the corresponding category that is collected in advance.

2. The linkage control method based on multimodal data according to claim 1, characterized in that: After the pre-trained deep learning model detects the target object in the video stream, the movement of the target object is tracked using a kernel correlation filter tracking algorithm.

3. The linkage control method based on multimodal data according to claim 1, characterized in that: When comprehensively analyzing the data stream obtained by the pre-trained machine learning model and the video stream obtained by the pre-trained deep learning model, a dynamic time warping algorithm is used to align the data stream and the video stream on the time axis.

4. The linkage control method based on multimodal data according to claim 1, characterized in that: While cutting off the power supply of the corresponding energy equipment, video footage of a preset time length before and after this time is also recorded.

5. The linkage control method based on multimodal data according to claim 1, characterized in that: Use IOT edge services to develop corresponding control strategies.

6. A linkage control system based on multimodal data, comprising a processor, characterized in that: The processor executes the computer program to implement the steps of the method comprising: The data stream including the operation data of the energy equipment and the surrounding environment data obtained in real time is input into the pre-trained machine learning model to determine whether the data stream is abnormal and classify the abnormal results of the data stream; the video stream including the monitoring screen of the defense zone where the energy equipment is located obtained in real time is input into the pre-trained deep learning model to determine whether the video stream is abnormal and classify the abnormal results of the video stream; when the data stream is abnormal but the video stream is normal at the same time, the corresponding energy equipment is controlled according to the data stream abnormal category; when the video stream is abnormal but the data stream is normal at the same time, the corresponding energy equipment is controlled or an alarm signal is generated according to the video stream abnormal category; when the video stream is abnormal and the data stream is abnormal at the same time, an alarm signal is generated and the power supply of the corresponding energy equipment is cut off; the machine learning model used is trained by the operation data and surrounding environment data of the energy equipment that have been marked with the corresponding category that are collected in advance; the deep learning model used is trained by the monitoring screen of the defense zone where the energy equipment is located that has been marked with the corresponding category that is collected in advance.

7. The linkage control system based on multimodal data according to claim 6, characterized in that: After the pre-trained deep learning model detects the target object in the video stream, the movement of the target object is tracked using a kernel correlation filter tracking algorithm.

8. The linkage control system based on multimodal data according to claim 6, characterized in that: When comprehensively analyzing the data stream obtained by the pre-trained machine learning model and the video stream obtained by the pre-trained deep learning model, a dynamic time warping algorithm is used to align the data stream and the video stream on the time axis.

9. The linkage control system based on multimodal data according to claim 6, characterized in that: While cutting off the power supply of the corresponding energy equipment, video footage of a preset time length before and after this time is also recorded.

10. The linkage control system based on multimodal data according to claim 6, characterized in that: Use IOT edge services to develop corresponding control strategies.

Images