A mechanical and electrical intelligent monitoring system and monitoring method thereof

Abstract

The present invention relates to the field of electromechanical monitoring technology, specifically to an electromechanical intelligent monitoring system and a monitoring method thereof, including an anomaly detection unit, a video positioning unit, a monitoring interface unit, a status monitoring unit, a status detection unit and an intelligent alarm unit, and also including: a video monitoring unit, the video monitoring unit is used to perform video monitoring of a target area where electromechanical equipment is located based on a high-definition camera to obtain a visible light video of the target area, and to perform video monitoring of the target area where the electromechanical equipment is located based on an infrared camera to obtain an infrared light video of the target area. By combining visible light and infrared monitoring, the present invention enables the system to obtain the status of electromechanical equipment in real time under different lighting conditions, thereby improving the comprehensiveness and reliability of monitoring, and utilizing a deep learning model for key frame selection and anomaly detection, so that the system can accurately identify potential abnormal situations and improve the timeliness of fault warning.

Description

Technical Field

[0001] The present invention relates to the field of electromechanical monitoring technology, and in particular to an electromechanical intelligent monitoring system and a monitoring method thereof. Background Art

[0002] Mechatronics is a discipline that combines mechanical and electronic technologies, and usually refers to the intersection of mechanical engineering and electronic engineering. It includes the design and manufacture of mechanical equipment and the development and application of electronic control systems. Mechatronics systems are widely used in automation, robotics, instrumentation, intelligent devices and other fields to improve production efficiency, precision and reliability.

[0003] Traditional systems usually rely on a single type of camera, which has poor effects in low light or bad weather conditions and is difficult to monitor comprehensively. Traditional systems rely on manual observation or timed recording, and cannot achieve real-time monitoring and immediate feedback, resulting in slow response to abnormal situations. Traditional systems are mostly passive monitoring, lacking intelligent data analysis and anomaly detection capabilities, and often can only rely on manual judgment, which can easily miss potential problems. Moreover, due to the lack of automated monitoring and alarm functions, equipment failures often require manual inspections, which increases maintenance costs and human resource requirements. Traditional systems are often unable to effectively integrate and analyze multiple data, and the information island phenomenon is serious, making it difficult to provide a comprehensive status assessment. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the disadvantages of the above-mentioned prior art and provide an electromechanical intelligent monitoring system and a monitoring method thereof.

[0005] The technical solution adopted to solve the above technical problems is: an electromechanical intelligent monitoring system, including an abnormality detection unit, a video positioning unit, a monitoring interface unit, a state monitoring unit, a state detection unit and an intelligent alarm unit, and also includes:

[0006] A video monitoring unit, the video monitoring unit is used to perform video monitoring of the target area where the electromechanical equipment is located based on a high-definition camera to obtain a visible light video of the target area, perform video monitoring of the target area where the electromechanical equipment is located based on an infrared camera to obtain an infrared light video of the target area, and transmit the visible light video and the infrared light video of the target area to the image fusion unit and the video positioning unit;

[0007] An image fusion unit, which receives the visible light video and the infrared light video of the target area transmitted by the video monitoring unit, and synchronously captures the visible light video and the infrared light video of the target area according to a preset sampling period to obtain a visible light video frame sequence and an infrared light video frame sequence, performs video frame fusion on the visible light video frame sequence and the infrared light video frame sequence to obtain a fused video frame sequence, and transmits the fused video frame sequence to a key frame selection unit;

[0008] A key frame selection unit receives the fused video frame sequence transmitted by the image fusion unit, and extracts feature points for each fused video frame in the fused video frame sequence based on a trained feature point extraction network, extracts descriptors for each fused video frame in the fused video frame sequence based on a trained descriptor extraction network, selects key frames for the fused video frame sequence based on a feature point set and a descriptor set of each fused video frame in the fused video frame sequence to obtain a key frame set corresponding to the fused video frame sequence, and transmits the key frame set corresponding to the fused video frame sequence to an anomaly detection unit.

[0009] Preferably, the anomaly detection unit receives the key frame set corresponding to the fused video frame sequence transmitted by the key frame selection unit, and performs anomaly detection on the key frame set based on a trained anomaly detection model to obtain an abnormal key frame set corresponding to the key frame set, and transmits the abnormal key frame set to the video positioning unit.

[0010] Preferably, the video positioning unit receives the visible light video and infrared light video of the target area transmitted by the video monitoring unit and the abnormal key frame set transmitted by the abnormality detection unit, and positions the visible light video and infrared light video of the target area based on the time series corresponding to the abnormal key frame set to obtain the visible light sub-video and infrared photon video with abnormalities, and transmits the visible light sub-video and infrared photon video with abnormalities to the monitoring interface unit.

[0011] Preferably, the state monitoring unit is used to monitor the state of the electromechanical equipment based on a sensor group to obtain state data of the electromechanical equipment, wherein the state data includes vibration data and temperature data, and transmit the state data of the electromechanical equipment to the state detection unit.

[0012] Preferably, the state detection unit receives the state data of the electromechanical equipment transmitted by the state monitoring unit, and compares the temperature data in the state data of the electromechanical equipment with a preset temperature threshold. If the temperature is greater than the preset temperature threshold, an alarm signal is generated, and feature extraction is performed on the vibration data in the state data of the electromechanical equipment to obtain a feature set corresponding to the vibration data, and state detection is performed on the feature set based on a trained state detection model to obtain a state label corresponding to the electromechanical equipment. The state labels corresponding to the electromechanical equipment at different times are obtained based on the time series corresponding to the state data of the electromechanical equipment, and the state labels corresponding to the electromechanical equipment at different times are transmitted to the monitoring interface unit, and the alarm signal is transmitted to the intelligent alarm unit.

[0013] Preferably, the monitoring interface unit receives the abnormal visible light sub-video and infrared photon video transmitted by the video positioning unit and the status tags corresponding to the electromechanical equipment at different times transmitted by the status detection unit, and displays the abnormal visible light sub-video and infrared photon video, and adds the status tags corresponding to the electromechanical equipment at different times below the abnormal visible light sub-video and infrared photon video.

[0014] Preferably, the intelligent alarm unit receives the alarm signal transmitted by the status detection unit, and issues an abnormal alarm based on the alarm signal based on the intelligent alarm device.

[0015] Preferably, performing video frame fusion on the visible light video frame sequence and the infrared light video frame sequence to obtain a fused video frame sequence includes:

[0016] Decomposing the visible light video frame sequence and the infrared light video frame sequence into a visible light video frame base layer sequence, a visible light video frame detail layer sequence, an infrared light video frame base layer sequence and an infrared light video frame detail layer sequence based on a filter;

[0017] Calculating the covariance matrix of the visible light video frame base layer sequence and the infrared light video frame base layer sequence to obtain the covariance matrix corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence;

[0018] Calculating eigenvectors of covariance matrices corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence to obtain eigenmatrices corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence;

[0019] Based on the feature matrices corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence, acquiring image features corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence;

[0020] Calculate the weights of the visible light video frame and the infrared light video frame based on the image features corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence;

[0021] Based on the weights of the visible light video frame and the infrared light video frame, the base layer sequence of the visible light video frame and the base layer sequence of the infrared light video frame are fused to obtain a fused feature map, and based on the transposition of the fused feature map and the feature matrix, a fused base layer sequence is obtained;

[0022] The fused base layer sequence is fused with the visible light video frame detail layer sequence to obtain a fused video frame sequence.

[0023] Preferably, the expressions of the visible light video frame base layer and the infrared light video frame base layer are as follows:

[0024] ;

[0025] in, and are the expressions for the base layer of the visible light video frame and the base layer of the infrared light video frame, respectively. represents the filter function, represents the filter radius, represents the regularization parameter, and Represent visible light video frames and infrared light video frames respectively;

[0026] The expressions of the visible light video frame base layer and the infrared light video frame detail layer are as follows:

[0027] ;

[0028] in, and Expressions representing the visible light video frame detail layer and the infrared light video frame detail layer respectively;

[0029] The calculation formula of the covariance matrix of the visible light video frame base layer sequence and the infrared light video frame base layer sequence is as follows:

[0030] ;

[0031] in, and They represent the covariance matrices of the base layer sequence of visible light video frames and the base layer sequence of infrared light video frames, respectively. represents the total number of video frames in the base layer sequence, Indicates the first visible light video frames, Indicates the first infrared video frames, represents the average value of the visible light video frame, Represents the average value of infrared light video frames;

[0032] The expressions of the image features corresponding to the base layer sequence of the visible light video frame and the base layer sequence of the infrared light video frame are as follows:

[0033] ;

[0034] in, and They represent the image features corresponding to the base layer sequence of visible light video frames and the base layer sequence of infrared light video frames, respectively. represents the feature matrix;

[0035] The calculation formula of the weights of the visible light video frame and the infrared light video frame is as follows:

[0036] ;

[0037] in, and Respectively represent the weights of the visible light video frame and the infrared light video frame, represents the size of the feature matrix, represents the average value of image features of the base layer sequence of visible light video frames, Represents the average value of image features of the base layer sequence of infrared video frames;

[0038] The expression of the fused feature map is as follows:

[0039] ;

[0040] in, Represents the fused feature map;

[0041] The expression of the fused base layer is as follows:

[0042] ;

[0043] in, represents the fusion base layer;

[0044] The expression of the fused video frame is as follows:

[0045] ;

[0046] in, Represents a fused video frame.

[0047] Preferably, performing key frame selection on the fused video frame sequence based on a feature point set and a descriptor set of each fused video frame in the fused video frame sequence to obtain a key frame set corresponding to the fused video frame sequence includes:

[0048] Traversing the fused video frame sequence, and sequentially acquiring a feature point set and a descriptor set of each fused video frame in the fused video frame sequence;

[0049] Matching the feature point set based on the descriptor set to obtain a key point set for each fused video frame in the fused video frame sequence;

[0050] The distance between the key point sets of each fused video frame in the fused video frame sequence is calculated in sequence, and the distance is compared with a preset distance threshold. If the distance is greater than the preset distance threshold, the corresponding fused video frame is added to the key frame set. The calculation formula of the distance is as follows:

[0051] ;

[0052] in, represents the distance between two consecutive fused video frames, Represents the total number of key points in the key point set, Indicates the first The first key point in the set of key points in the fused video frame A key point.

[0053] Preferably, the anomaly detection model includes a feature extraction module, four cross-attention modules and a detection module, wherein the feature extraction module is used to perform feature extraction on the key frame to obtain key frame features, the cross-attention module is used to perform feature denoising on the key frame features to obtain key frame denoising fusion features, and the four cross-attention modules are connected in sequence, and the detection module is used to perform feature detection on the key frame denoising features. The expression of the cross-attention module is as follows:

[0054] ;

[0055] in, represents the key frame denoising fusion feature, , and Represents 3 different convolutional layers, represents the self-attention layer, represents the keyframe denoising feature, and , represents the feature weight, represents a matrix transformation operation, represents the key frame features, , Represents the transformation function of the feature.

[0056] Preferably, the state detection model adopts a support vector machine model.

[0057] The technical solution adopted to solve the above technical problems is: an electromechanical intelligent monitoring method, which is applicable to the electromechanical intelligent monitoring system, comprising:

[0058] Use high-definition cameras and infrared cameras to conduct real-time video monitoring of the target area where the electromechanical equipment is located, and obtain visible light video and infrared light video;

[0059] Synchronous video frame interception is performed according to a preset sampling period, and the intercepted video frames are fused to generate a fused video frame sequence;

[0060] Extract feature points and descriptors using the trained feature point extraction network and descriptor extraction network, and select a key frame set based on the extracted feature points and descriptors;

[0061] Use the trained anomaly detection model to perform anomaly detection to identify the abnormal key frame set, and combine the visible light video, infrared video and the abnormal key frame set to locate the video clips with abnormalities;

[0062] Monitor the status of electromechanical equipment through the sensor group, obtain vibration data and temperature data, compare the temperature with the preset threshold, generate an alarm signal if the threshold is exceeded, extract vibration data features and perform status detection based on the status detection model to generate a status label;

[0063] Abnormal video clips and electromechanical equipment status labels are displayed visually, and the status labels are attached under the corresponding videos. Based on the alarm signal, the intelligent alarm device is triggered to sound an alarm.

[0064] The beneficial effects of the present invention are as follows: (1) By combining visible light and infrared monitoring, the system can obtain the status of electromechanical equipment in real time under different lighting conditions, thereby improving the comprehensiveness and reliability of monitoring. In addition, the deep learning model is used for key frame selection and anomaly detection, so that the system can accurately identify potential abnormal situations and improve the timeliness of fault warning. (2) The present invention monitors the vibration, temperature and other status data of the equipment through the sensor group, which can effectively detect the operation abnormalities of the equipment and provide real-time feedback, which is helpful for maintenance and management. It can also automatically generate alarm signals and notify maintenance personnel in time through the intelligent alarm unit, reducing the workload of manual monitoring and improving the response speed. (3) The present invention can intuitively display abnormal videos and equipment status through the monitoring interface unit, which is convenient for operators to quickly judge and make decisions, improving the user experience of monitoring, and can record the time series data of the equipment status to provide support for subsequent data analysis and troubleshooting, thereby helping enterprises to achieve more efficient management. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of the system architecture of the overall system in an embodiment of the present invention;

[0066] Figure 2 A schematic flow chart of the steps of an overall method in an embodiment of the present invention.

[0067] Figure numerals: 1. Video monitoring unit; 2. Image fusion unit; 3. Key frame selection unit; 4. Anomaly detection unit; 5. Video positioning unit; 6. Monitoring interface unit; 7. Status monitoring unit; 8. Status detection unit; 9. Intelligent alarm unit. DETAILED DESCRIPTION

[0068] Embodiment 1, as Figure 1 As shown, the electromechanical intelligent monitoring system proposed by the present invention includes an abnormality detection unit 4, a video positioning unit 5, a monitoring interface unit 6, a state monitoring unit 7, a state detection unit 8 and an intelligent alarm unit 9, and also includes:

[0069] The video monitoring unit 1 is used to perform video monitoring on the target area where the electromechanical equipment is located based on a high-definition camera to obtain a visible light video of the target area, and to perform video monitoring on the target area where the electromechanical equipment is located based on an infrared camera to obtain an infrared light video of the target area, and transmit the visible light video and the infrared light video of the target area to the image fusion unit 2 and the video positioning unit 5;

[0070] The image fusion unit 2 receives the visible light video and the infrared light video of the target area transmitted by the video monitoring unit 1, and synchronously captures the visible light video and the infrared light video of the target area according to a preset sampling period to obtain a visible light video frame sequence and an infrared light video frame sequence, performs video frame fusion on the visible light video frame sequence and the infrared light video frame sequence to obtain a fused video frame sequence, and transmits the fused video frame sequence to the key frame selection unit 3;

[0071] The key frame selection unit 3 receives the fused video frame sequence transmitted by the image fusion unit 2, and extracts feature points for each fused video frame in the fused video frame sequence based on the trained feature point extraction network, extracts descriptors for each fused video frame in the fused video frame sequence based on the trained descriptor extraction network, selects key frames for the fused video frame sequence based on the feature point set and descriptor set of each fused video frame in the fused video frame sequence to obtain a key frame set corresponding to the fused video frame sequence, and transmits the key frame set corresponding to the fused video frame sequence to the anomaly detection unit 4.

[0072] In the present invention, the high-definition camera is a camera that can capture high-resolution images, and is usually used to obtain clear visible light videos, and is suitable for monitoring scenes with high detail requirements; the infrared camera is mainly used to capture infrared light images, and can work in low-light or dark environments, and is usually used to detect temperature changes or the heat of objects; the feature point extraction network refers to a deep learning model that is specifically used to identify and extract representative points from images, which are usually significant features in the image; the descriptor extraction network refers to a deep learning model that is used to generate descriptive information for the extracted feature points so that these feature points can be effectively compared and matched.

[0073] In an optional embodiment, the anomaly detection unit 4 receives the key frame set corresponding to the fused video frame sequence transmitted by the key frame selection unit 3, and performs anomaly detection on the key frame set based on the trained anomaly detection model to obtain an abnormal key frame set corresponding to the key frame set, and transmits the abnormal key frame set to the video positioning unit 5.

[0074] In an optional embodiment, the video positioning unit 5 receives the visible light video and infrared light video of the target area transmitted by the video monitoring unit 1 and the abnormal key frame set transmitted by the abnormality detection unit 4, and locates the visible light video and infrared light video of the target area based on the time series corresponding to the abnormal key frame set to obtain the visible light sub-video and infrared photon video with abnormalities, and transmits the visible light sub-video and infrared photon video with abnormalities to the monitoring interface unit 6.

[0075] It should be noted that the visible light video and infrared light video of the target area are located based on the time series corresponding to the set of abnormal key frames to obtain the visible light sub-video and infrared photon video with abnormalities, specifically through the following steps: the extracted abnormal key frames are associated with their corresponding timestamps to form an abnormal time series, which makes it possible to clearly know the moments when the abnormalities occurred; based on the abnormal time series, video clips of the corresponding time period are extracted from the original visible light and infrared videos, and these clips contain the specific scenes where the abnormalities occurred; the extracted visible light video clips can be fused with the infrared video clips to generate a comprehensive video, so as to observe the abnormal situation more comprehensively.

[0076] In an optional embodiment, the state monitoring unit 7 is used to monitor the state of the electromechanical equipment based on the sensor group to obtain state data of the electromechanical equipment, wherein the state data includes vibration data and temperature data, and transmit the state data of the electromechanical equipment to the state detection unit 8.

[0077] It should be noted that the sensor group includes a vibration sensor and a temperature sensor.

[0078] In an optional embodiment, the state detection unit 8 receives the state data of the electromechanical equipment transmitted by the state monitoring unit 7, and compares the temperature data in the state data of the electromechanical equipment with a preset temperature threshold. If the temperature is greater than the preset temperature threshold, an alarm signal is generated, and feature extraction is performed on the vibration data in the state data of the electromechanical equipment to obtain a feature set corresponding to the vibration data. The feature set is subjected to state detection based on a trained state detection model to obtain a state label corresponding to the electromechanical equipment. The state labels corresponding to the electromechanical equipment at different times are obtained based on the time series corresponding to the state data of the electromechanical equipment, and the state labels corresponding to the electromechanical equipment at different times are transmitted to the monitoring interface unit 6, and the alarm signal is transmitted to the intelligent alarm unit 9.

[0079] In an optional embodiment, the monitoring interface unit 6 receives the abnormal visible light sub-video and infrared photon video transmitted by the video positioning unit 5 and the status labels corresponding to the electromechanical equipment at different times transmitted by the status detection unit 8, and displays the abnormal visible light sub-video and infrared photon video, and adds the status labels corresponding to the electromechanical equipment at different times below the abnormal visible light sub-video and infrared photon video.

[0080] In an optional embodiment, the intelligent alarm unit 9 receives the alarm signal transmitted by the state detection unit 8, and issues an abnormal alarm based on the alarm signal based on the intelligent alarm device.

[0081] It should be noted that intelligent alarm equipment is a system that uses modern sensing technology and intelligent algorithms to perform abnormal monitoring and alarm. They are usually composed of sensors, control units and alarm outputs. They can detect the environment or equipment status in real time and issue alarms when abnormalities occur.

[0082] Embodiment 2, a mechatronic intelligent monitoring system proposed by the present invention, compared with embodiment 1, this embodiment further includes: fusing a visible light video frame sequence with an infrared light video frame sequence to obtain a fused video frame sequence, including:

[0083] A1. Decomposing the visible light video frame sequence and the infrared light video frame sequence into a visible light video frame base layer sequence, a visible light video frame detail layer sequence, an infrared light video frame base layer sequence and an infrared light video frame detail layer sequence based on a filter;

[0084] A2. Calculating the covariance matrix of the visible light video frame base layer sequence and the infrared light video frame base layer sequence to obtain the covariance matrix corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence;

[0085] A3. Calculate the eigenvectors of the covariance matrices corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence to obtain the eigenmatrices corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence;

[0086] A4. Based on the feature matrices corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence, obtain image features corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence;

[0087] A5. Calculate the weights of the visible light video frame and the infrared light video frame based on the image features corresponding to the base layer sequence of the visible light video frame and the base layer sequence of the infrared light video frame;

[0088] A6. Based on the weights of the visible light video frame and the infrared light video frame, a base layer sequence of the visible light video frame and a base layer sequence of the infrared light video frame are fused to obtain a fused feature map, and a fused base layer sequence is obtained based on the transposition of the fused feature map and the feature matrix;

[0089] A7. Fusing the fused base layer sequence with the visible light video frame detail layer sequence to obtain a fused video frame sequence.

[0090] In an optional embodiment, the expressions of the visible light video frame base layer and the infrared light video frame base layer are as follows:

[0091] ;

[0092] in, and are the expressions for the base layer of the visible light video frame and the base layer of the infrared light video frame, respectively. represents the filter function, represents the filter radius, represents the regularization parameter, and Represent visible light video frames and infrared light video frames respectively;

[0093] The expressions of the base layer of the visible light video frame and the detail layer of the infrared light video frame are as follows:

[0094] ;

[0095] in, and Expressions representing the visible light video frame detail layer and the infrared light video frame detail layer respectively;

[0096] The calculation formula of the covariance matrix of the base layer sequence of the visible light video frame and the base layer sequence of the infrared light video frame is as follows:

[0097] ;

[0098] in, and They represent the covariance matrices of the base layer sequence of visible light video frames and the base layer sequence of infrared light video frames, respectively. represents the total number of video frames in the base layer sequence, Indicates the first visible light video frames, Indicates the first infrared video frames, represents the average value of the visible light video frame, Represents the average value of infrared light video frames;

[0099] The expressions of the image features corresponding to the base layer sequence of visible light video frames and the base layer sequence of infrared light video frames are as follows:

[0100] ;

[0101] in, and They represent the image features corresponding to the base layer sequence of visible light video frames and the base layer sequence of infrared light video frames, respectively. represents the feature matrix;

[0102] The calculation formula for the weights of visible light video frames and infrared light video frames is as follows:

[0103] ;

[0104] in, and Represent the weights of visible light video frames and infrared light video frames respectively, represents the size of the feature matrix, represents the average value of image features of the base layer sequence of visible light video frames, Represents the average value of image features of the base layer sequence of infrared video frames;

[0105] The expression of the fused feature map is as follows:

[0106] ;

[0107] in, Represents the fused feature map;

[0108] The expression of the fused base layer is as follows:

[0109] ;

[0110] in, represents the fusion base layer;

[0111] The expression of fused video frame is as follows:

[0112] ;

[0113] in, Represents a fused video frame.

[0114] In an optional embodiment, key frame selection is performed on the fused video frame sequence based on the feature point set and the descriptor set of each fused video frame in the fused video frame sequence to obtain a key frame set corresponding to the fused video frame sequence, including:

[0115] B1. Traverse the fused video frame sequence, and sequentially obtain the feature point set and descriptor set of each fused video frame in the fused video frame sequence;

[0116] B2. matching the feature point set based on the descriptor set to obtain a key point set of each fused video frame in the fused video frame sequence;

[0117] B3. Calculate the distance between the key point sets of each fused video frame in the fused video frame sequence in turn, and compare the distance with the preset distance threshold. If the distance is greater than the preset distance threshold, add the corresponding fused video frame to the key frame set. The distance calculation formula is as follows:

[0118] ;

[0119] in, represents the distance between two consecutive fused video frames, Represents the total number of key points in the key point set, Indicates the first The first key point in the set of key points in the fused video frame A key point.

[0120] In an optional embodiment, the anomaly detection model includes a feature extraction module, four cross-attention modules and a detection module, the feature extraction module is used to extract features from key frames to obtain key frame features, the cross-attention module is used to perform feature denoising on key frame features to obtain key frame denoising fusion features, and the four cross-attention modules are connected in sequence, the detection module is used to perform feature detection on the key frame denoising features, and the expression of the cross-attention module is as follows:

[0121] ;

[0122] in, represents the key frame denoising fusion feature, , and Represents 3 different convolutional layers, represents the self-attention layer, represents the keyframe denoising feature, and , represents the feature weight, represents a matrix transformation operation, represents the key frame features, , Represents the transformation function of the feature.

[0123] In an optional embodiment, the state detection model adopts a support vector machine model.

[0124] It should be noted that Support Vector Machine (SVM) is a supervised learning algorithm mainly used for classification and regression analysis. Its core idea is to find an optimal hyperplane to maximize the interval between different categories, thereby achieving effective classification of data.

[0125] Embodiment three, as Figure 2 As shown, the present invention proposes an electromechanical intelligent monitoring method, which is applicable to the electromechanical intelligent monitoring system, comprising:

[0126] S1. Use high-definition cameras and infrared cameras to conduct real-time video monitoring of the target area where the electromechanical equipment is located, and obtain visible light video and infrared light video;

[0127] S2. Capture synchronous video frames according to a preset sampling period, fuse the captured video frames, and generate a fused video frame sequence;

[0128] S3, extracting feature points and descriptors using the trained feature point extraction network and descriptor extraction network, and selecting a key frame set based on the extracted feature points and descriptors;

[0129] S4. Perform anomaly detection using the trained anomaly detection model to identify anomaly key frame sets, and locate the video clips with anomalies by combining the visible light video, infrared video, and anomaly key frame sets;

[0130] S5. Monitor the status of the electromechanical equipment through the sensor group, obtain vibration data and temperature data, compare the temperature with a preset threshold, and generate an alarm signal if the threshold is exceeded, extract vibration data features, perform status detection based on a status detection model, and generate a status label;

[0131] S6. Visually display the abnormal video clips and the status labels of the electromechanical equipment, attach the status labels under the corresponding videos, and trigger the intelligent alarm device to sound an alarm based on the alarm signal.

[0132] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited thereto, and various changes can be made within the knowledge scope of technicians in the relevant technical field without departing from the purpose of the present invention.

Claims

1. An electromechanical intelligent monitoring system, comprising an abnormality detection unit (4), a video positioning unit (5), a monitoring interface unit (6), a state monitoring unit (7), a state detection unit (8) and an intelligent alarm unit (9), characterized in that: A video monitoring unit (1), the video monitoring unit (1) being used to perform video monitoring of a target area where the electromechanical equipment is located based on a high-definition camera to obtain a visible light video of the target area, perform video monitoring of the target area where the electromechanical equipment is located based on an infrared camera to obtain an infrared light video of the target area, and transmit the visible light video and the infrared light video of the target area to an image fusion unit (2) and a video positioning unit (5); An image fusion unit (2), the image fusion unit (2) receiving the visible light video and infrared light video of the target area transmitted by the video monitoring unit (1), and synchronously capturing the visible light video and infrared light video of the target area according to a preset sampling period to obtain a visible light video frame sequence and an infrared light video frame sequence, performing video frame fusion on the visible light video frame sequence and the infrared light video frame sequence to obtain a fused video frame sequence, and transmitting the fused video frame sequence to a key frame selection unit (3); A key frame selection unit (3), the key frame selection unit (3) receives the fused video frame sequence transmitted by the image fusion unit (2), extracts feature points from each fused video frame in the fused video frame sequence based on a trained feature point extraction network, extracts descriptors from each fused video frame in the fused video frame sequence based on a trained descriptor extraction network, selects key frames from the fused video frame sequence based on a feature point set and a descriptor set of each fused video frame in the fused video frame sequence, so as to obtain a key frame set corresponding to the fused video frame sequence, and transmits the key frame set corresponding to the fused video frame sequence to an anomaly detection unit (4); The method further comprises: performing key frame selection on the fused video frame sequence based on a feature point set and a descriptor set of each fused video frame in the fused video frame sequence to obtain a key frame set corresponding to the fused video frame sequence, including: Traversing the fused video frame sequence, and sequentially acquiring a feature point set and a descriptor set of each fused video frame in the fused video frame sequence; Matching the feature point set based on the descriptor set to obtain a key point set for each fused video frame in the fused video frame sequence; The distance between the key point sets of each fused video frame in the fused video frame sequence is calculated in sequence, and the distance is compared with a preset distance threshold. If the distance is greater than the preset distance threshold, the corresponding fused video frame is added to the key frame set. The calculation formula of the distance is as follows: ； in, represents the distance between two consecutive fused video frames, Represents the total number of key points in the key point set, Indicates the first The first key point in the set of key points in the fused video frame A key point.

2. The electromechanical intelligent monitoring system according to claim 1, characterized in that: The anomaly detection unit (4) receives the key frame set corresponding to the fused video frame sequence transmitted by the key frame selection unit (3), performs anomaly detection on the key frame set based on the trained anomaly detection model to obtain an abnormal key frame set corresponding to the key frame set, and transmits the abnormal key frame set to the video positioning unit (5); The video positioning unit (5) receives the visible light video and infrared light video of the target area transmitted by the video monitoring unit (1) and the abnormal key frame set transmitted by the abnormality detection unit (4), and positions the visible light video and infrared light video of the target area based on the time sequence corresponding to the abnormal key frame set to obtain the visible light sub-video and infrared photon video with abnormalities, and transmits the visible light sub-video and infrared photon video with abnormalities to the monitoring interface unit (6).

3. The electromechanical intelligent monitoring system according to claim 2, characterized in that: The state monitoring unit (7) is used to monitor the state of the electromechanical device based on the sensor group to obtain state data of the electromechanical device, wherein the state data includes vibration data and temperature data, and transmit the state data of the electromechanical device to the state detection unit (8); The state detection unit (8) receives the state data of the electromechanical device transmitted by the state monitoring unit (7), and compares the temperature data in the state data of the electromechanical device with a preset temperature threshold. If the temperature is greater than the preset temperature threshold, an alarm signal is generated, and feature extraction is performed on the vibration data in the state data of the electromechanical device to obtain a feature set corresponding to the vibration data. The state detection is performed on the feature set based on a trained state detection model to obtain a state label corresponding to the electromechanical device. The state labels corresponding to the electromechanical device at different times are obtained based on the time series corresponding to the state data of the electromechanical device. The state labels corresponding to the electromechanical device at different times are transmitted to the monitoring interface unit (6), and the alarm signal is transmitted to the intelligent alarm unit (9).

4. The electromechanical intelligent monitoring system according to claim 3, characterized in that: The monitoring interface unit (6) receives the abnormal visible light sub-video and infrared photon video transmitted by the video positioning unit (5) and the status tags corresponding to the electromechanical equipment at different times transmitted by the status detection unit (8), displays the abnormal visible light sub-video and infrared photon video, and adds the status tags corresponding to the electromechanical equipment at different times below the abnormal visible light sub-video and infrared photon video; The intelligent alarm unit (9) receives the alarm signal transmitted by the state detection unit (8), and issues an abnormal alarm based on the alarm signal based on the intelligent alarm device.

5. The electromechanical intelligent monitoring system according to claim 1, characterized in that: Performing video frame fusion on the visible light video frame sequence and the infrared light video frame sequence to obtain a fused video frame sequence, including: Decomposing the visible light video frame sequence and the infrared light video frame sequence into a visible light video frame base layer sequence, a visible light video frame detail layer sequence, an infrared light video frame base layer sequence and an infrared light video frame detail layer sequence based on a filter; Calculating the covariance matrix of the visible light video frame base layer sequence and the infrared light video frame base layer sequence to obtain the covariance matrix corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence; Calculating eigenvectors of covariance matrices corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence to obtain eigenmatrices corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence; Based on the feature matrices corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence, acquiring image features corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence; Calculate the weights of the visible light video frame and the infrared light video frame based on the image features corresponding to the visible light video frame base layer sequence and the infrared light video frame base layer sequence; Based on the weights of the visible light video frame and the infrared light video frame, the base layer sequence of the visible light video frame and the base layer sequence of the infrared light video frame are fused to obtain a fused feature map, and based on the transposition of the fused feature map and the feature matrix, a fused base layer sequence is obtained; The fused base layer sequence is fused with the visible light video frame detail layer sequence to obtain a fused video frame sequence.

6. The electromechanical intelligent monitoring system according to claim 5, characterized in that: The expressions of the visible light video frame base layer and the infrared light video frame base layer are as follows: ； in, and are the expressions for the base layer of the visible light video frame and the base layer of the infrared light video frame, respectively. represents the filter function, represents the filter radius, represents the regularization parameter, and Represent visible light video frames and infrared light video frames respectively; The expressions of the visible light video frame base layer and the infrared light video frame detail layer are as follows: ； in, and Expressions representing the visible light video frame detail layer and the infrared light video frame detail layer respectively; The calculation formula of the covariance matrix of the visible light video frame base layer sequence and the infrared light video frame base layer sequence is as follows: ； in, and They represent the covariance matrices of the base layer sequence of visible light video frames and the base layer sequence of infrared light video frames, respectively. represents the total number of video frames in the base layer sequence, Indicates the first visible light video frames, Indicates the first infrared video frames, represents the average value of the visible light video frame, Represents the average value of infrared light video frames; The expressions of the image features corresponding to the base layer sequence of the visible light video frame and the base layer sequence of the infrared light video frame are as follows: ； in, and They represent the image features corresponding to the base layer sequence of visible light video frames and the base layer sequence of infrared light video frames, respectively. represents the feature matrix; The calculation formula of the weights of the visible light video frame and the infrared light video frame is as follows: ； in, and Respectively represent the weights of the visible light video frame and the infrared light video frame, represents the size of the feature matrix, represents the average value of image features of the base layer sequence of visible light video frames, Represents the average value of image features of the base layer sequence of infrared video frames; The expression of the fused feature map is as follows: ； in, Represents the fused feature map; The expression of the fused base layer is as follows: ； in, represents the fusion base layer; The expression of the fused video frame is as follows: ； in, Represents a fused video frame.

7. The electromechanical intelligent monitoring system according to claim 2, characterized in that: The anomaly detection model includes a feature extraction module, four cross-attention modules and a detection module. The feature extraction module is used to extract features from the key frame to obtain key frame features. The cross-attention module is used to perform feature denoising on the key frame features to obtain key frame denoising fusion features. The four cross-attention modules are connected in sequence. The detection module is used to perform feature detection on the key frame denoising features. The expression of the cross-attention module is as follows: ； in, represents the key frame denoising fusion feature, , and Represents 3 different convolutional layers, represents the self-attention layer, represents the keyframe denoising feature, and , represents the feature weight, represents a matrix transformation operation, represents the key frame features, , Represents the transformation function of the feature.

8. The electromechanical intelligent monitoring system according to claim 3, characterized in that: The state detection model adopts a support vector machine model.

9. An electromechanical intelligent monitoring method, applicable to an electromechanical intelligent monitoring system according to any one of claims 1 to 8, characterized in that: include: Use high-definition cameras and infrared cameras to conduct real-time video monitoring of the target area where the electromechanical equipment is located, and obtain visible light video and infrared light video; Synchronous video frame interception is performed according to a preset sampling period, and the intercepted video frames are fused to generate a fused video frame sequence; Extract feature points and descriptors using the trained feature point extraction network and descriptor extraction network, and select a key frame set based on the extracted feature points and descriptors; Use the trained anomaly detection model to perform anomaly detection to identify the abnormal key frame set, and combine the visible light video, infrared video and the abnormal key frame set to locate the video clips with abnormalities; Monitor the status of electromechanical equipment through the sensor group, obtain vibration data and temperature data, compare the temperature with the preset threshold, generate an alarm signal if the threshold is exceeded, extract vibration data features and perform status detection based on the status detection model to generate a status label; Abnormal video clips and electromechanical equipment status labels are displayed visually, and the status labels are attached under the corresponding videos. Based on the alarm signal, the intelligent alarm device is triggered to sound an alarm.

Images