Lightweight image recognition method and system for power plant safety

Abstract

The invention discloses a lightweight image recognition method and system for power plant safety, and relates to the field of image recognition, and the method comprises the steps: constructing a multi-scale adaptive perception image anomaly detection model, and recognizing the violation behaviors of power plant operators in power plant real-time video data through the image anomaly detection model; constructing a multi-stage binocular stereo area distribution model to match the feature points in the binocular image of the power equipment, and calculating the safety distance between the power plant operating personnel and the electrified body based on the matching result; and constructing an edge equipment reasoning model of lightweight knowledge distillation in combination with illegal behaviors and safety distances of power plant operating personnel, and monitoring potential safety hazards of the power plant in real time by using the edge equipment reasoning model. According to the method, the teacher model with high expression ability is deployed on the cloud for training, semantic information and structural knowledge are transmitted to the student model on the edge device through the dynamic adaptation strategy, and the recognition ability of the edge model in a computing power limited scene is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a lightweight image recognition method and system for power plant safety. Background Art

[0002] Power plant safety refers to ensuring the comprehensive safety of personnel, equipment, and systems during power production, transmission, and operation and maintenance. It covers multiple aspects such as operating specifications, explosion and touch protection, electrical isolation, and warning systems. Lightweight image recognition refers to the use of image recognition models with fast reasoning, low power consumption, and high adaptability deployed on edge devices to ensure the safety of operators and equipment in the high-risk environment of power plants. This enables efficient identification and real-time warning of typical violations such as wearing safety helmets, using insulating gloves, crossing line behaviors, and intrusion into live areas.

[0003] Existing image recognition technology lacks a deep understanding and dynamic adaptability of complex behavioral patterns and the evolution of spatiotemporal states, making it difficult to identify high-risk behaviors such as workers not wearing insulating gloves, crossing safety distance lines, and illegally entering live areas. This leads to insufficient anomaly recognition capabilities in dealing with sudden behavioral changes or occlusion interference, which in turn causes hidden dangers to escalate and even accidents to occur.

[0004] Currently, no effective solutions have been proposed for the problems in related technologies. Summary of the Invention

[0005] In response to the problems in the related art, the present invention proposes a lightweight image recognition method and system for power plant safety to overcome the above-mentioned technical problems existing in the existing related art.

[0006] To this end, the specific technical solutions adopted in the present invention are as follows:

[0007] According to one aspect of the present invention, a lightweight image recognition method for power plant safety is provided, the method comprising:

[0008] Acquire real-time video data from power plants, build a multi-scale adaptive perception image anomaly detection model, and use the image anomaly detection model to identify violations by power plant operators in real-time video data from power plants.

[0009] Acquire binocular images of power equipment, build a multi-stage binocular stereo matching model to match feature points in the binocular images of power equipment, and calculate the safe distance between power plant operators and live objects based on the matching results;

[0010] Based on the violations and safety distances of power plant operators, a lightweight knowledge distillation edge device inference model is constructed, and the edge device inference model is used to monitor power plant safety hazards in real time.

[0011] Preferably, a multi-scale adaptive perception image anomaly detection model is constructed, and the image anomaly detection model is used to identify the illegal behaviors of power plant operators in the real-time video data of the power plant, including:

[0012] A bilinear convolutional neural network is used to mine the visual features and spatial topological features of real-time video data from power plants, and the visual and spatial topological features are fused to identify reference power operation scenarios.

[0013] Based on a reference power operation scenario, the temporal context detector in the transition-aware context network is used to encode context information, and the transition-aware classifier is used to segment the real-time video data of the power plant into behavioral video segments and transition video segments.

[0014] Combining a multi-scale feature pyramid network with an adaptive attention mechanism to identify abnormal behaviors of power plant operators in transitional video clips;

[0015] Using time series models to identify abnormal posture behaviors of power plant workers in behavioral video clips.

[0016] Preferably, the multi-scale feature pyramid network is combined with the adaptive attention mechanism to identify abnormal operation behaviors of power plant operators in transitional video clips, including:

[0017] Extract transition image sequences from transition video clips and construct a multi-scale feature pyramid network based on the transition image sequences;

[0018] Calculate the sparse optical flow field of the transition image in the top layer of the multi-scale feature pyramid network, and perform inter-frame difference processing on adjacent transition images to obtain the initial optical flow vector of adjacent transition images;

[0019] Calculate the optical flow vectors of the remaining transition images based on the initial transport optical flow vector, and fuse them with the initial optical flow vector to obtain a fused optical flow vector;

[0020] The fused optical flow vector is compared with the preset threshold. If the fused optical flow vector is greater than the preset threshold, the optical flow vector of the maximum transition image is iteratively selected as the outlier and modified to the mean optical flow vector. Otherwise, the fused optical flow vector is passed to the next layer until it is passed to the bottom layer of the multi-scale feature pyramid network.

[0021] The fused optical flow vector of each layer is converted into a depth map, and the depth map is channel-stitched based on the attention mechanism. The transition-aware context network is used to segment abnormal operation behaviors from the spliced ​​channels.

[0022] Preferably, obtaining a binocular image of the power equipment, constructing a multi-stage binocular stereo matching model to match feature points in the binocular image of the power equipment, and calculating the safe distance between power plant operators and charged objects based on the matching results includes:

[0023] Use binocular cameras to obtain binocular images of power equipment, and build a multi-stage binocular stereo matching model to match the binocular images to obtain disparity images;

[0024] The depth value of the power equipment point in the disparity image is obtained by using triangulation method, and the three-dimensional coordinate points of the disparity image are reconstructed by using the improved semi-global block matching algorithm;

[0025] Based on the abnormal posture behavior of power plant operators, the spatial distance between the reconstructed three-dimensional coordinate points and the feature points of the target object is calculated, and the safe distance between power plant operators and live objects is delineated based on the spatial distance;

[0026] An electronic fence is drawn in the charged area based on the safety distance, and the power plant operators are monitored to see if they are within the electronic fence area. If the power plant operators exceed the electronic fence area, voice prompts are given to the power plant operators to re-enter the electronic fence area.

[0027] Preferably, based on the abnormal posture behavior of the power plant operator, the spatial distance between the reconstructed three-dimensional coordinate point and the characteristic point of the target object is calculated, and the safe distance between the power plant operator and the charged object is defined according to the spatial distance, including:

[0028] The abnormal posture behaviors of power plant operators are decomposed into atomic action units. A preset-sized image region is captured with the moving target as the center point. The image region is divided into small regions of preset sizes. Several prior anchor frames of different sizes are set at the center point of each small region.

[0029] A power plant operator recognition model is established, and prior anchor boxes of different sizes are assigned to target areas of different scales according to their sizes.

[0030] Calculate the target category confidence and target position confidence of each target area, and use the confidence calculation result as the prediction result of the prior anchor box. Modify the confidence according to the existence of the target and target category inside the prior anchor box;

[0031] Based on the prediction results of the modified prior anchor frame, the non-maximum suppression algorithm is used to remove duplicate prediction results as the output of the power plant operator recognition model;

[0032] Based on the output results of the power plant operator recognition model, the target object feature points are extracted, and the three-dimensional coordinates of the target object feature points and the surface centers of the charged objects are obtained to calculate the safe distance between the power plant operators and the charged objects.

[0033] Preferably, the confidence level is modified according to the presence of the target and target category inside the prior anchor box, including:

[0034] A dual causal path is established based on the existence of targets, target categories and predefined environmental interference factors within the prior anchor frame, and an intervening variable is introduced to test the dual causal path;

[0035] A main branch encoder and an auxiliary branch encoder are constructed separately to calculate the mutual information of the dual causal paths. Based on the mutual information, causal intervention is performed inside the prior anchor box through the dual causal paths to evaluate the impact of environmental interference factors on confidence.

[0036] Based on the degree of influence of environmental interference factors on confidence, the confidence is corrected by introducing interference compensation and category prior probability, and the corrected confidence is reversely injected into the main branch encoder so that the output of the main branch encoder is semantically consistent with the corrected confidence.

[0037] Preferably, a lightweight knowledge distillation edge device inference model is constructed based on the illegal behaviors and safety distances of power plant operators. The edge device inference model is then used to monitor power plant safety hazards in real time, including:

[0038] The illegal behaviors and safety distances of power plant operators are analyzed to build an edge device inference model based on lightweight knowledge distillation of convolutional networks;

[0039] The edge device inference model is compressed through operator optimization technology, tensor optimization technology, and data optimization technology.

[0040] Identify power plant electrical hazards based on model compression results, and optimize edge device inference model performance by combining self-supervised contrastive learning and meta-learning to adapt to new types of power plant electrical hazards.

[0041] The predefined teacher model is deployed in the cloud for training, and knowledge is transferred to the lightweight student model through a dynamic adaptation strategy to optimize the reasoning capability of the edge device inference model.

[0042] Preferably, a predefined teacher model is deployed in the cloud for training, and knowledge is transferred to a lightweight student model through a dynamic adaptation strategy to optimize the reasoning capability of the edge device inference model, including:

[0043] The real-time video data of the power plant and binocular images of power equipment are integrated to obtain a sample set. The sample set is divided into labeled data and input into the teacher model deployed in the cloud and the local lightweight student model respectively.

[0044] The teacher model and the local lightweight student model output predicted labels respectively, and the predicted samples with clear labels are introduced into the classification loss function to calculate the label difference between the predicted labels of the student model and the true labels;

[0045] Adjust the weight of the teacher model based on the label difference of the student model, transfer knowledge to the student model through a dynamic adaptation strategy based on the optimized teacher model, and adjust the weight of the teacher model in reverse;

[0046] The teacher model weight update and knowledge distillation are iteratively executed until the iteration conditions are met, realizing the knowledge exchange between the teacher model and the student model to optimize the reasoning capability of the edge device inference model.

[0047] Preferably, the confidence correction formula is:

[0048]

[0049] Where, Pr(Class i ) represents the probability that the sample belongs to category i; Pr(Class i |Object) indicates the possibility of predicting the target data as the i-th category; Pr(Object) indicates the possibility of the existence of the target in the region; It represents the intersection-over-union ratio between the predicted target and the actual target.

[0050] According to another aspect of the present invention, a lightweight image recognition system for power plant safety is provided, comprising:

[0051] The violation identification module is used to obtain real-time video data from the power plant, build a multi-scale adaptive perception image anomaly detection model, and use the image anomaly detection model to identify violations by power plant operators in the real-time video data of the power plant;

[0052] The safety distance calculation module is used to obtain binocular images of power equipment, build a multi-stage binocular stereo matching model to match feature points in the binocular images of power equipment, and calculate the safety distance between power plant operators and live objects based on the matching results;

[0053] The safety hazard monitoring module is used to build an edge device inference model for lightweight knowledge distillation based on the violations and safety distances of power plant operators, and use the edge device inference model to monitor power plant safety hazards in real time.

[0054] The beneficial effects of the present invention are:

[0055] 1. The present invention effectively enhances the model's perception of the relationships between people, objects, and backgrounds in operational scenarios by synchronously mining the visual features and spatial topological structures of images in real-time video data from power plants. It also introduces a transition-aware context network to accurately divide behavioral video segments from behavioral transition segments. Simultaneously, it integrates a multi-scale feature pyramid structure with an adaptive attention mechanism to perform feature focus on subtle action changes in transition segments, identifying fine-grained dangerous actions such as bending over and crossing the line, and illegal hand contact. This enhances the temporal sensitivity of the behavioral evolution process, meeting the needs for precise identification and early warning of abnormal behaviors in high-risk, strong real-time, and safety-constrained scenarios at power plants.

[0056] 2. The present invention obtains binocular images of power equipment, constructs a multi-stage stereo matching model to accurately match feature points in the image, generates a high-quality disparity map, and combines it with an improved semi-global block matching algorithm to achieve spatial reconstruction of the power plant operation site. Based on the abnormal posture behavior of the operators, the spatial distance between them and the charged objects is calculated, thereby scientifically defining the safe distance range between personnel and dangerous sources, thereby effectively enhancing the comprehensive perception ability of operating behavior and spatial risks.

[0057] 3. The present invention constructs a lightweight knowledge distillation reasoning model based on a convolutional network, deploys a teacher model with high expressive ability on the cloud for training, and transmits semantic information and structural knowledge to the student model on the edge device through a dynamic adaptation strategy, significantly improving the recognition ability of the edge model in scenarios with limited computing power; at the same time, it combines compression technologies such as operator optimization, tensor rearrangement, and data format reconstruction to perform structural and data-level trimming of the model, significantly reducing memory usage and computing latency; based on the compressed model, it efficiently identifies electrical hazards in the power plant operating environment, thereby constructing a highly reliable power plant safety intelligent perception system that takes into account performance, resource efficiency, and self-learning capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] Figure 1 is a flow chart of a lightweight image recognition method for power plant safety according to an embodiment of the present invention;

[0060] Figure 2 This is a principle block diagram of a lightweight image recognition system for power plant safety according to an embodiment of the present invention;

[0061] Figure 3This is a technical framework diagram of safety protection measures identification based on fine-grained detection in a lightweight image recognition method for power plant safety according to an embodiment of the present invention;

[0062] Figure 4 This is a schematic diagram of a binocular parallel structure imaging principle in a lightweight image recognition method for power plant safety according to an embodiment of the present invention;

[0063] Figure 5 This is a schematic diagram of Census transformation in a lightweight image recognition method for power plant safety according to an embodiment of the present invention;

[0064] Figure 6 2. A schematic diagram of quadratic curve interpolation in a lightweight image recognition method for power plant safety according to an embodiment of the present invention;

[0065] Figure 7 This is a flowchart of power operation proximity detection based on binocular stereo matching in a lightweight image recognition method for power plant safety according to an embodiment of the present invention;

[0066] Figure 8 This is a network structure diagram of a YOLO algorithm in a lightweight image recognition method for power plant safety according to an embodiment of the present invention;

[0067] Figure 9 is a schematic diagram of a tensor optimizer in a lightweight image recognition method for power plant safety according to an embodiment of the present invention;

[0068] Figure 10 2. This is a schematic diagram of ReLU operations before quantization in a lightweight image recognition method for power plant safety according to an embodiment of the present invention;

[0069] Figure 11 2. This is a schematic diagram of a ReLU operation after quantization in a lightweight image recognition method for power plant safety according to an embodiment of the present invention;

[0070] Figure 12 This is a flow chart of convolutional neural network parameter quantization in a lightweight image recognition method for power plant safety according to an embodiment of the present invention.

[0071] In the picture:

[0072] 1. Violation identification module; 2. Safety distance calculation module; 3. Safety hazard monitoring module. DETAILED DESCRIPTION

[0073] To further illustrate the embodiments, the present application provides drawings which are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can be used to explain the operation principle of the embodiments in conjunction with the related description of the specification. Those of ordinary skill in the art can understand other possible implementations and advantages of the present application in conjunction with these contents. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0074] According to an embodiment of the present application, a power plant safety-oriented lightweight image recognition method and system are provided.

[0075] The present application will be further described in conjunction with the drawings and specific embodiments. As shown in the drawings and specific embodiments, according to the power plant safety-oriented lightweight image recognition method of the embodiment of the present application, the method comprises: Figure 1

[0076] S1, acquiring real-time video data of a power plant, constructing a multi-scale adaptive perception image anomaly detection model, and using the image anomaly detection model to identify the violation behavior of the power plant operating personnel in the real-time video data of the power plant.

[0077] Among them, constructing a multi-scale adaptive perception image anomaly detection model, and using the image anomaly detection model to identify the violation behavior of the power plant operating personnel in the real-time video data of the power plant comprises:

[0078] Using a bilinear convolutional neural network to mine visual features and spatial topological features in the real-time video data of the power plant, and fusing the visual features and spatial topological features to identify a reference power operation scene.

[0079] It should be noted that the intelligent identification and early warning technology based on AI image recognition technology is based on the violation behavior of the power plant operation site. By using a lightweight mobile edge terminal with AI computing capability, using target identification and detection technology, fine intelligent identification and real-time monitoring of behaviors such as not wearing a safety helmet, not wearing a safety belt correctly, improper dress, smoking, etc. are realized, and voice alarm is performed according to the identification result.

[0080] The purpose of power plant near-electricity operation scene recognition is to fully mine the visual features in the image and the spatial topological relationship features between production safety objects according to the perceived real-time video data of the power plant, realize the cognition and understanding of the operation scene and type, and lay a foundation for safety risk identification.

[0081] Considering the special problems in power plant near-electricity operation scene understanding, the present application adopts a power scene understanding framework based on cross-modal fusion of visual features and spatial topological features.

[0082] ​Based on the reference power operation scenario, the temporal context detector in the transition-aware context network is used to encode the context information, and the transition-aware classifier is used to segment the real-time video data of the power plant into behavioral video segments and transition video segments.

[0083] It should be noted that, in response to the demand for refined identification of personnel and other objects in power plant production safety management, the present invention is based on a power production safety hazard object identification technology based on fine-grained target detection. Considering that the feature differences between power plant production safety objects are small, the present invention proposes a fine-grained detection method based on a bilinear convolutional neural network to improve the recognition performance of power production safety hazard objects. The specific architecture of this method is as follows: Figure 3 shown.

[0084] Depend on Figure 3 It can be seen that the fine-grained target detection of the bilinear convolutional neural network is a four-tuple, specifically:

[0085] F=(f A ,f B ,P,C);

[0086] Where, F represents the fine-grained target detection result; f A and f B express Figure 3 In the figure, P represents the feature extraction function of the two bilinear convolutional neural networks A and B, and C represents a classification function that classifies and recognizes the bilinear feature vector after the Softmax normalization layer.

[0087] The feature extraction f of this model is a function mapping process, f:L×I→R c *D, this process converts the input image L and the location area I into a vector of size c×D through a mapping relationship, and then extracts the feature function f A and f B The outer product operation is performed on the output of to obtain the bilinear feature vector at the corresponding position. The pooling function P integrates the obtained bilinear features to obtain a feature function for fine-grained classification.

[0088] The pooling process is performed by accumulating the bilinear features of each corresponding position on the image. The specific calculation is as follows:

[0089]

[0090] Where, Represents the result of bilinear feature accumulation, l represents the sub-region of the area pooling kernel size in the region, B i Represents a bilinear operation.

[0091] According to the relevant guide manual, a standard action library is established for a typical work scene, and key actions are segmented, effective actions are identified, and redundant transition actions are removed to form a key action video sample library. First, the live-line work site video is marked as two video segments, i.e., effective key operation actions and interval transition actions, and the transition perception context network is trained to realize automatic segmentation of the post training monitoring video.

[0092] The transition perception context network mainly includes two core parts, i.e., a timing context detector and a transition perception classifier. The timing context detector takes a standard SSD (single-stage multi-frame detector) framework as a main part, and can encode long-time scale context information by embedding multi-scale bidirectional Conv-LSTM (a deep learning module combining convolutional neural network and long short-term memory network) units in different feature maps. The transition perception classifier classifies actions and action states by tracking and identifying dynamic parts, thereby distinguishing segments in the monitoring video, extracting effective action videos of the work personnel in the monitoring video, and ensuring that the video segments containing complete operations can be detected during action detection.

[0093] Long-term timing context information is crucial for spatio-temporal action detection. However, the standard SSD performs action detection from multiple feature maps of different sizes, and does not consider timing context information. To extract timing context, a Bi-ConvLST unit can be embedded in the SSD to design a cyclic detector for detecting actions. As a kind of LSTM, ConvLSTM can encode long-term information and is more suitable for processing video data, so the ConvLSTM unit can replace the multiplication operation of the fully connected operation in the LSTM unit with a convolution operation, thereby maintaining the spatial structure of the frame over time.

[0094] The specific details of the transition perception classifier include that in the monitoring video, effective operation video segments and transition action video segments often appear alternately, and if the effective operation video segments cannot be accurately extracted from the monitoring video, the task completion of the work personnel cannot be analyzed. The instances in the transition state have similarities with the target actions, and the detection is prone to confusion. The transition perception classifier classifies the actions and action states in the video frame by designing two classifiers, can locate the start frame and end frame of the effective operation video segment in the monitoring video, and thereby divides the entire monitoring video into key operation video segments and transition video segments.

[0095] The multi-scale feature pyramid network is combined with the adaptive attention mechanism to identify abnormal behaviors of the power plant workers in the transition video segments.

[0096] The multi-scale feature pyramid network is combined with an adaptive attention mechanism to identify abnormal operation behaviors of the power plant operators in the transition video segment, including:

[0097] A transition image sequence is extracted from the transition video segment, and a multi-scale feature pyramid network is constructed based on the transition image sequence;

[0098] A sparse optical flow field of the transition image in the topmost layer of the multi-scale feature pyramid network is calculated, and an initial optical flow vector of adjacent transition images is obtained through inter-frame difference processing;

[0099] The optical flow vectors of the remaining transition images are calculated based on the initial optical flow vector, and are fused with the initial optical flow vector to obtain a fused optical flow vector;

[0100] The fused optical flow vector is compared with a preset threshold value, if the fused optical flow vector is greater than the preset threshold value, the optical flow vector of the largest transition image is selected as an abnormal point and is modified to the mean value of the optical flow vector, otherwise, the fused optical flow vector is passed to the next layer until it is passed to the base layer of the multi-scale feature pyramid network;

[0101] The fused optical flow vector of each layer is converted into a depth map, the depth map is spliced based on an attention mechanism, and the abnormal operation behavior is segmented from the spliced channel by using a transition perception context network.

[0102] The posture abnormal behavior of the power plant operator in the behavior video segment is identified by using a time series model.

[0103] S2, acquire the binocular image of the power equipment, construct a multi-stage binocular stereo block matching model, match the feature points in the binocular image of the power equipment, and calculate the safety distance between the power plant operators and the live body based on the matching result.

[0104] The binocular image of the power equipment is acquired, a multi-stage binocular stereo block matching model is constructed, the feature points in the binocular image of the power equipment are matched, and the safety distance between the power plant operators and the live body is calculated based on the matching result, including:

[0105] The binocular image of the power equipment is acquired by using a binocular camera, and a multi-stage binocular stereo block matching model is constructed to match the binocular image, and a disparity image is obtained;

[0106] The depth value of the power equipment point in the disparity image is obtained by using the triangulation method, and the three-dimensional coordinate points of the disparity image are reconstructed by improving the semi-global block matching algorithm;

[0107] Based on the posture abnormal behavior of the power plant operator, the spatial distance between the reconstructed three-dimensional coordinate points and the feature points of the target object is calculated, and the safety distance between the power plant operator and the live body is determined according to the spatial distance.

[0108] Among them, based on the abnormal posture behavior of power plant operators, the spatial distance between the reconstructed three-dimensional coordinate points and the feature points of the target object is calculated, and the safe distance between power plant operators and charged objects is delineated based on the spatial distance, including:

[0109] The abnormal posture behaviors of power plant operators are decomposed into atomic action units. A preset-sized image region is captured with the moving target as the center point. The image region is divided into small regions of preset sizes. Several prior anchor frames of different sizes are set at the center point of each small region.

[0110] A power plant operator recognition model is established, and prior anchor boxes of different sizes are assigned to target areas of different scales according to their sizes.

[0111] The target category confidence and target position confidence of each target area are calculated, and the confidence calculation result is used as the prediction result of the prior anchor box. The confidence is corrected according to the existence of the target and target category inside the prior anchor box.

[0112] Among them, the confidence correction based on the existence of targets and target categories inside the prior anchor box includes:

[0113] A dual causal path is established based on the existence of targets, target categories and predefined environmental interference factors within the prior anchor frame, and an intervening variable is introduced to test the dual causal path;

[0114] A main branch encoder and an auxiliary branch encoder are constructed separately to calculate the mutual information of the dual causal paths. Based on the mutual information, causal intervention is performed inside the prior anchor box through the dual causal paths to evaluate the impact of environmental interference factors on confidence.

[0115] Based on the degree of influence of environmental interference factors on confidence, the confidence is corrected by introducing interference compensation and category prior probability, and the corrected confidence is reversely injected into the main branch encoder so that the output of the main branch encoder is semantically consistent with the corrected confidence.

[0116] Based on the prediction results of the modified prior anchor frame, the non-maximum suppression algorithm is used to remove duplicate prediction results as the output of the power plant operator recognition model;

[0117] Based on the output results of the power plant operator recognition model, the target object feature points are extracted, and the three-dimensional coordinates of the target object feature points and the surface centers of the charged objects are obtained to calculate the safe distance between the power plant operators and the charged objects.

[0118] An electronic fence is drawn in the charged area based on the safety distance, and the power plant operators are monitored to see if they are within the electronic fence. If the power plant operators exceed the electronic fence, voice prompts are given to the operators to re-enter the electronic fence.

[0119] It should be noted that an intelligent safety distance monitoring edge device has been developed for near-electrical operations in power plants. Using high-precision binocular cameras, intelligent target recognition technology, and advanced algorithms such as binocular image stereo matching, it performs image processing, image analysis, target recognition, and safety distance calculation on-site, and visualizes the results.

[0120] Binocular stereo matching is an important technology in computer vision, capable of acquiring three-dimensional information about an object from images captured by two cameras. However, traditional binocular stereo matching methods have several challenges, such as sensitivity to noise, illumination variations, and low matching efficiency. Therefore, a binocular stereo matching method based on deep learning can improve matching efficiency and accuracy, and enhance the accuracy of image processing.

[0121] The binocular stereo matching method based on deep learning includes the following steps: extracting features from images using a multi-stage binocular stereo matching model based on the channel attention module; matching the features of the two images using a traditional stereo matching algorithm, and training a classifier using an efficient recognition model for elements in live working scenes to screen and optimize the matching results.

[0122] The proposed binocular stereo matching-based distance measurement technology for power equipment can calculate the spatial distance of the equipment based on a depth map using methods such as triangulation. This technology can be used to obtain the depth values ​​of all points in the disparity image and reconstruct the three-dimensional coordinates of the image.

[0123] Based on the reconstructed three-dimensional coordinate point data, intelligent target recognition technology can be combined to calculate the spatial distance between feature points, further solving the problem of difficult to accurately measure the safe distance of large vehicles working at height during substation near-line operations. By identifying the feature points of vehicles, near-line workers, and energized equipment and calculating the distances between them, it is possible to accurately understand and identify the near-line operation scene, improving operational safety and efficiency. Specifically, this includes:

[0124] A binocular camera is used to obtain binocular images of the power equipment, and then binocular stereo matching is performed to obtain a disparity image. The depth values ​​of all points in the disparity image are obtained using triangulation, and the three-dimensional coordinate points of the image are reconstructed to obtain the spatial information of the power equipment. The intelligent target recognition technology is combined to extract the feature points of the target object for spatial distance measurement. The spatial distance between the feature points is calculated based on the reconstructed three-dimensional coordinate point data.

[0125] The present invention uses edge intelligent devices as the detection platform. First, it uses a binocular camera to perceive the substation operation image, and adopts binocular stereo vision technology based on SGBM (Semi-Global Block Matching) stereo matching technology for distance detection to obtain a dense depth map and a three-dimensional point cloud model. Secondly, it uses a lightweight substation operator equipment identification model to detect the operation object. Finally, based on the dense depth map of the substation operation site and the target detection results, dynamic 3D distance identification of the substation operator is performed to achieve accurate judgment of various distances between the operator and live equipment, operation objects, etc., and out-of-bounds reminders.

[0126] The binocular stereo vision matching principle can be simplified as follows Figure 4 As shown, the imaging planes of the left and right cameras are located in the same plane, O L , O R are the optical centers of the left and right cameras respectively. The two cameras observe the same target point P in space at the same time, and the imaging points P of the left and right cameras are obtained respectively. L 、P R , T is the baseline distance of the binocular camera, Z is the distance between the target point and the optical center, generally called depth, and f is the focal length of the camera.

[0127] According to the principle of similar triangles, when the coordinates of the target point in the imaging points of the left and right images are known, the three-dimensional coordinates of the target point in the actual world coordinates can be obtained, that is:

[0128]

[0129] Where, X L 、X R Respectively represent the coordinates of the same object or feature in the left image and the right image; X L -X R represents the parallax; f represents the focal length of the camera.

[0130] Binocular camera calibration: During the camera photography process, the coordinates of the target point in the world coordinate system are projected and converted into two-dimensional coordinates in the image coordinate system, while retaining the color information. Conversely, during the three-dimensional reconstruction process of binocular vision, the two-dimensional coordinates in the image coordinate system need to be restored to three-dimensional coordinates in the world coordinate system. That is, if the positions of the left and right imaging points in the image coordinate system are known and want to be converted into three-dimensional information in the world coordinate system, a spatial transformation matrix is ​​required, which requires the internal and external parameters of the binocular camera. The parameters are generally obtained through binocular camera calibration and are expressed as follows:

[0131]

[0132] Where [αβ] is the focal length of the camera; γ is the distortion coefficient of the camera; [u0 vo ] is the coordinate of the principal point; K is the intrinsic parameter of the camera; R is the rotation matrix of the camera; t is the translation vector of the camera; [R t] is the extrinsic parameter position parameter of the camera; (X w , Y w , Z w ) are the three-dimensional point coordinates in the world coordinate system.

[0133] The intrinsic parameters, extrinsic parameters and distortion coefficients obtained from camera calibration are used to correct the power operation images, laying the foundation for the subsequent acquisition of three-dimensional point cloud data for live operations.

[0134] The SGBM algorithm, a semi-global stereo matching algorithm, is widely used due to its high accuracy. Its core principle is to leverage the strengths of global stereo matching algorithms to find the minimum value of the global energy function, thereby obtaining the optimal right matching point for the left imaging point and, consequently, the optimal disparity map. This method primarily relies on mutual information for stereo matching. The algorithm's workflow consists of preprocessing, cost calculation, dynamic programming, and post-processing.

[0135] 1) Preprocessing: The algorithm first processes the initial image using the Sobel operator. It then traverses each pixel of the preprocessed image and regenerates a new image using the function. The purpose of preprocessing is to collect changes in the image in a certain direction in preparation for cost calculation. The function expression is:

[0136]

[0137] Where pre represents the preset threshold parameter, which is used to control the mapping behavior of input P; P NEW It is the new image obtained after function processing.

[0138] 2) Cost calculation: This part consists of two parts. The gradient of the image processed by the Sobel operator is calculated, and the unprocessed image is sampled to obtain the SAD cost, which is expressed as:

[0139]

[0140] Where, SAD cost is the sum of absolute differences; p is the pixel value; (x, y) is the coordinate of the pixel to be matched in the left image; d is the candidate disparity; I L , I R are the grayscale values ​​of the left and right images respectively; k is the radius of the matching window.

[0141] 3) Dynamic Programming: SGBM's cost aggregation method adopts the same global energy optimization strategy as the global matching algorithm. First, the energy function is constructed as follows:

[0142]

[0143] Where C is the total matching cost of the entire disparity map, P1 and P2 are two penalty coefficients, which penalize different disparity differences respectively. The ultimate goal is to make the obtained disparity map as smooth as possible; D is the disparity map, p is the current pixel, N is the p represents the neighborhood of pixel p, T[] is the indicator function (1 when the condition is met, otherwise 0), where:

[0144] When the disparity between the neighboring pixels q and p differs by 1, a fixed penalty P1 is imposed. The main function is to allow slight changes in disparity (such as tilted surfaces).

[0145] When the disparity change is greater than 1, a larger penalty P2 is imposed (usually P2>>P1). The main function is to suppress discontinuous disparity jumps (such as occlusion boundaries).

[0146] 4) Post-processing: After the above dynamic programming, an initial disparity map is obtained. However, this disparity map often has significant errors and requires further processing to optimize the disparity map and improve the depth accuracy of the target point. The left-right consistency method is generally used to eliminate false matches in the initial disparity map. False matches are filtered based on the assumption that each pixel can only have one correct disparity.

[0147] However, the SGBM algorithm that comes with OPENCV is based on mutual information. Although it has high matching accuracy, its computational complexity is often very large and has high requirements on the hardware for image processing, resulting in low image processing efficiency. In practical applications, it is difficult to be widely used due to its poor real-time performance. Census transform is widely used in the field of stereo matching because it can better detect local features of images and has a fast speed. Figure 5 As shown, the present invention is based on the traditional SGBM algorithm and applies the Census transformation to the initial cost calculation of SGBM, which specifically includes:

[0148] Construct a window of size n×m, and compare the grayscale values ​​of the pixels in the window with the grayscale value of the center pixel of the window based on the preset conditions. The compared values ​​are combined into a bit string, which can represent the closeness between the matching point pairs. The greater the probability that the two matching points are the same point, the greater the similarity of the bit strings. The expression of the preset conditions is:

[0149]

[0150] After obtaining the Census transformation value corresponding to the pixel, the matching cost of the matching point cannot be intuitively displayed, that is, the best matching point cannot be directly selected. Further processing of the transformation value is required. Therefore, based on the obtained transformation value, the Hamming distance of the Census transformation values ​​of the two pixels in the left and right image pairs is calculated. That is, the corresponding bits of the two bit strings are traversed to find the number of differences. This value directly reflects the similarity of the two points. That is, the smaller the value, the greater the probability that the two points in the left and right images are the same point in three-dimensional space. After this step, the matching cost of the entire image can be finally obtained. The calculation formula of the Hamming distance is:

[0151]

[0152] Where Q is the Hamming distance calculation result of the two windows; I l , I r are the Census transformation bit string values ​​of the left and right images respectively; n is the number of pixels in the window; ∧, >>, & represent XOR operation, shift operation and bitwise AND operation respectively, Q m is the Hamming distance calculation result of the m-th window, and the matching cost of the left and right pixels is obtained by Hamming distance calculation.

[0153] At the same time, in order to improve the matching accuracy of pixel points, the quadratic curve interpolation method is used to obtain the optimal pixel accuracy of the target point. That is, through the three lowest matching points, according to the solution of the binary equation, the lowest point of the parabola is calculated, that is, the extreme point of the curve. The disparity value corresponding to this point is the new sub-pixel disparity value of the target point, such as Figure 6 As shown in the figure, the accuracy of pixel depth is greatly improved by solving the sub-disparity value. Among them, the expression of the lowest point of the parabola is calculated by solving the binary equation through the three lowest matching points:

[0154]

[0155] Calculation of safe distance between live working personnel and equipment: Algorithm processing steps are as follows Figure 7 As shown in the figure, a binocular camera is used as the data acquisition device, and the improved SGBM stereo matching algorithm is used to perform real-time 3D reconstruction of substation on-site operations. The left view is used as the input of the lightweight target detection algorithm to detect workers and equipment, and the corresponding 3D coordinates are obtained for safe distance judgment to ensure that power plant personnel and construction equipment are within the safe operating area.

[0156] Considering that the occurrence of safety risks at live working sites is strongly correlated with the subjective behavior of workers, and that the active behavior of workers has strong dynamic continuity and instantaneity, the safety risks and violations in the power industry also occur instantaneously, showing dynamic and strong instantaneous characteristics. The present invention uses the detection frame of workers and equipment for target detection, calculates the mean of the corresponding point cloud data within the detection frame as the three-dimensional surface center of the workers and equipment, and then accurately calculates the spatial distance between the workers and equipment. The distance calculation method between coordinate points includes:

[0157] The three-dimensional coordinates (X k ,Y k ,Z k ) and (X,Y,Z), directly find the Euclidean distance between two points:

[0158]

[0159] There are many moving targets in power plant near-electrical operations. In addition to the operating objects, environmental factors such as equipment movement and indicator light status changes may be detected as moving targets. To solve this problem, the present invention establishes an operating object recognition model based on the YOLO algorithm. Based on the target detection in the previous step, the operating object information is further confirmed.

[0160] After receiving the video frame, the model first resizes it. Based on the moving object detection results previously described, a 608×608 image region is captured with the moving object as the center point as input. The entire image is evenly divided into 19×19 regions, and nine anchor boxes of different sizes are set at the center of each region. The model then extracts image features using a feature extraction network and predicts objects within each region using a prediction head. For each region, the number of categories C = 1, as the only target type to be predicted is work objects. For each target, four coordinate values ​​and one confidence value are predicted, totaling five values. Therefore, the prediction result is represented by a 19×19×(9×(5+1)) matrix. Finally, the model uses non-maximum suppression to filter out duplicate predictions for the same target, ultimately obtaining the work object detection results and location information in the surveillance image.

[0161] To achieve multi-scale object detection, this model extracts feature maps of three different scales and assigns nine prior anchor boxes of different sizes to target predictions at different scales according to their sizes. The feature maps of the three scales correspond to 19×19, 38×38, and 76×76 detection areas, respectively. There are three prior anchor boxes corresponding to each scale, so the output at each scale is 19×19×18. For each prior anchor box, it is only responsible for predicting targets whose center points are within the area where the anchor box is located.

[0162] The final confidence level is used to determine whether there is an object in each area, without discarding the background area anchor box during the prediction process. Therefore, when calculating the confidence level, the possibility of the existence of the object needs to be included. Specifically, it includes two parts: the target category confidence level and the target position confidence level. The calculation formula is as follows:

[0163]

[0164] In the formula, confidence is the confidence level. For each region, if there is an object, the Pr(Object) value of all anchor boxes in the region is close to 1, otherwise it is 0. To evaluate the intersection-over-union ratio between the predicted target and the actual target, the larger the intersection-over-union ratio, the more accurate the position prediction. In general, the prediction result of each anchor box needs to be represented by the position coordinates (x, y, w, h), confidence and softmax vector representing the category information. The entire network structure is as follows: Figure 8 As shown ( Figure 8 In the figure, Type is the layer type, Filters is the number of convolution kernels, Size is the convolution kernel size, Output is the spatial size of the output feature map of this layer, Convolutional is the convolution layer, Residual is the residual connection, Avgpool is the average pooling layer, Connected is the fully connected layer, Softmax is the activation function for classification output, Global is the global average pooling, YOLO Detection is the YOLO target detection head, and Scale1 / 2 / 3 are three scales).

[0165] When predicting, the present invention needs to consider both the existence of the target in the anchor box and the category of the target, so the confidence is further modified to:

[0166]

[0167] Where, Pr(Object) is the probability that there is an object in the current position / area; Pr(Class i ) is the probability that the sample belongs to category i, that is, the confidence of the model predicting that the target is the i-th category; Pr(Class i|Object) is the probability of predicting the target data to be of class i. Since this model only predicts the class of the job object, Pr(Class i |Object)=1,Pr(Object) and The confidence level of the predicted target is obtained by multiplying these three confidence levels together. For the predicted results, it is necessary to filter out invalid targets using the confidence level and use the non-maximum suppression algorithm to remove duplicate prediction results to obtain accurate target detection results.

[0168] S3. Based on the violations and safety distances of power plant operators, a lightweight knowledge distillation edge device inference model is constructed, and the edge device inference model is used to monitor power plant safety hazards in real time.

[0169] Among them, combining the illegal behaviors and safety distances of power plant operators, we built an edge device inference model for lightweight knowledge distillation, and used the edge device inference model to monitor power plant safety hazards in real time, including:

[0170] The illegal behaviors and safety distances of power plant operators are analyzed to build an edge device inference model based on lightweight knowledge distillation of convolutional networks;

[0171] The edge device inference model is compressed through operator optimization technology, tensor optimization technology, and data optimization technology.

[0172] Identify power plant electrical hazards based on model compression results, and optimize edge device inference model performance by combining self-supervised contrastive learning and meta-learning to adapt to new types of power plant electrical hazards.

[0173] The predefined teacher model is deployed in the cloud for training, and knowledge is transferred to the lightweight student model through a dynamic adaptation strategy to optimize the reasoning capability of the edge device inference model.

[0174] The predefined teacher model is deployed in the cloud for training, and knowledge is transferred to the lightweight student model through a dynamic adaptation strategy to optimize the reasoning capability of the edge device inference model.

[0175] The real-time video data of the power plant and binocular images of power equipment are integrated to obtain a sample set. The sample set is divided into labeled data and input into the teacher model deployed in the cloud and the local lightweight student model respectively.

[0176] The teacher model and the local lightweight student model output predicted labels respectively, and the predicted samples with clear labels are introduced into the classification loss function to calculate the label difference between the predicted labels of the student model and the true labels;

[0177] Adjust the weight of the teacher model based on the label difference of the student model, transfer knowledge to the student model through a dynamic adaptation strategy based on the optimized teacher model, and adjust the weight of the teacher model in reverse;

[0178] The teacher model weight update and knowledge distillation are iteratively executed until the iteration conditions are met, realizing the knowledge exchange between the teacher model and the student model to optimize the reasoning capability of the edge device inference model.

[0179] It should be noted that with the development of the power Internet of Things and the digital transformation of energy, the application scenarios of power edge intelligent terminals are constantly expanding. AI chips, due to their inherent intelligent analysis and parallel computing capabilities, have become one of the mainstream core components of power edge terminals. How to achieve rapid identification of power-related hidden dangers based on convolutional networks on AI chips with limited computing power requires the use of model compression methods. This paper will compress the model from three perspectives: operator optimization, tensor optimization, and data optimization.

[0180] Computational graphs are a common way to represent programs in deep learning frameworks. Currently, mainstream frameworks use dynamic computational graphs, where large numbers of multidimensional tensors flow through the computational graph as intermediate data. This type of computational graph offers flexibility, but the high volume of data exchange between operators makes it unsuitable for low-power, low-bandwidth embedded development environments at the edge. To address this, the present invention employs a novel computational graph optimization scheme, using nodes to represent tensor operations and edges to represent the data flow and data dependencies between tensors.

[0181] This computational graph composed of nodes and edges provides a global view of the computational task, while also avoiding describing how each computational task is implemented. In a graph, the static memory plan Pass can be processed by pre-allocating all intermediate result Tensors. This allocation stage is similar to the register allocation Pass in a traditional compiler. In theory, a computational graph can be converted into a function formula graph. For example, a constant folding Pass can be statically executed in the computational graph pre-computation stage to save runtime costs. Based on the static computational graph, the present invention also uses two methods, operator fusion and data layout transformation, to optimize the computational graph.

[0182] Operator fusion: Operator fusion combines multiple operators from the original computation graph into a single one. For multi-neural network computing card architectures based on domestic chips, this optimization method of fusing multiple operations can significantly reduce execution time. This is because the operations originating from a single kernel function save the time required to write intermediate results back to global memory. Specifically, this invention achieves fusion for the following operations: addition, subtraction, square root, summation, maximum, minimum, and standard two-dimensional convolution.

[0183] To verify the effectiveness of operator fusion, the present invention performed operator fusion on the standard two-dimensional convolution operations on the backbone network ResNet50 and the backbone network MobileNet, and gave the calculation time required for a single convolution operation before and after fusion. After operator fusion, the time consumed for a single convolution was reduced by an average of about 30%, which is a significant effect.

[0184] Data layout conversion: Tensor operations are fundamental to computational graphs. The operations involved in tensors have different data layout requirements depending on the operator. For the domestic neural network computing RK1808, 4x4 tensor operations are generally used, so data needs to be split into 4x4 blocks for storage to optimize local memory access efficiency. The original tensor matrix size is 4x8. To accommodate 4x4 tensor operations, the original tensor needs to be split into 2x4x2x2 blocks to align with the hardware layout of the neural network computing card, improving computational efficiency and utilization.

[0185] Tensor optimization based on random forest and particle swarm optimization algorithms: Tensor optimization fully considers the co-optimization of operations and data when generating target code. To optimize memory access performance, deep learning processors often design on-chip caches close to the compute units. Accessing on-chip caches is significantly faster than accessing off-chip DDR. To simplify the design, deep learning processors do not design on-chip caches as software-transparent structures, so software programmers must explicitly manage these caches. Proper use of on-chip caches can significantly improve program performance, which cannot be achieved solely through computational optimization. To fully utilize on-chip caches in applications that support variable-size vector-matrix and matrix-matrix operations, the code generator must consider both computational and data optimization. Furthermore, because deep learning processors support vectorized operations and fixed-dimensional loop unrolling, special off-chip data alignment and dimensionality transformation are often required to improve memory access efficiency.

[0186] Step 1: Random Forest and Particle Swarm Optimization Algorithms:

[0187] Random forest is a supervised learning algorithm based on bagging and decision trees. Decision trees are a widely used classification and regression method, and in most cases, they offer high accuracy. However, when the data is complex or noisy, decision trees are prone to overfitting, resulting in reduced accuracy. Random forest, an ensemble of decision trees, is less susceptible to noise and significantly improves the overfitting problem of decision trees. It can also achieve good fitting results for high-dimensional data. Furthermore, random forests can run in parallel, resulting in faster and more efficient training.

[0188] Bagging (Boostrap aggregating) is an ensemble learning algorithm based on the Bootstrap concept. It randomly extracts training subsets from the original training sample set with replacement, trains multiple regression trees, repeats this process to generate multiple regression trees to form a random forest, and the average of the predicted values ​​of all trees determines the final prediction result. When using the Bagging method to generate training subsets, the probability that each sample in the sample set D is not extracted is (1-1 / N)N, where N is the number of samples in the sample set. When N is large enough, (1-1 / N)N converges to 0.368. Therefore, during each sampling process, 37% of the data in the sample set will not be used to train the random forest regression model. These samples are called out-of-bag (OOB) data. Using OOB data to evaluate the model's fit is called OOB estimation.

[0189] The steps to build a random forest regression model are as follows:

[0190] 1) Use the Bootstrap method to randomly select n (n <N)个样本生成m个训练子集;

[0191] 2) Use the training subset to train the regression tree. Randomly select a portion of sample features from all sample features on the node, divide the regression tree into left and right subtrees based on the minimum mean square error, and recursively build the tree until the termination condition is met.

[0192] 3) Repeat the above steps to form a random forest with multiple regression trees;

[0193] 4) Input the test sample into the random forest regression model, take the average of all regression tree prediction values ​​as the final prediction result, and compare it with the actual value to evaluate the model fitting effect.

[0194] The particle swarm optimization algorithm achieves global search through collaboration and competition between individuals. The system is initialized with a set of random solutions, called particles. The optimization is achieved by the particles flying through the search space, which is mathematically called iteration. It does not use the crossover and mutation operators of genetic algorithms. Instead, particles search the solution space by following the optimal particle. The basic principle is as follows:

[0195] In a search space with D variables, the position and velocity of a particle can be expressed as:

[0196] x i (t) = [x i,1 (t),x i,2 (t),…,x i,D (t)], i = 1, 2, 3, ... N;

[0197] v i (t)=[v i,1(t),v i,2 (t),…,v i,D (t)], i = 1, 2, 3, ... N;

[0198] Where N is the total number of particles, x i,D (t) and v i,D (t) are the position and velocity of the D-th dimension variable in the i-th particle at the t-th iteration.

[0199] When the iterative optimization reaches the tth time, the optimal position and variable state of the current population are pbest and pbest respectively. i (t) and P i,D (t), the global optimal position is gbest i (t) indicates that the optimal state of each variable in the particle is g i,D (t), specifically expressed as:

[0200] pbest i (t)=[P i,1 (t),P i,2 (t),…,P i,D (t)],…i=1,2,3,…,N;

[0201] gbest i (t)=[g i,1 (t),g i,2 (t),…,g i,D (t)],…i=1,2,3,…,N;

[0202] The update of particle position in the multi-objective particle swarm algorithm is mainly based on the update of velocity, and the velocity is mainly based on pbest i (t) and gbest i (t) is updated, and the specific update expression for the t+1th iteration is:

[0203] v i (t+1)=ω(t)·v i (t)+c1·rand·(pbest i (t)-x i (t))+c2·rand()·(gbest i (t)-x i (t));

[0204] x i (t+1)=x i (t)+v i (t+1);

[0205] Where w is the inertia weight value, rand is a random number between 0 and 1, and c1 and c2 are both learning factors.

[0206] The speed expression can be divided into three parts. The first part is the previous speed of the particle. The second part is the "cognitive" part, which represents the particle's own thinking. The third part is the "social" part, which represents the information sharing and mutual cooperation between particles.

[0207] The second step, the main calculation steps of the PSO algorithm are as follows:

[0208] 1) Initialize the particle swarm, set the acceleration constants c1, c2, the maximum evolutionary generation, and set the current evolutionary generation to 1, and randomly generate the positions and velocities of m particles in the defined space;

[0209] Evaluate the population and calculate the fitness value of each particle in each dimension of space;

[0210] 2) Compare the particle's fitness value with its own optimal value pbest. If the current value is better than pbest, set pbest as the current value, that is, use the current position to update the historical best position;

[0211] 3) Compare the particle fitness value with the population optimal value. If the current value is better than gbest, use the current position to update the global optimal position;

[0212] Update the historical position and speed according to the formula to generate a new population;

[0213] 4) Check the end condition. If it is met, the optimization ends; otherwise, go to step 2. The end condition is usually that the optimization reaches the maximum evolutionary number or the evaluation value is less than the given precision.

[0214] Objective function:

[0215] To achieve automatic code generation, the present invention uses tensor expressions to specify tensor operators. Let ε be the tensor expression space. Tensor expressions retain many underlying implementation details, such as loop order and memory range.

[0216] Step 3: Cost generation module based on random forest:

[0217] A random forest prediction method based on machine learning theory is used to predict the hardware runtime cost of generating low-level code. The random forest prediction method extracts domain-specific features from a given low-level abstract syntax tree. These features include loop structure information (e.g., memory access counts and data reuse rates) and common annotations (e.g., vectorization, unrolling, and thread binding).

[0218] For a given data set D = {(e i ,s i ,ci )} Training statistical cost model, taking into account that in the selection process we only care about the relative order of program running time, rather than their absolute value, the expression of the training objective function is:

[0219]

[0220] Where c1 and c2 are acceleration constants, is the hardware runtime cost, x i 、x j There are two instance samples, which are used to compare their relative scores.

[0221] Step 4: Optimization module based on particle swarm optimization algorithm:

[0222] Since the optimization space is too large, the optimization module of the present invention uses the particle swarm optimization algorithm to generate the cost module to predict the hardware running time cost. As the objective function to control the search loop, the specific process is shown in the algorithm pseudo code Table 1. In each iteration, it must predict the hardware runtime cost based on the cost generation module A batch of candidate programs are selected and the actual hardware runtime cost f(x) is queried on the actual hardware.

[0223] Table 1: Optimization modules based on particle swarm optimization algorithm

[0224]

[0225] Specifically, the present invention uses batch parallel Markov chains to improve the prediction throughput of statistical cost models. The present invention selects a batch of candidates with the best performance and runs them on real hardware. The collected performance data is used to update The present invention utilizes the state of the Markov chain in The update remains unchanged. We use greedy selection (b = 0.05) to randomly select candidates to ensure exploration. At the same time, we use diversity-aware exploration to consider both quality and diversity when selecting b candidate hardware for evaluation. Assume that the planned configuration s can be decomposed into m components s = [s1, s2, ... s m ], then the expression for maximizing the objective to select the candidate set S from the λb candidate set is:

[0226]

[0227] Where the first term prioritizes candidates with low runtime cost, the second term calculates the number of different configuration components covered by S, L(S) is a submodule function, and the greedy algorithm is applied to obtain an approximate solution. s is the planned configuration, m is the number of configuration components, and s jis the jth component, α is the regularization coefficient, balancing the importance of the two parts, and g(e, s) is a feature constructor.

[0228] Data optimization:

[0229] Data optimization is a secondary optimization of the model data for the hardware platform based on the optimal data layout required by the AI ​​chip hardware platform used in this invention. The main optimizations include:

[0230] 1) Align data according to computational requirements to speed up memory access. Deep learning processor vectorized operations require data alignment, and aligned access can speed up memory access.

[0231] 2) Transform the model data according to the computational requirements to improve memory locality. Algorithms like convolution require interpreting data as multidimensional arrays. The order of multidimensional arrays affects the number of memory jumps. To minimize memory jumps, transform the data.

[0232] 3) Precision tuning to improve computing speed. Deep learning processors often support low-precision operations, such as 16-bit and 8-bit quantization. Different precisions have different computing performance. Therefore, during compilation, the model data can be optimized for precision to improve performance.

[0233] The working principle of the data optimizer is as follows Figure 9 As shown in the figure, first perform the three optimizations mentioned above on the model data according to the data layout requirements of the hardware platform code; then package the hardware platform code and the optimized model data to generate a deployable model.

[0234] This approach primarily achieves precision tuning and improves operating speed through neural network model quantization. First, the model is quantized based on chip conditions. Second, deep neural network model compression technology is used to significantly compress the model size, significantly reduce the number of model parameters and computational complexity, and accelerate model inference, while minimizing accuracy loss. This facilitates model deployment in smart edge devices with limited computing resources, enabling real-time monitoring of power operations sites. This effectively improves the quality of monitoring by on-site operators and reduces safety risks.

[0235] Due to the performance limitations of the computing hardware used in the present invention, the size of the network model and the computing requirements must also be optimized accordingly to adapt to the use of the equipment. The present invention first quantifies the model.

[0236] Neural network quantization requires converting common operations (convolution, matrix multiplication, activation functions, pooling, concatenation, etc.) into equivalent 8-bit integer versions. Quantization and dequantization are then performed before and after each operation. Quantization converts the input from floating-point numbers (floats) to 8-bit integers (uint8), while dequantization converts the output back to floating-point numbers. Quantization is essentially an operation that reduces the number of bits in a value, which represents a numerical representation. Currently, trained neural network weights are stored as 32-bit floating-point numbers. However, neural networks, with millions or more weights connecting neurons, occupy a significant amount of storage space. Running these models on a PC presents no significant challenges due to its powerful computing performance and storage resources. However, running them on an embedded system is impractical, as PC computing performance and storage resources are often dozens of times greater. However, simply rounding or other methods of converting the data to integers in one step can have a devastating impact on model accuracy. It is hoped that a large number of parameter weights and feature maps can be represented using 8-bit integer data format without sacrificing model accuracy. Even using lower bit widths such as 4, 2, or 1 bit can achieve performance similar to that of floating-point models.

[0237] The most significant advantage of quantization is its ability to significantly reduce bandwidth and storage space, as shown in the table. For example, using 8-bit integers for parameter weights and feature maps can reduce bandwidth requirements by a factor of four compared to models using 32-bit floating-point numbers. Furthermore, compared to floating-point numbers, integers are faster and consume less power and require less memory.

[0238] Table 2: Comparison of power consumption and storage space between 8-bit integer and 32-bit floating-point operations

[0239]

[0240] Taking the Relu activation function as an example, the Relu operation before quantization is as follows Figure 10 As shown (in Figure 10 In the example, Input is input, Relu is Relu activation function, and Output is output). The Relu operation after network quantization is as follows: Figure 11As shown, Input (float) is the neural network feature value in floating-point format and serves as input; Min / Max is the minimum / maximum value, which is used to calculate the dynamic range of the input data, Quantize is used to convert floating-point data (float) to integer representation, Eight Bit is eight-bit integer data, indicating that it is an 8-bit integer after quantization, QuantizedRelu is the quantized ReLU activation function, and ReLU activation is applied in the integer space: all negative numbers are set to zero, and non-negative numbers are retained. Unlike ordinary ReLU, it operates within the quantized numerical range, Dequantize is used to remap 8-bit integer values ​​back to floating-point values, restoring them to approximate original features, and Output (float) is the output. For convolutional neural network models, there are two types of parameters that need to be quantized, one is the weight of the model, and the other is the activation value of the model, that is, the feature map. As shown Figure 12 As shown in the figure, if the weight range of a layer is [a, b], the quantization method is to consider the minimum weight value as a and the maximum weight value as b. The weights are fine-tuned through training, mapping the weight range to [0, 255], thus converting them into 8-bit integers. As for the activation value, its value depends largely on the network input. Assuming that the activation value range during the retraining phase is still [a, b], during training, the EMA (Exponential Moving Average) with a smoothing parameter close to 1 is used to update the activation value. The purpose of the EMA is to slow down the trend when the activation value range fluctuates drastically.

[0241] According to another embodiment of the present invention, Figure 2 As shown, a lightweight image recognition system for power plant safety is also provided, which includes:

[0242] Illegal behavior identification module 1 is used to obtain real-time video data of the power plant, build a multi-scale adaptive perception image anomaly detection model, and use the image anomaly detection model to identify illegal behaviors of power plant operators in the real-time video data of the power plant;

[0243] Safety distance calculation module 2 is used to obtain binocular images of power equipment, build a multi-stage binocular stereo matching model to match feature points in the binocular images of power equipment, and calculate the safety distance between power plant operators and live objects based on the matching results;

[0244] Safety hazard monitoring module 3 is used to build an edge device inference model for lightweight knowledge distillation based on the violations and safety distances of power plant operators, and use the edge device inference model to monitor power plant safety hazards in real time.

[0245] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A lightweight image recognition method for power plant safety, characterized in that: The method includes: Acquire real-time video data from power plants, build a multi-scale adaptive perception image anomaly detection model, and use the image anomaly detection model to identify violations by power plant operators in real-time video data from power plants. Acquire binocular images of power equipment, build a multi-stage binocular stereo matching model to match feature points in the binocular images of power equipment, and calculate the safe distance between power plant operators and live objects based on the matching results; Based on the violations and safety distances of power plant operators, a lightweight knowledge distillation edge device inference model is constructed, and the edge device inference model is used to monitor power plant safety hazards in real time.

2. The lightweight image recognition method for power plant safety according to claim 1, characterized in that: The method of constructing a multi-scale adaptive perception image anomaly detection model and using the image anomaly detection model to identify violations by power plant operators in real-time video data of the power plant includes: A bilinear convolutional neural network is used to mine the visual features and spatial topological features of real-time video data from power plants, and the visual and spatial topological features are fused to identify reference power operation scenarios. Based on a reference power operation scenario, the temporal context detector in the transition-aware context network is used to encode context information, and the transition-aware classifier is used to segment the real-time video data of the power plant into behavioral video segments and transition video segments. Combining a multi-scale feature pyramid network with an adaptive attention mechanism to identify abnormal behaviors of power plant operators in transitional video clips; Using time series models to identify abnormal posture behaviors of power plant workers in behavioral video clips.

3. The lightweight image recognition method for power plant safety according to claim 2, characterized in that: The method of combining the multi-scale feature pyramid network with the adaptive attention mechanism to identify abnormal operating behaviors of power plant operators in transitional video clips includes: Extract transition image sequences from transition video clips and construct a multi-scale feature pyramid network based on the transition image sequences; Calculate the sparse optical flow field of the transition image in the top layer of the multi-scale feature pyramid network, and perform inter-frame difference processing on adjacent transition images to obtain the initial optical flow vector of adjacent transition images; Calculate the optical flow vectors of the remaining transition images based on the initial transport optical flow vector, and fuse them with the initial optical flow vector to obtain a fused optical flow vector; The fused optical flow vector is compared with the preset threshold. If the fused optical flow vector is greater than the preset threshold, the optical flow vector of the maximum transition image is iteratively selected as the outlier and modified to the mean optical flow vector. Otherwise, the fused optical flow vector is passed to the next layer until it is passed to the bottom layer of the multi-scale feature pyramid network. The fused optical flow vector of each layer is converted into a depth map, and the depth map is channel-stitched based on the attention mechanism. The transition-aware context network is used to segment abnormal operation behaviors from the spliced ​​channels.

4. The lightweight image recognition method for power plant safety according to claim 1, characterized in that: The method of acquiring binocular images of power equipment, constructing a multi-stage binocular stereo matching model to match feature points in the binocular images of power equipment, and calculating the safe distance between power plant operators and charged objects based on the matching results includes: Use binocular cameras to obtain binocular images of power equipment, and build a multi-stage binocular stereo matching model to match the binocular images to obtain disparity images; The depth value of the power equipment point in the disparity image is obtained by using triangulation method, and the three-dimensional coordinate points of the disparity image are reconstructed by using the improved semi-global block matching algorithm; Based on the abnormal posture behavior of power plant operators, the spatial distance between the reconstructed three-dimensional coordinate points and the feature points of the target object is calculated, and the safe distance between power plant operators and live objects is delineated based on the spatial distance; An electronic fence is drawn in the charged area based on the safety distance, and the power plant operators are monitored to see if they are within the electronic fence area. If the power plant operators exceed the electronic fence area, voice prompts are given to the power plant operators to re-enter the electronic fence area.

5. The lightweight image recognition method for power plant safety according to claim 4, characterized in that: The calculation of the spatial distance between the reconstructed three-dimensional coordinate point and the feature point of the target object based on the abnormal posture behavior of the power plant operator and the safe distance between the power plant operator and the charged object according to the spatial distance includes: The abnormal posture behaviors of power plant operators are decomposed into atomic action units. A preset-sized image region is captured with the moving target as the center point. The image region is divided into small regions of preset sizes. Several prior anchor frames of different sizes are set at the center point of each small region. A power plant operator recognition model is established, and prior anchor boxes of different sizes are assigned to target areas of different scales according to their sizes. Calculate the target category confidence and target position confidence of each target area, and use the confidence calculation result as the prediction result of the prior anchor box. Modify the confidence according to the existence of the target and target category inside the prior anchor box; Based on the prediction results of the modified prior anchor frame, the non-maximum suppression algorithm is used to remove duplicate prediction results as the output of the power plant operator recognition model; Based on the output results of the power plant operator recognition model, the target object feature points are extracted, and the three-dimensional coordinates of the target object feature points and the surface centers of the charged objects are obtained to calculate the safe distance between the power plant operators and the charged objects.

6. The lightweight image recognition method for power plant safety according to claim 5, characterized in that: The correction of the confidence according to the existence of the target and the target category in the prior anchor frame includes: A dual causal path is established based on the existence of targets, target categories and predefined environmental interference factors within the prior anchor frame, and an intervening variable is introduced to test the dual causal path; A main branch encoder and an auxiliary branch encoder are constructed separately to calculate the mutual information of the dual causal paths. Based on the mutual information, causal intervention is performed inside the prior anchor box through the dual causal paths to evaluate the impact of environmental interference factors on confidence. Based on the degree of influence of environmental interference factors on confidence, the confidence is corrected by introducing interference compensation and category prior probability, and the corrected confidence is reversely injected into the main branch encoder so that the output of the main branch encoder is semantically consistent with the corrected confidence.

7. The lightweight image recognition method for power plant safety according to claim 1, characterized in that: The aforementioned method combines the illegal behaviors and safety distances of power plant operators to build an edge device inference model with lightweight knowledge distillation, and uses the edge device inference model to monitor power plant safety hazards in real time, including: The illegal behaviors and safety distances of power plant operators are analyzed to build an edge device inference model based on lightweight knowledge distillation of convolutional networks; The edge device inference model is compressed through operator optimization technology, tensor optimization technology, and data optimization technology. Identify power plant electrical hazards based on model compression results, and optimize edge device inference model performance by combining self-supervised contrastive learning and meta-learning to adapt to new types of power plant electrical hazards. The predefined teacher model is deployed in the cloud for training, and knowledge is transferred to the lightweight student model through a dynamic adaptation strategy to optimize the reasoning capability of the edge device inference model.

8. The lightweight image recognition method for power plant safety according to claim 7, characterized in that: The process of deploying a predefined teacher model in the cloud for training and transferring knowledge to a lightweight student model through a dynamic adaptation strategy to optimize the reasoning capability of the edge device inference model includes: The real-time video data of the power plant and binocular images of power equipment are integrated to obtain a sample set. The sample set is divided into labeled data and input into the teacher model deployed in the cloud and the local lightweight student model respectively. The teacher model and the local lightweight student model output predicted labels respectively, and the predicted samples with clear labels are introduced into the classification loss function to calculate the label difference between the predicted labels of the student model and the true labels; Adjust the weight of the teacher model based on the label difference of the student model, transfer knowledge to the student model through a dynamic adaptation strategy based on the optimized teacher model, and adjust the weight of the teacher model in reverse; The teacher model weight update and knowledge distillation are iteratively executed until the iteration conditions are met, realizing the knowledge exchange between the teacher model and the student model to optimize the reasoning capability of the edge device inference model.

9. The lightweight image recognition method for power plant safety according to claim 1, characterized in that: The confidence correction formula is: Where, Pr(Class i ) represents the probability that the sample belongs to category i; Pr(Class i |Object) indicates the possibility of predicting the target data as the i-th category; Pr(Object) indicates the possibility of the existence of the target in the region; It represents the intersection-over-union ratio between the predicted target and the actual target.

10. A lightweight image recognition system for power plant safety, characterized by: The lightweight image recognition method for power plant safety according to any one of claims 1 to 9 is implemented, the system comprising: The violation identification module is used to obtain real-time video data from the power plant, build a multi-scale adaptive perception image anomaly detection model, and use the image anomaly detection model to identify violations by power plant operators in the real-time video data of the power plant; The safety distance calculation module is used to obtain binocular images of power equipment, build a multi-stage binocular stereo matching model to match feature points in the binocular images of power equipment, and calculate the safety distance between power plant operators and live objects based on the matching results; The safety hazard monitoring module is used to build an edge device inference model for lightweight knowledge distillation based on the violations and safety distances of power plant operators, and use the edge device inference model to monitor power plant safety hazards in real time.

Images