Cloud-side collaborative power equipment fault video detection method
Through the cloud-edge collaborative power equipment fault video detection method, the target tracking of attention mechanism and cloud-side ViT model are used to solve the problem of traditional fault detection inefficiency, and efficient and accurate fault detection and rapid response are achieved.
Patent Information
- Application Number
- CN202411580567.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-07
- Publication Date
- 2025-02-28
AI Technical Summary
Traditional power equipment fault detection methods are inefficient and difficult to deal with sudden failures. The storage and transmission of video data are high, and information loss or detection delay may occur.
The cloud-edge collaborative power equipment fault video detection method is adopted, and the keyframes are dynamically selected, abnormal frame images are cropped and compressed, and fault detection and feedback are carried out in combination with the cloud-based ViT model.
It improves the real-time and accuracy of fault detection, effectively utilizes computing resources at the cloud and edge, reduces latency and bandwidth consumption, and can better cope with the problems of insufficient local computing power and limited upload bandwidth.
Smart Images
Figure CN119540166A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of power equipment fault detection, and specifically relates to a cloud-edge collaborative power equipment fault video detection method. Background Art
[0002] In modern power systems, stable operation of equipment is crucial. With the rapid development of smart grids, monitoring and fault diagnosis technologies for power equipment are also evolving. Traditional power equipment fault detection methods rely on manual inspections or regular maintenance, which is not only inefficient but also difficult to deal with the risks brought by sudden failures. Therefore, how to achieve real-time and accurate fault detection has become a research hotspot.
[0003] In recent years, the application of video surveillance technology in power equipment monitoring has gradually become popular. By installing high-definition cameras, the operating status of power equipment can be captured in real time. However, there are some shortcomings in relying solely on video surveillance for fault detection. First, the amount of video data is large, and the storage and transmission costs are high. Especially when the network bandwidth is limited, transmission delays may cause information loss or detection delays. Secondly, important information in the video is often obscured by a large amount of redundant data. How to effectively extract key frames related to the fault has become a problem that needs to be solved urgently.
[0004] In order to overcome the above challenges, academia and industry have begun to explore target detection and tracking technologies based on deep learning. In particular, the introduction of the RoI (Region of Interest) attention mechanism has greatly improved the accuracy and efficiency of target tracking. This mechanism can automatically identify important areas in an image and focus computing resources on these areas, thereby improving the sensitivity and accuracy of fault detection.
[0005] In order to further improve the effect of fault detection, cloud-edge collaborative computing has become an emerging solution. Cloud computing provides powerful computing resources, which is suitable for complex deep learning model training and large-scale data processing, while edge computing can perform real-time analysis close to the data source, thereby reducing latency and bandwidth consumption. By effectively combining the two, fast and real-time fault detection can be achieved. Based on the above facts, the present invention hopes to provide a cloud-edge collaborative power equipment fault video detection method. Summary of the invention
[0006] The purpose of the present invention is to provide a cloud-edge collaborative power equipment fault video detection method to solve the problems raised in the above background technology.
[0007] To achieve the above object, the present invention provides the following technical solutions:
[0008] A cloud-edge collaborative power equipment fault video detection method comprises the following steps:
[0009] S1. Power equipment target tracking based on attention mechanism, the target tracking algorithm includes target recognition, estimation of movement size change of objects in the area of interest, and dynamic selection of key frames;
[0010] S2. Determine the abnormal situation of the key frame and distinguish whether the power equipment in the key frame fails;
[0011] S3. Crop and compress the abnormal frame image and upload the abnormal frame information;
[0012] S4. Refined detection and feedback of power equipment faults in the cloud, building a ViT model to detect the type of power equipment faults and provide feedback on the treatment plan.
[0013] Preferably, in S1, the region of interest is a region in an image or video frame that is marked as important and needs further processing; the region of interest used is a combination of multiple rectangular regions, and the region of interest is used to selectively process the image content and focus on a specific target in image analysis; for the first frame of the tracking sequence, the initial region of interest comes from salient target detection, and the saliency map and the self-attention mechanism are expressed as:
[0014] X(x)=s(p(x))
[0015]
[0016] where X(x) is the saliency map, p(x) is the input image, s is the mapping function from image features to saliency scores, C is the query, J is the key, Y is the value, and w k is the dimension of the key, Z is the self-attention weight;
[0017] For subsequent frames, the initial region of interest comes from the encoding output of the previous frame; the tracking module is applied to N consecutive frames, where N is preset or dynamically adjusted according to the encoding process; the initial region of interest is combined with the motion vector of the frame being encoded, which represents the displacement from the reference frame to the current frame, and the formula is expressed as
[0018] Vec=(i',j')-(i,j)
[0019] Where (i', j') is the position of the pixel in the current frame, and (i, j) is the corresponding pixel position in the reference frame;
[0020] Accurately update the position and size of the region of interest for the next frame. The motion vector calculated by the encoder represents the displacement of pixel blocks between frames.
[0021] Preferably, a lightweight deep learning-based ROI tracking module proposed in S1, which uses the full-frame motion vector and initial ROI coordinates of the previous frame as input; it outputs the estimated ROI position and size of the current frame; by improving the motion vector-based tracking using deep learning, the need for various other refinement steps such as motion vector filters or hand-crafted ROI size estimation is eliminated;
[0022] The motion data generated by the encoder is formatted as a three-channel input: a horizontal motion vector, a vertical motion vector, and a flag indicating intra or intra prediction; intra blocks lack motion data, so their vectors are set to zero; motion vectors that partially or completely overlap the region of interest on the frame are first selected; the selected motion vectors are then processed by a multi-layer neural network designed to infer the displacement and size changes of the region of interest.
[0023] In the region of interest update phase, the initial region of interest coordinates from the saliency map are used as the starting point, and the network output is applied to incrementally adjust the region of interest coordinates frame by frame; the updated region of interest coordinates are also used to select the motion vector for the next frame to ensure continuous tracking before starting a new saliency map detection cycle; for cases involving multiple regions of interest, each region of interest is processed independently.
[0024] Preferably, in S1, the multi-layer neural network architecture consists of a resizing layer, two convolutional layers and a fully connected layer; the ReLU function is used as the activation function of the convolutional layer output.
[0025] Preferably, in S2, a lightweight deep learning network is used to perform fault detection on power equipment locally, and a model pruning method is used to optimize the model. The specific steps are as follows:
[0026] In the first step, a batch normalization layer is added after the convolutional layer, a sparse factor is added to the batch normalization layer network, and the batch normalization layer is sparsely processed through training; the weights of all batch normalization layers in the sparse training model are sorted by a statistical sorting method, and ordered weight thresholds are obtained by obtaining a specified number of reserved batch normalization layers;
[0027] The second step is to traverse the weights of the batch normalization layer in the model and obtain the mask value of each layer. In the pruning operation, a new model structure is constructed after the mask of each layer, the index of the non-zero value of the weight mask of the batch normalization layer is obtained, the weights and bias of each channel in the original convolutional layer are obtained, and the batch normalization layer and linear layer corresponding to the non-zero index are assigned to each layer of the new model; the formula of the objective function Loss is:
[0028]
[0029] Among them, l is the basic loss, Q is the loss weight, λ is the L1 regularization coefficient, g is the L1 regularization function, and γ is the affine transformation parameter;
[0030] In the third step, in addition to the basic loss, the L1 regularization term is introduced to obtain the sparsity of the batch normalization layer, where the LI constraint function g is only applied to the γ coefficient of the batch normalization layer; when backpropagating and updating the gradient, the derivative of the γ coefficient needs to add the product of the coefficient sign and the penalty coefficient;
[0031] In the fourth step, after sparse training of the network, a network with a scaling factor close to zero is obtained; through proportional pruning, the input connected to the channel is removed; after repeated training, the lightweight compression process of the entire network is completed.
[0032] Preferably, the image in S3 is cropped according to the area of interest, and the cropped image only retains the part related to the power equipment fault, thereby reducing the size of the image; the compressed image file and its related information are encapsulated into a data packet, and the data packet is sent to the designated server endpoint through a request; the image file and metadata are attached to the request so that the server can correctly parse and store the data; and the server's response is received to confirm whether the upload is successful.
[0033] Preferably, in S4, a hybrid neural network is built for specific fault detection of power equipment, and the network consists of a deep neural network and a Transformer network. The deep neural network part first decomposes the locally uploaded abnormal frames into image blocks to adapt to inputs of different sizes and resolutions; then, the pre-trained model processes each image block, including convolutional neural network layers and balanced adjustments of depth, width and resolution to extract spatial features and reduce the computational burden; the Transformer network part transforms or reconstructs the extracted frame feature sequence as needed; then, the fault information in the uploaded image sequence is captured through linear embedding, QKV calculation and self-attention mechanism.
[0034] Compared with the prior art, the present invention has the following beneficial effects:
[0035] (1) The present invention provides a cloud-edge collaborative video detection method for power equipment faults. The cloud-edge collaboration mentioned in the scheme refers to the comprehensive use of limited computing resources at the location of the power equipment and computing resources of online large servers to complete the judgment and detection of power equipment faults;
[0036] (2) The cloud-edge collaborative power equipment fault video detection method provided by the present invention not only improves the real-time and accuracy of fault detection, but also effectively utilizes the computing resources of the cloud and the edge; compared with traditional methods, this technology can better deal with the problems of insufficient local computing power and limited upload bandwidth, and provides a new idea for real-time detection of power equipment faults. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a flow chart of a cloud-edge collaborative power equipment fault video detection method provided in an embodiment of the present application;
[0038] Figure 2 is a flowchart of the region of interest tracking provided by an embodiment of the present application;
[0039] Figure 3 is a model pruning flow chart provided in an embodiment of the present application;
[0040] Figure 4 Schematic diagram of a cloud-based detection hybrid neural network provided in an embodiment of the present application. DETAILED DESCRIPTION
[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0042] The present invention discloses a cloud-edge collaborative power equipment fault video detection method, firstly, power equipment target tracking based on the interest region attention mechanism, the target tracking algorithm includes target recognition, estimation of movement and size change of objects in the interest region, and dynamic selection of key frames;
[0043] Determine abnormal conditions of key frames and distinguish whether power equipment in key frames fails;
[0044] Crop and compress abnormal frame images, and upload abnormal frame information;
[0045] Refined detection and feedback of power equipment faults in the cloud, building a ViT model to detect power equipment fault types and provide feedback on treatment plans.
[0046] The region of interest is an area in an image or video frame that is marked as important and needs further processing. The region of interest used is a combination of multiple rectangular regions. It is used to selectively process image content and focus on specific targets in image analysis. For the first frame of the tracking sequence, the initial region of interest comes from salient target detection, the saliency map, and the self-attention mechanism are expressed as:
[0047] X(x)=s(p(x))
[0048]
[0049] where X(x) is the saliency map, p(x) is the input image, s is the mapping function from image features to saliency scores, C is the query, J is the key, Y is the value, and w k is the dimension of the key and Z is the self-attention weight.
[0050] For subsequent frames, the initial region of interest comes from the encoding output of the previous frame, such as Figure 2 The tracking module is applied to N consecutive frames, where N is preset or dynamically adjusted according to the encoding process. The initial region of interest is combined with the motion vector of the frame being encoded. The motion vector represents the displacement from the reference frame to the current frame, and the formula is expressed as:
[0051] Vec=(i',j')-(i,j)
[0052] Where (i', j') is the position of the pixel in the current frame, and (i, j) is the corresponding pixel position in the reference frame.
[0053] Accurately update the position and size of the region of interest for the next frame. The motion vector calculated by the encoder represents the displacement of pixel blocks between frames.
[0054] A lightweight deep learning based ROI tracking module that uses the full frame motion vectors and initial ROI coordinates of the previous frame as input. It outputs the estimated ROI position and size of the current frame. By using deep learning to improve motion vector based tracking, the need for various other refinement steps such as motion vector filters or handcrafted ROI size estimates is eliminated.
[0055] The motion data generated by the encoder is formatted as a three-channel input: a horizontal motion vector, a vertical motion vector, and a flag indicating intra or intra prediction. Intra blocks lack motion data, so their vectors are set to zero. Motion vectors that partially or completely overlap the region of interest on the frame are first selected. The selected motion vectors are then processed by a multi-layer neural network designed to infer displacement and size changes of the region of interest.
[0056] In the ROI update phase, the initial ROI coordinates from the saliency map are used as a starting point, and the network output is applied to incrementally adjust the ROI coordinates on a frame-by-frame basis. The updated ROI coordinates are also used to select the motion vector for the next frame to ensure continuous tracking before starting a new saliency map detection cycle. For cases involving multiple ROIs, each ROI is processed independently.
[0057] The multi-layer neural network architecture consists of a resizing layer, two convolutional layers, and a fully connected layer. The ReLU function is used as the activation function for the output of the convolutional layer.
[0058] A lightweight deep learning network is used to detect faults in power equipment locally, and the model is optimized by model pruning, such as Figure 3 As shown, the specific steps are as follows:
[0059] Step 1: Add a batch normalization layer after the convolutional layer, add a sparse factor to the batch normalization layer network, and perform sparse processing on the batch normalization layer through training. Use a statistical sorting method to sort the weights of all batch normalization layers in the sparse training model, and obtain an ordered weight threshold by obtaining a specified number of reserved batch normalization layers.
[0060] Step 2: Traverse the weights of the batch normalization layer in the model and obtain the mask value of each layer. In the pruning operation, a new model structure is constructed after the mask of each layer, and the index of the non-zero value of the weight mask of the batch normalization layer is obtained. The weights and biases of each channel in the original convolutional layer are obtained, and the batch normalization layer and linear layer corresponding to the non-zero index are assigned to each layer of the new model. The formula of the objective function Loss is:
[0061]
[0062] Where l is the basic loss, Q is the loss weight, λ is the L1 regularization coefficient, g is the L1 regularization function, and γ is the affine transformation parameter.
[0063] Step 3: In addition to the basic loss, the L1 regularization term is introduced to obtain the sparsity of the batch normalization layer, where the LI constraint function g is only applied to the γ coefficient of the batch normalization layer. Therefore, when backpropagating and updating the gradient, the derivative of the γ coefficient needs to add the product of the coefficient sign and the penalty coefficient
[0064] Step 4: After sparse training of the network, a network with a scaling factor close to zero is obtained. Through proportional pruning, the input connected to the channel is removed. After repeated training, the lightweight compression process of the entire network is completed.
[0065] The image is cropped based on the region of interest. The cropped image only retains the part related to the power equipment fault, thereby reducing the image size. The compressed image file and its related information (such as timestamp, detection type, location information, etc.) are encapsulated into a data packet, and the data packet is sent to the specified server endpoint through a request. The image file and metadata are attached to the request so that the server can correctly parse and store the data. The server's response is received to confirm whether the upload is successful.
[0066] A hybrid neural network is built on the cloud server for specific fault detection of power equipment, such as Figure 4 As shown in the figure, the network consists of a deep neural network and a Transformer network. The deep neural network part first decomposes the locally uploaded abnormal frames into image blocks to adapt to inputs of different sizes and resolutions. Then, the pre-trained model processes each image block, including convolutional neural network layers and balanced adjustments of depth, width, and resolution to extract spatial features and reduce the computational burden. The Transformer network part transforms or reconstructs the extracted frame feature sequence as needed. Then, the fault information in the uploaded image sequence is captured through linear embedding, QKV calculation, and self-attention mechanism.
[0067] The cloud-edge collaboration mentioned in this method refers to the comprehensive use of the limited computing resources at the location of the power equipment and the computing resources of the online large-scale servers to complete the judgment and detection of power equipment failures. Among them, one or more processors are included to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component can also be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the above method.
[0068] The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0069] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.
[0070] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.
[0071] The communication component is configured to facilitate wired or wireless communication between the computing device and other devices. The computing device includes a physical device and an elastic computing host provided by the cloud computing platform. In this case, the computing device may refer to a cloud server, and the processing components, storage components, etc. may be basic server resources rented or purchased from the cloud computing platform.
[0072] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1A cloud-edge collaborative method for video detection of power equipment faults in the illustrated embodiment.
[0073] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, systems and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. The system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art can understand and implement it without paying any creative labor.
[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0075] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A cloud-edge collaborative power equipment fault video detection method, characterized in that: The following steps are involved: S1. Power equipment target tracking based on attention mechanism, the target tracking algorithm includes target recognition, estimation of movement size change of objects in the area of interest, and dynamic selection of key frames; S2. Determine the abnormal situation of the key frame and distinguish whether the power equipment in the key frame fails; S3. Crop and compress the abnormal frame image and upload the abnormal frame information; S4. Refined detection and feedback of power equipment faults in the cloud, building a ViT model to detect the type of power equipment faults and provide feedback on the treatment plan.
2. According to claim 1, a cloud-edge collaborative power equipment fault video detection method is characterized by: In S1, the region of interest is the area in the image or video frame that is marked as important and needs further processing; the region of interest used is a combination of multiple rectangular regions, and the region of interest is used to selectively process the image content and focus on specific targets in image analysis; for the first frame of the tracking sequence, the initial region of interest comes from salient target detection, and the saliency map and self-attention mechanism are expressed as: X(x)=s(p(x)) where X(x) is the saliency map, p(x) is the input image, s is the mapping function from image features to saliency scores, C is the query, J is the key, Y is the value, and w k is the dimension of the key, Z is the self-attention weight; For subsequent frames, the initial region of interest comes from the encoding output of the previous frame; the tracking module is applied to N consecutive frames, where N is preset or dynamically adjusted according to the encoding process; the initial region of interest is combined with the motion vector of the frame being encoded, which represents the displacement from the reference frame to the current frame, and the formula is expressed as 3ec=(i',j')-(i,j) Where (i', j') is the position of the pixel in the current frame, and (i, j) is the corresponding pixel position in the reference frame; Accurately update the position and size of the region of interest for the next frame. The motion vector calculated by the encoder represents the displacement of pixel blocks between frames.
3. According to the cloud-edge collaborative power equipment fault video detection method of claim 1, it is characterized by: The proposed deep learning-based lightweight ROI tracking module in S1 uses the full-frame motion vectors and initial ROI coordinates of the previous frame as input; it outputs the estimated ROI position and size of the current frame; by improving motion vector-based tracking using deep learning, it eliminates the need for various other refinement steps such as motion vector filters or hand-crafted ROI size estimates; The motion data generated by the encoder is formatted as a three-channel input: a horizontal motion vector, a vertical motion vector, and a flag indicating intra or intra prediction; intra blocks lack motion data, so their vectors are set to zero; motion vectors that partially or completely overlap the region of interest on the frame are first selected; the selected motion vectors are then processed by a multi-layer neural network designed to infer the displacement and size changes of the region of interest. In the region of interest update phase, the initial region of interest coordinates from the saliency map are used as the starting point, and the network output is applied to incrementally adjust the region of interest coordinates frame by frame; the updated region of interest coordinates are also used to select the motion vector for the next frame to ensure continuous tracking before starting a new saliency map detection cycle; for cases involving multiple regions of interest, each region of interest is processed independently.
4. According to claim 1, a cloud-edge collaborative power equipment fault video detection method is characterized by: In S1, the multi-layer neural network architecture consists of a resizing layer, two convolutional layers and a fully connected layer; the ReLU function is used as the activation function of the convolutional layer output.
5. According to the cloud-edge collaborative power equipment fault video detection method of claim 1, it is characterized in that: In S2, a lightweight deep learning network is used to perform fault detection on power equipment locally, and a model pruning method is used to optimize the model. The specific steps are as follows: In the first step, a batch normalization layer is added after the convolutional layer, a sparse factor is added to the batch normalization layer network, and the batch normalization layer is sparsely processed through training; the weights of all batch normalization layers in the sparse training model are sorted by a statistical sorting method, and ordered weight thresholds are obtained by obtaining a specified number of reserved batch normalization layers; The second step is to traverse the weights of the batch normalization layer in the model and obtain the mask value of each layer. In the pruning operation, a new model structure is constructed after the mask of each layer, the index of the non-zero value of the weight mask of the batch normalization layer is obtained, the weights and biases of each channel in the original convolutional layer are obtained, and the batch normalization layer and linear layer corresponding to the non-zero index are assigned to each layer of the new model; The formula of objective function Loss is: Among them, l is the basic loss, Q is the loss weight, λ is the L1 regularization coefficient, g is the L1 regularization function, and γ is the affine transformation parameter; In the third step, in addition to the basic loss, the L1 regularization term is introduced to obtain the sparsity of the batch normalization layer, where the LI constraint function g is only applied to the γ coefficient of the batch normalization layer; when backpropagating and updating the gradient, the derivative of the γ coefficient needs to add the product of the coefficient sign and the penalty coefficient; In the fourth step, after sparse training of the network, a network with a scaling factor close to zero is obtained; through proportional pruning, the input connected to the channel is removed; after repeated training, the lightweight compression process of the entire network is completed.
6. The cloud-edge collaborative power equipment fault video detection method according to claim 1 is characterized by: The image in S3 is cropped according to the region of interest, and only the portion related to the power equipment fault is retained in the cropped image, thereby reducing the size of the image; the compressed image file and its related information are encapsulated into a data packet, and the data packet is sent to the designated server endpoint through a request; Include the image file and metadata in the request so the server can parse and store the data correctly; Receive a response from the server to confirm whether the upload was successful.
7. The cloud-edge collaborative power equipment fault video detection method according to claim 1 is characterized by: In S4, a hybrid neural network is built for specific fault detection of power equipment. The network consists of a deep neural network and a Transformer network. The deep neural network part first decomposes the locally uploaded abnormal frames into image blocks to adapt to inputs of different sizes and resolutions; then, the pre-trained model processes each image block, including convolutional neural network layers and balanced adjustments of depth, width and resolution to extract spatial features and reduce the computational burden; the Transformer network part transforms or reconstructs the extracted frame feature sequence as needed; then, the fault information in the uploaded image sequence is captured through linear embedding, QKV calculation and self-attention mechanism.