A method and related device for detecting hidden dangers in power distribution network equipment based on large field-of-view stitched video data

Abstract

This invention provides a method and related apparatus for detecting potential hazards in power distribution network equipment based on large field-of-view stitched video data. This method significantly expands the detection field of view by merging video data from different perspectives, and solves the problem of repeated shooting of multiple power distribution network devices at the same location through a large field of view. By combining synchronous timestamps to precisely align the spatial and temporal parameters of multiple video data streams, the continuity and consistency of the stitched video in time and space are ensured, improving the reliability and effectiveness of the detection results. Through precise extraction and matching of image features from multiple video frames, visual distortion and seam marks during the stitching process are reduced, generating high-quality fused video data, providing a clear and continuous visual foundation for subsequent hazard detection. The use of a pre-trained hazard detection model for automatic detection of the fused video data greatly improves detection speed and efficiency, and enhances the objectivity and standardization of the detection process.

Description

Technical Field

[0001] This invention belongs to the field of power inspection technology, specifically relating to a method and related device for detecting hidden dangers in power distribution network equipment based on large field-of-view stitched video data. Background Technology

[0002] The safe and stable operation of power distribution network equipment is a crucial aspect of the normal operation of the power system; therefore, the inspection of power distribution network equipment is essential. With the significant advancements in drone technology, drone-based power inspection methods are becoming widely adopted.

[0003] Currently, commonly used drone inspection technology still relies on manual data collection and hazard identification. Multiple data collections are needed for multiple distribution network devices in the same location, leading to a surge in data volume and significantly increasing the workload for manual identification. Furthermore, drone flight and camera shooting are easily affected by the surrounding environment, often resulting in out-of-focus or shaky images. This necessitates multiple, detailed shots of individual distribution network devices to obtain usable, high-quality images. This not only demands high technical skills from operators but also further increases the inspection workload, thereby reducing the efficiency of power grid inspections.

[0004] Therefore, developing a power distribution network inspection method that can acquire high-quality image data of power distribution network equipment while reducing the workload of inspection personnel, so as to ensure the efficient operation of the inspection of potential hazards in power distribution network equipment, has become a key issue that the power industry urgently needs to solve. Summary of the Invention

[0005] In view of this, the present invention aims to provide a method and related device for detecting hidden dangers in power distribution equipment based on large field-of-view stitched video data. By comprehensively utilizing multi-view video data collected by UAVs and advanced video processing technology, the invention achieves efficient detection of hidden dangers in power distribution equipment based on large field-of-view video data, thereby improving the accuracy of detection and the convenience of operation.

[0006] To achieve the above objectives, the technical solution provided by the present invention is as follows:

[0007] In a first aspect, the present invention provides a method for detecting potential hazards in power distribution network equipment based on large field-of-view stitched video data, comprising the following steps:

[0008] Acquire video data from multiple distribution network devices with synchronized timestamps from different perspectives;

[0009] Based on the perspective differences of video data from multiple distribution network equipment, the spatial and temporal parameters of the video data from multiple distribution network equipment are aligned by combining the synchronization timestamp;

[0010] Based on aligned multi-channel power distribution network equipment video data, features of multi-channel video frame images are extracted and matched to obtain the splicing relationship between multi-channel video frame images.

[0011] Determine the seam between two video frame images to be spliced ​​based on the splicing relationship;

[0012] Based on the splicing relationship and splicing seams of multiple video frame images, the multiple video frame images are fused and spliced ​​to obtain fused video data;

[0013] By using a pre-trained hazard detection model to detect hazards in fused video data, the results of hazard detection for power distribution network equipment are obtained.

[0014] Furthermore, features are extracted from multiple video frame images and matched, including:

[0015] By using neural networks to extract multi-level features from multiple video frame images, coarse feature maps that meet small size requirements and fine feature maps that meet large size requirements are obtained for each video frame image.

[0016] After adding learnable positional encoding to the coarse feature map of each path, it is transformed into a one-dimensional vector. Then, the one-dimensional vector is transformed into an easy-to-match feature representation through the Transformer-based feature matching module to obtain the feature vector.

[0017] Taking any one path as the main path, the similarity matrix between the video frame images of other paths and the video frame images of the main path is calculated using the pixel-wise vector inner product method. The optimal match is calculated using the dual-Softmax method. Some outlier matching pairs are filtered out by the mutual nearest neighbor algorithm to obtain coarse matching point pairs that match the video frame images of multiple paths. The similarity matrix is ​​used to represent the similarity between the feature vectors of two video frame images.

[0018] The coarse matching point pairs are mapped to the corresponding fine feature maps, and the mapped regions are input into the Transformer-based feature matching module. The matching probability of the center feature of the main video frame image and all features of the other matching video frame images is calculated. The feature position with the highest matching probability is taken as the sub-pixel precision matching point position. The sub-pixel precision matching point position represents the stitching relationship between the two matched video frame images.

[0019] Furthermore, the optimal stitching seam is determined using a dynamic programming method with the objective of minimizing image differences. The determination process includes:

[0020] Determine the pixel-wise energy summation function for the two video frames to be stitched together. The energy summation function is used to quantify the degree of image difference in the overlapping area of ​​the two video frames.

[0021] Initialize the seam path, taking the first row of the overlapping area as the minimum energy and path for the first step; iterate based on the minimum energy and path, traversing each row in turn to obtain the position of the minimum energy and path source, and at the same time record the minimum energy and vector reaching each row and each column, as well as the position of the path source reaching each row and each column.

[0022] The splicing seams are traced back from the last row by recording the position matrix of the source, and the splicing seam with the smallest total energy is selected as the best splicing seam.

[0023] Furthermore, the multiple video frame images are fused and stitched together, including:

[0024] Wavelet transform is performed on all video frame images to be stitched, and the Mallat algorithm is used for decomposition to obtain low-frequency and high-frequency components.

[0025] A weighted average fusion method is used to fuse the low-frequency components to obtain the fused low-frequency components;

[0026] The high-frequency components are fused using a method based on maximizing the absolute value of the window coefficient to obtain the fused high-frequency components.

[0027] The fused low-frequency and high-frequency components are subjected to inverse wavelet transform to obtain the fused image.

[0028] Furthermore, based on the perspective differences of the video data from multiple distribution network devices, and in conjunction with the synchronization timestamps, the spatial and temporal parameters of the video data from multiple distribution network devices are aligned, including:

[0029] Time alignment: Select the main path and obtain the video frame data of the main path at a set time. Index the time and video frame data of other paths before and after the set time. Obtain the video frame data of other paths at the set time through linear interpolation.

[0030] Spatial alignment involves geometrically correcting the video data from each camera based on the camera calibration parameters from different perspectives, thereby eliminating the varying degrees of distortion that occur during imaging by each camera.

[0031] Brightness alignment involves converting the images of each video data frame from RGB format to HSV format and then unifying the brightness of the video frames at the same time in each video data stream.

[0032] Coordinate system alignment is performed using the image reference system acquired by the main path as the base reference system. Based on the camera extrinsic parameters between the other paths and the main path, the projection transformation model of the video frame images of the other paths is calculated, and the multi-path video frame data is projected onto the base reference system.

[0033] Furthermore, the hazard detection model is trained using a YOLOv8 network. This pre-trained model is used to detect hazards in the fused video data, including:

[0034] The fused video data is divided into several sub-images based on the input image size of the YOLOv8 network, and all sub-images are stacked into a batch.

[0035] Input data into the hazard detection model in batches, output the detection results of power distribution equipment hazard targets in all sub-images within the batch, merge all detection results, and convert the position coordinates of the targets in the sub-images into the position coordinates of the original images;

[0036] Non-maximum suppression processing is applied to all the transformed detection results to obtain the final hazard detection results.

[0037] Furthermore, acquire video data from multiple distribution network devices with synchronized timestamps from different perspectives, including:

[0038] Multiple viewpoints are arranged according to the rule of overlapping degrees between adjacent viewpoints to construct a multi-view shooting system. The video shooting parameters of each viewpoint in the multi-view shooting system are the same.

[0039] Place the multi-view shooting system in the scene of the power distribution network equipment to be tested, and set the video shooting mode to continuously acquire data.

[0040] Secondly, the present invention provides a power distribution network equipment hazard detection device based on large field-of-view stitched video data, comprising:

[0041] The data acquisition module is used to acquire video data from multiple distribution network devices with synchronized timestamps from different perspectives;

[0042] The preprocessing module is used to align the spatial and temporal parameters of the video data from multiple distribution network equipment based on the perspective differences and the synchronization timestamps.

[0043] The video fusion and splicing module is used to extract features from multiple video frame images based on aligned multi-channel power distribution network equipment video data and perform matching to obtain the splicing relationship between the multiple video frame images; it is also used to determine the splicing seam between two video frame images to be spliced ​​based on the splicing relationship; and it is also used to fuse and splice multiple video frame images based on the splicing relationship and splicing seam to obtain fused video data.

[0044] The hazard detection module is used to detect hazards in fused video data using a pre-trained hazard detection model, and obtain the hazard detection results for power distribution network equipment.

[0045] Accordingly, the present invention provides a computer device, the device including a processor and a memory:

[0046] The memory is used to store computer programs and send the instructions of the computer programs to the processor;

[0047] The processor executes instructions from a computer program, such as the method for detecting potential hazards in power distribution equipment based on large field-of-view stitched video data, as described in the first aspect.

[0048] Accordingly, the present invention provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements a method for detecting potential hazards in power distribution equipment based on large field-of-view spliced ​​video data as described in the first aspect.

[0049] In summary, this invention provides a method and related apparatus for detecting potential hazards in power distribution network equipment based on large field-of-view stitched video data. This method significantly expands the detection field of view by merging video data from different perspectives, and solves the problem of repeated shooting of multiple power distribution network devices at the same location through a large field of view. By combining synchronous timestamps to precisely align the spatial and temporal parameters of multiple video data streams, the continuity and consistency of the stitched video in time and space are ensured, avoiding false detections or missed detections caused by time misalignment or perspective differences, thus improving the reliability and effectiveness of the detection results. Through precise extraction and matching of image features from multiple video frames, this method can efficiently determine the stitching relationship between video frames and the optimal stitching seam, reducing visual distortion and seam traces during the stitching process, generating high-quality fused video data, and providing a clear and continuous visual foundation for subsequent hazard detection. Utilizing a pre-trained hazard detection model for automatic detection of the fused video data eliminates the need for manual review of each video, significantly improving detection speed and efficiency, while also reducing errors caused by human factors and enhancing the objectivity and standardization of the detection process. This invention solves the problem of requiring multiple detailed shots of a single power distribution network device by efficiently integrating multiple video resources for unified detection. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart of a method for detecting potential hazards in power distribution network equipment based on large field-of-view stitched video data provided in an embodiment of the present invention;

[0052] Figure 2 This is a technical roadmap for the method of detecting hidden dangers in power distribution equipment based on large field-of-view stitched video data provided in the embodiments of the present invention;

[0053] Figure 3 This is a block diagram of a power distribution network equipment hidden danger detection device based on large field-of-view stitched video data, provided in an embodiment of the present invention.

[0054] Figure 4 This is a block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0056] Please see Figure 1 This embodiment provides a method for detecting potential hazards in power distribution network equipment based on large field-of-view stitched video data, including the following steps:

[0057] S1: Acquire video data of multiple power distribution network devices with synchronized timestamps from different perspectives.

[0058] It should be noted that in practice, this step can involve deploying multiple cameras or video capture devices at different locations and angles within the power distribution network to ensure they simultaneously record video data. Crucially, all devices are equipped with a time synchronization mechanism (such as GPS clock synchronization), ensuring that each video frame shares a common timestamp, even when shot from different perspectives. This provides a temporal reference for subsequent processing, guaranteeing data consistency.

[0059] S2: Based on the perspective differences of video data from multiple distribution network devices, the spatial and temporal parameters of the video data from multiple distribution network devices are aligned using the synchronization timestamps.

[0060] It should be noted that due to the different positions and orientations of the various cameras, the captured videos are spatially inconsistent. This step utilizes the timestamps in the video data and the known camera position information to calculate and adjust the spatiotemporal parameters of each video stream, aligning the videos from different perspectives in time and space, thus preparing for subsequent stitching.

[0061] S3: Based on the aligned multi-channel power distribution network equipment video data, extract the features of the multi-channel video frame images and perform matching to obtain the splicing relationship between the multi-channel video frame images.

[0062] It should be noted that finding matching feature points between video frames, such as SIFT and ORB feature descriptors, and determining how different video frames are connected through the correspondence of these feature points, involves image processing techniques. The goal is to find seamlessly connected frame pairs, thereby determining the stitching order and method of the video frame images.

[0063] S4: Determine the seam between the two video frame images to be spliced ​​based on the splicing relationship.

[0064] It should be noted that after clarifying the splicing relationship of the video frames, the optimal overlap area (i.e., splicing seam) between each pair of adjacent frames is further calculated precisely in order to minimize splicing marks while maintaining scene continuity. This is usually achieved through image fusion algorithms to achieve a smooth transition.

[0065] S5: Based on the splicing relationship and splicing seams of multiple video frame images, the multiple video frame images are fused and spliced ​​to obtain fused video data.

[0066] It should be noted that, using the information obtained in the previous steps, multiple video frames are merged into a single large field-of-view video in the correct order and overlapping areas through image stitching technology. This process may include color and brightness adjustments to ensure the visual coherence and naturalness of the final video.

[0067] S6: Use a pre-trained hazard detection model to detect hazards in the fused video data to obtain the hazard detection results for power distribution network equipment.

[0068] It should be noted that this step involves applying machine learning or deep learning-based hazard detection models to analyze the stitched video data. These models, during their training phase, learn patterns to identify common hazards in power distribution network equipment, such as insulation damage and loose components. Through model inference, the system can automatically mark areas suspected of hazard, providing maintenance personnel with accurate inspection clues and accelerating hazard identification and repair.

[0069] This embodiment provides a method for detecting potential hazards in power distribution network equipment based on large field-of-view stitched video data. This method significantly expands the detection field of view by merging video data from different perspectives, and solves the problem of repeated shooting of multiple power distribution network devices at the same location through a large field of view. By combining synchronous timestamps to precisely align the spatial and temporal parameters of multiple video data streams, the method ensures the continuity and consistency of the stitched video in time and space, avoiding false detections and missed detections caused by time misalignment or perspective differences, thus improving the reliability and effectiveness of the detection results. Through precise extraction and matching of image features from multiple video frames, this method can efficiently determine the stitching relationship between video frames and the optimal stitching seam, reducing visual distortion and seam traces during the stitching process, generating high-quality fused video data, and providing a clear and continuous visual foundation for subsequent hazard detection. Using a pre-trained hazard detection model to automatically detect the fused video data eliminates the need for manual review of each video, greatly improving detection speed and efficiency, while also reducing errors caused by human factors and enhancing the objectivity and standardization of the detection process. This invention solves the problem of requiring multiple detailed shots of a single power distribution network device by efficiently integrating multiple video resources for unified detection. This invention provides a new intelligent inspection technology for detecting potential hazards in power distribution network equipment during power inspections, thereby improving inspection efficiency and promoting the intelligent development of power inspections.

[0070] Please see Figure 2 , Figure 2 This paper presents a technical roadmap for a method of detecting potential hazards in power distribution network equipment based on large field-of-view stitched video data. The following section combines... Figure 2 The present invention will be further described.

[0071] In a preferred embodiment of the present invention, step S1, acquiring multi-channel power distribution network equipment video data with synchronized timestamps from different viewpoints, includes:

[0072] Multiple viewpoints are arranged according to the rule of overlapping degrees between adjacent viewpoints to construct a multi-view shooting system. The video shooting parameters of each viewpoint in the multi-view shooting system are the same. The multi-view shooting system is placed in the scene of the power distribution network equipment to be tested, and the video shooting mode is set to continuous data acquisition.

[0073] In some specific embodiments of this example, a multi-view camera can be mounted on a gimbal using a drone. The multi-view camera consists of multiple color cameras with identical parameters, and the field of view of each camera overlaps by 5°. The multi-view camera is used to acquire images of power distribution network equipment from different perspectives.

[0074] Multi-view cameras offer several advantages in acquiring video data: they can stitch together videos from multiple perspectives to obtain a wide field of view, which can include multiple power distribution network devices in a single scene and large-sized power distribution network devices. Furthermore, video data has stronger information redundancy compared to image data, eliminating the need to deliberately capture high-quality and multi-angle image data. This effectively reduces the technical requirements for inspection personnel and improves the data acquisition efficiency of power distribution network devices during power inspections.

[0075] In a preferred embodiment of the present invention, step S2, aligning the spatial and temporal parameters of the multi-channel distribution network equipment video data based on the perspective differences and in conjunction with the synchronization timestamp, includes:

[0076] S21: Time alignment, select the main path and obtain the video frame data of the main path at the set time, index the time and video frame data of other paths before and after the set time, and obtain the video frame data of other paths at the set time through linear interpolation.

[0077] In some specific implementations, time alignment of the video data includes selecting and acquiring the main camera. exist Time-lapse data, indexing other cameras exist Frame time before and after the moment , And data, obtained from other cameras using linear interpolation methods. Data at any given time:

[0078]

[0079] in, , These represent the main camera and other cameras, respectively. , These represent other cameras. exist , Video frame data collected in real time; Indicates other cameras exist Time-aligned video frame data.

[0080] S22: Spatial alignment, based on camera calibration parameters from different perspectives, performs geometric correction on video data from each channel to eliminate the varying degrees of distortion that occur during imaging by each camera.

[0081] In some specific implementations, spatial alignment of video data includes geometric correction of each video stream based on the calibration parameters of each camera, eliminating the different degrees of distortion that occur in each camera during the imaging process.

[0082] S23: Brightness alignment. After converting the video data frame images from RGB format to HSV format, the brightness of the video frame images at the same time in each video data stream is unified.

[0083] In some specific implementations, brightness alignment of video data includes converting frame images from RGB format to HSV format, and then adaptively adjusting and unifying the brightness of video frame images at the same time in each video stream using an illumination gain compensation coefficient.

[0084]

[0085] in, , These represent the brightness values ​​of the images before and after illumination gain compensation, respectively. This represents the illumination gain compensation coefficient.

[0086] The illumination gain compensation coefficients for each camera image are determined by finding the extreme points of the illumination gain error function between the main camera image and other camera images.

[0087]

[0088] in, Indicates gain compensation error. , These represent the gain coefficients of the main camera image and the images from other cameras, respectively. , These represent the brightness values ​​of the images from the main camera and other cameras, respectively. , These represent the pixel coordinates of images from the main camera and other cameras, respectively. , These represent the overlapping areas of the images from the main camera and other cameras, respectively.

[0089] To simplify calculations and improve the robustness of the gain, the empirical formula for the illumination gain error function is as follows:

[0090]

[0091] in, , These represent the average brightness values ​​of the areas where the main camera image overlaps with other camera images; , These represent the standard deviations of error and gain, respectively. Approximately 0.7, Approximately 0.1 This indicates the number of pixels in the overlapping area.

[0092] To solve for the gain coefficient, a closed-form solution can be obtained by setting the derivative of the above function to zero. Let this equation... , Differentiation yields the following system of linear equations:

[0093]

[0094] Seek =1, for .

[0095] S24: Coordinate system alignment. Using the image reference system obtained from the main path as the reference system, calculate the projection transformation model of the video frame images of other paths based on the camera external parameters between other paths and the main path, and project the multi-path video frame data onto the reference system.

[0096] In some specific implementations, aligning the video data to a coordinate system includes using the image reference system acquired by the main camera as the reference system, calculating the projection transformation model of the video frame images of other cameras based on camera external parameters such as the angle difference and displacement difference between other cameras and the main camera, and projecting the multiple video frame data onto a unified coordinate system to reduce the parallax between the various video data.

[0097] Cylindrical projection is used to project multiple video frame images onto a unified coordinate system, projecting planar images onto the curved surface of a cylinder:

[0098]

[0099] in, This represents the pixel coordinates of the image after cylindrical projection; , These represent the width and height of the image, respectively.

[0100] In this embodiment, a generalized normalization operation is performed on multiple video data streams, preprocessing the data in four aspects: time, space, brightness, and coordinate system. Time alignment is used to unify the video frame images of each stream at the same moment; spatial alignment is used to eliminate the differences in data acquisition from cameras with the same parameters caused by camera distortion; brightness alignment is used to unify the brightness of the video frame images of each stream at the same moment; and coordinate system alignment is used to eliminate the spatial coordinates of video data from different viewpoints.

[0101] The generalized normalization of multi-channel video data has the following advantages: it can reduce or eliminate the differences between video data caused by objective factors such as environment and equipment, reduce the difficulty of video data fusion, improve the quality of fused video data, and provide a high-quality data foundation for subsequent detection of hidden dangers in power distribution network equipment.

[0102] In a preferred embodiment of the present invention, extracting and matching features from multiple video frame images includes:

[0103] S31: Utilize neural networks to extract multi-level features from multiple video frame images, obtaining coarse feature maps that meet small size requirements and fine feature maps that meet large size requirements for each video frame image.

[0104] In some specific implementations, feature extraction is performed on video frame images. This involves using a ResNet deep neural network loaded with ImageNet pre-trained weights to extract multi-level features from the video frame images, including the original image. Small-scale coarse feature map and original image Large-scale detailed feature map.

[0105] S32: After adding learnable positional encoding to the coarse feature map of each path, it is transformed into a one-dimensional vector. Then, the one-dimensional vector is transformed into an easy-to-match feature representation through the Transformer-based feature matching module to obtain the feature vector.

[0106] It should be noted that learnable positional codes are added pixel by pixel to the coarse feature maps of multiple video frame images, and they are transformed into one-dimensional vectors. The information between the multiple video frame images is then fused through a Transformer-based feature matching module and transformed into a feature representation that is easy to match.

[0107] S33: Using any one path as the main path, calculate the similarity matrix between the video frame images of other paths and the video frame images of the main path using the pixel-wise vector inner product method, and calculate the optimal match using the dual-Softmax method. Filter out some outlier matching pairs through the mutual nearest neighbor algorithm to obtain coarse matching point pairs that match the video frame images of multiple paths. The similarity matrix is ​​used to represent the similarity between the feature vectors of two video frame images.

[0108] It should be noted that the formula for calculating dual-Softmax is as follows:

[0109]

[0110] in, Represents the matching probability matrix; , These represent the video frame image features of the main camera and other cameras, respectively. , These represent the pixel coordinates of the video frame image features of the main camera and other cameras, respectively.

[0111] S34: Map the coarse matching point pairs to the corresponding fine feature maps, and input the mapped regions into the Transformer-based feature matching module. Calculate the matching probability of the center feature of the main video frame image with all features of the other matching video frame images. Take the feature position with the highest matching probability as the sub-pixel precision matching point position. The sub-pixel precision matching point position represents the splicing relationship between the two matched video frame images.

[0112] It should be noted that the point pairs in the coarse match , The mapping is applied to the corresponding fine feature map, and the mapped region is input into the Transformer-based feature matching module for further fine-tuning feature matching and computation. Central features and By calculating the matching probabilities of all features, the corresponding probability distribution is obtained, thus yielding... The subpixel precision of the matching point position.

[0113] In this embodiment, a ResNet neural network pre-trained on ImageNet is used to extract features from video frame images, obtaining small-sized coarse feature maps and large-sized fine feature maps. Coarse and fine feature matching are achieved through a similarity measurement between feature maps of multiple video frame images and a LoFTR feature matching module. By calculating the probability distribution between features, sub-pixel-level matching point positions are obtained. A dynamic programming method is used to find the optimal stitching seam between multiple video frame images, and a wavelet transform-based image fusion algorithm is used to stitch the video frame images together. Post-processing is performed on the stitched video frame images to obtain a high-quality, large-field-of-view stitched video.

[0114] In a preferred embodiment of the present invention, the stitching seam is the optimal stitching seam, which is determined by a dynamic programming method with the objective of minimizing image differences. The determination process includes:

[0115] S41: Determine the pixel-wise energy summation function for the two video frame images to be stitched together. The energy summation function is used to quantify the degree of image difference in the overlapping area of ​​the two video frame images.

[0116] In some specific implementations, a pixel-by-pixel energy summation function for multiple video frame images is constructed using the idea of ​​minimizing differences. To evaluate the structural and color differences between images at the seam, the energy function is calculated using the following formula:

[0117]

[0118] in, This represents all the pixels through which the seam passes; This indicates the weight used to balance the color difference and structural difference terms; Indicates the color difference of pixels at the same location in the overlapping area; This indicates the structural differences between pixels at the same location in the overlapping region.

[0119]

[0120]

[0121] in, , They represent , Sobel operators in direction; , These represent video frame images from the main camera and other cameras, respectively.

[0122] S42: Initialize the seam path, taking the first row of the overlapping area as the minimum energy and path for the first step; iterate based on the minimum energy and path, traversing each row in turn to obtain the position of the minimum energy and path source, and at the same time record the minimum energy and path vector reaching each row and each column, as well as the position of the path source reaching each row and each column.

[0123] It should be noted that dynamic programming is used to find the optimal seam. The seam path is initialized, with the first row as the minimum energy and path for the first step. Iteration is performed based on the minimum energy, traversing each row sequentially to obtain the position of the minimum energy and path source: top left, top, top right. During the traversal, the minimum energy and vector reaching each row and column, as well as the position of the path source for each row and column, need to be recorded for backtracking.

[0124] S43: Backtrack the seams by recording the position matrix of the source and backtracking the seams from the last row. Select the seam with the smallest total energy as the best seam.

[0125] It should be noted that the seam tracing is performed by backtracking from the last row of the source position matrix. In terms of direction, subtract 1 if it's top left, add 1 if it's top right, and leave it unchanged if it's top. Decrease by 1 in the direction. Compare all seams and select the suture with the lowest total energy as the optimal suture.

[0126] In a preferred embodiment of the present invention, fusing and stitching multiple video frame images includes:

[0127] S51: Perform wavelet transform on all video frame images to be stitched, and use the Mallat algorithm to decompose them to obtain low-frequency and high-frequency components.

[0128] S52: A weighted average fusion method is used to fuse the low-frequency components to obtain the fused low-frequency components.

[0129] In some specific implementations, a weighted average fusion method is used for the low-frequency components. The fusion algorithm is as follows:

[0130]

[0131] in, , These represent the low-frequency coefficients of video frame images from the main camera and other cameras, respectively. , These represent the weight values ​​corresponding to the low-frequency coefficients, and their sum is 1.

[0132] S53: The high-frequency components are fused using a method based on maximizing the absolute value of the window coefficient to obtain the fused high-frequency components.

[0133] In some specific implementations, a method based on maximizing the absolute value of the window coefficient is used for fusion of high-frequency components. The fusion algorithm is as follows:

[0134]

[0135] in, , These represent the high-frequency coefficients of video frame images from the main camera and other cameras, respectively.

[0136] S54: Perform inverse wavelet transform on the fused low-frequency and high-frequency components to obtain the fused image.

[0137] S55: Post-processing of large field-of-view stitched video.

[0138] The details and edges of the stitched image are enhanced through sharpening filtering and local contrast enhancement methods. Secondly, image denoising techniques are used to reduce the noise level of the stitched image. Finally, the stitched video frame images are saved as a video sequence, thus outputting large field-of-view video data.

[0139] In a preferred embodiment of the present invention, the hazard detection model is trained using a YOLOv8 network, and the training steps include:

[0140] S61: Construct a target detection dataset for power distribution network equipment based on large field-of-view stitched video data.

[0141] First, data cleaning is performed to remove video frames with poor imaging quality caused by factors such as camera overexposure, camera underexposure, camera out-of-focus, motion blur, and internal parameter calibration errors.

[0142] Secondly, the potential hazards in the power distribution network equipment were labeled and filtered frame by frame in the video data, and the data was divided into training and validation sets to form a standard dataset that the neural network can learn from. The dataset contains images of potential hazards in the power distribution network equipment, along with their categories and locations.

[0143] S62: Train the YOLOV8 target detection neural network using the hazard detection training dataset.

[0144] First, minimizing the difference between predicted and true values ​​is used as the optimization criterion for the neural network. Focal loss is adopted as the classification loss function for the target detection network of hidden dangers in distribution network equipment, reducing the impact of the imbalance between the amount of hidden danger data and the identification complexity of different distribution network equipment on the network stability. Cross-intersection over union (CIU) loss is adopted as the loss function for the localization of target boxes of hidden dangers in distribution network equipment.

[0145]

[0146]

[0147] in, Represents the loss function for classifying hidden dangers in power distribution network equipment; This indicates the degree of similarity between the predicted and actual categories of potential hazards in power distribution network equipment. The closer the result is to 1, the more accurate the prediction. This indicates an adjustable factor, which is greater than 0; This indicates the loss caused by the location of potential hazards in power distribution network equipment. , This indicates the predicted and actual areas of the target box for potential hazards in power distribution network equipment.

[0148] Secondly, gradient descent is used to update the network parameters, with mini-batch stochastic gradient descent as the parameter optimization method:

[0149]

[0150] in, , , This represents the learnable parameters of the YOLOv8 network; lr represents the learning rate, which is the step size for each optimization. The exponential moving average representing the gradient; Represents the exponential moving average of the squared gradient; This represents a very small non-zero number, to avoid network optimization failure caused by division by 0 during the optimization process. Here, it is assumed to be... .

[0151] S63: The trained network is evaluated using mAP and IoU to assess the effectiveness of the YOLOv8 object detection neural network in classifying and locating potential hazards in power distribution network equipment. mAP is calculated by averaging the average precision (AP) for each category. For each category's AP, precision (PR) and recall (Recall) curves are constructed, and the area between the PR curve and the coordinate axes is calculated to obtain the average precision (AP) for each category. IoU is calculated using the intersection-over-union (IoU) ratio of each predicted target and the true target. The common area and the total area are calculated separately, and the common area is divided by the total area to obtain the final result.

[0152] Calculate mAP:

[0153]

[0154] Calculate IoU:

[0155]

[0156] Where P(r) represents the PR curve function; This represents the maximum recall rate; Indicates the target area for prediction; This represents the actual target area.

[0157] In a further embodiment of the present invention, a pre-trained hazard detection model is used to detect fused video data, including:

[0158] S71: Divide the fused video data into several sub-images according to the input image size of the YOLOv8 network, and stack all the sub-images into a batch.

[0159] In some specific implementations, the sliding window size is set according to the input image size of the YOLOV8 network, and the large field-of-view power distribution network hidden danger video frame image is evenly divided into several sub-images according to a 10% overlap area, and all sub-images are stacked into a batch.

[0160] S71: Input batch data into the hidden danger detection model, output the detection results of hidden danger targets of power distribution equipment in all sub-images within the batch, merge all detection results, and convert the position coordinates of the target in the sub-image into the position coordinates of the original image.

[0161] In some specific implementations, the batch input data is fed into the trained YOLOv8 network, the output results of power grid hazard target detection for all sub-images within the batch are used to merge all detection results, and the position coordinates of the target in the sub-image are converted into the position coordinates of the original image.

[0162]

[0163] in, , These represent the coordinates of the top-left and bottom-right corners of the predicted bounding box region in the original image, respectively. , These represent the coordinates of the top-left and bottom-right corners of the predicted bounding box region in the original image, respectively. This indicates the coordinates of the top-left corner of the subgraph within the original graph.

[0164] S71: Perform non-maximum suppression processing on all transformed detection results to obtain the final hidden danger detection results.

[0165] Non-maximum suppression processing is applied to all converted detection results to obtain the final hidden danger detection results. Duplicate detection of hidden dangers is removed, thus completing the detection of hidden dangers in large field-of-view power distribution equipment with high resolution.

[0166] The above embodiments utilize video data to construct a target detection dataset for potential hazards in power distribution network equipment. The dataset is used to train and validate the target detection network, and the sliding block detection method is combined to achieve the detection of potential hazards in power distribution network equipment while maintaining high-resolution, large-field-of-view images.

[0167] Based on the same inventive concept, this application also provides a device for detecting potential hazards in power distribution equipment based on large field-of-view stitched video data, used to implement the aforementioned method for detecting potential hazards in power distribution equipment based on large field-of-view stitched video data. The solution provided by this system is similar to the solution described in the above method. Therefore, the specific limitations in the embodiments of the device for detecting potential hazards in power distribution equipment based on large field-of-view stitched video data provided below can be found in the limitations of the method for detecting potential hazards in power distribution equipment based on large field-of-view stitched video data described above, and will not be repeated here.

[0168] Please see Figure 3 This embodiment provides a power distribution network equipment hazard detection device based on large field-of-view stitched video data, including:

[0169] The data acquisition module is used to acquire video data from multiple distribution network devices with synchronized timestamps from different perspectives;

[0170] The preprocessing module is used to align the spatial and temporal parameters of the video data from multiple distribution network equipment based on the perspective differences and the synchronization timestamps.

[0171] The video fusion and splicing module is used to extract features from multiple video frame images based on aligned multi-channel power distribution network equipment video data and perform matching to obtain the splicing relationship between the multiple video frame images; it is also used to determine the splicing seam between two video frame images to be spliced ​​based on the splicing relationship; and it is also used to fuse and splice multiple video frame images based on the splicing relationship and splicing seam to obtain fused video data.

[0172] The hazard detection module is used to detect hazards in fused video data using a pre-trained hazard detection model, and obtain the hazard detection results for power distribution network equipment.

[0173] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0174] Reference Figure 4 The present invention also provides a computer device 40, including a memory 402 and a processor 401, and a computer program 403 stored in the memory 402. When the computer program 403 is executed on the processor 401, it implements the method for detecting hidden dangers in power distribution equipment based on large field-of-view spliced ​​video data as described in any of the above methods.

[0175] The computer device 40 may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device 40 may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will understand that... Figure 4 The computer device 40 is merely an example and does not constitute a limitation on the computer device 40. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0176] The processor 401 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0177] In some embodiments, the memory 402 may be an internal storage unit of the computer device 40, such as a hard disk or memory of the computer device 40. In other embodiments, the memory 402 may be an external storage device of the computer device 40, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device 40. Furthermore, the memory 402 may include both internal and external storage units of the computer device 40. The memory 402 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 402 can also be used to temporarily store data that has been output or will be output.

[0178] This invention also provides a computer-readable storage medium storing a computer program thereon. When the computer program is run by a processor, it implements the method for detecting potential hazards in power distribution equipment based on large field-of-view stitched video data as described in any of the above methods.

[0179] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0180] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0181] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0182] In the embodiments disclosed in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0183] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting potential hazards in power distribution network equipment based on large field-of-view stitched video data, characterized in that, Includes the following steps: Acquire video data from multiple distribution network devices with synchronized timestamps from different perspectives; Based on the perspective differences of the multi-channel distribution network equipment video data, the spatial and temporal parameters of the multi-channel distribution network equipment video data are aligned in conjunction with the synchronization timestamp; Based on the aligned video data of the multi-channel power distribution network equipment, features of the multi-channel video frame images are extracted and matched to obtain the splicing relationship between the multi-channel video frame images. The stitching seam between the two video frame images to be stitched is determined based on the stitching relationship. Based on the splicing relationship and splicing seam of the multi-channel video frame images, the multi-channel video frame images are fused and spliced ​​to obtain fused video data; The fused video data is then used to detect potential hazards in the power distribution network equipment using a pre-trained hazard detection model. The stitching seam is the optimal stitching seam, which is determined by a dynamic programming method with the objective of minimizing image differences. The determination process includes: Determine a pixel-wise energy summation function for two video frame images to be stitched together, wherein the energy summation function is used to quantify the degree of image difference in the overlapping area of ​​the two video frame images; Initialize the seam path, taking the first row of the overlapping area as the minimum energy and path for the first step; iterate based on the minimum energy and path, traversing each row in turn to obtain the position of the minimum energy and path source, and at the same time record the minimum energy and vector reaching each row and each column, as well as the position of the path source reaching each row and each column. The splicing seams are traced back from the last row by recording the position matrix of the source, and the splicing seam with the minimum total energy is selected as the best splicing seam; The process of fusing and stitching the multiple video frame images includes: All video frame images to be stitched are subjected to wavelet transform and decomposed using the Mallat algorithm to obtain low-frequency and high-frequency components. A weighted average fusion method is applied to the low-frequency components to obtain the fused low-frequency components; The high-frequency components are fused using a method based on maximizing the absolute value of the window coefficient to obtain the fused high-frequency components; The fused low-frequency component and the high-frequency component are subjected to inverse wavelet transform to obtain the fused image.

2. The method for detecting potential hazards in power distribution network equipment based on large field-of-view stitched video data according to claim 1, characterized in that, Extracting features from multiple video frame images and performing matching includes: By using neural networks to extract multi-level features from multiple video frame images, coarse feature maps that meet small size requirements and fine feature maps that meet large size requirements are obtained for each video frame image. After adding learnable positional encoding to the coarse feature map of each path, it is transformed into a one-dimensional vector. Then, the one-dimensional vector is transformed into an easily matched feature representation through the Transformer-based feature matching module to obtain the feature vector. Taking any one path as the main path, the similarity matrix between the video frame images of other paths and the video frame images of the main path is calculated using the pixel-wise vector inner product method. The optimal match is calculated using the dual-Softmax method, and some outlier matching pairs are filtered out by the mutual nearest neighbor algorithm to obtain coarse matching point pairs that match the video frame images of multiple paths. The similarity matrix is ​​used to represent the similarity between the feature vectors of two video frame images. The coarse matching point pairs are mapped to the corresponding fine feature maps, and the mapped regions are input into the Transformer-based feature matching module. The matching probability of the center feature of the main video frame image and all features of the other matching video frame images is calculated. The feature position with the highest matching probability is taken as the sub-pixel precision matching point position. The sub-pixel precision matching point position represents the splicing relationship between the two matched video frame images.

3. The method for detecting potential hazards in power distribution network equipment based on large field-of-view stitched video data according to claim 1, characterized in that, Based on the perspective differences of the multi-channel distribution network equipment video data, and in conjunction with the synchronization timestamp, the spatial and temporal parameters of the multi-channel distribution network equipment video data are aligned, including: Time alignment: Select the main path and obtain the video frame data of the main path at a set time; index the time and video frame data of other paths before and after the set time; obtain the video frame data of other paths at the set time through linear interpolation. Spatial alignment involves geometrically correcting the video data from each camera based on the camera calibration parameters from different perspectives, thereby eliminating the varying degrees of distortion that occur during imaging by each camera. Brightness alignment involves converting the images of each video data frame from RGB format to HSV format and then unifying the brightness of the video frames at the same time in each video data stream. Coordinate system alignment is performed, using the image reference system acquired by the main path as the reference reference system. Based on the camera extrinsic parameters between the other paths and the main path, the projection transformation model of the video frame images of the other paths is calculated, and the video frame data of multiple paths is projected onto the reference reference system.

4. The method for detecting potential hazards in power distribution network equipment based on large field-of-view stitched video data according to claim 1, characterized in that, The hazard detection model is trained using a YOLOv8 network. The pre-trained hazard detection model is used to detect hazards in the fused video data, including: The fused video data is divided into several sub-images according to the input image size of the YOLOv8 network, and all sub-images are stacked into a batch. Input data into the hidden danger detection model according to the batch, output the detection results of hidden danger targets of power distribution equipment in all sub-images within the batch, merge all detection results, and convert the position coordinates of the target in the sub-image into the position coordinates of the original image; Non-maximum suppression processing is applied to all the transformed detection results to obtain the final hazard detection results.

5. The method for detecting potential hazards in power distribution network equipment based on large field-of-view stitched video data according to claim 1, characterized in that, Acquire video data from multiple distribution network devices with synchronized timestamps from different perspectives, including: A multi-view shooting system is constructed by arranging multiple viewpoints according to the rule of overlapping adjacent viewpoints by a set degree. In the multi-view shooting system, the video shooting parameters of each viewpoint are the same. The multi-view shooting system is placed in the scene of the power distribution network equipment to be tested, and the video shooting mode is set to continuously acquire data.

6. A device for detecting potential hazards in power distribution network equipment based on large field-of-view stitched video data, characterized in that, include: The data acquisition module is used to acquire video data from multiple distribution network devices with synchronized timestamps from different perspectives; The preprocessing module is used to align the spatial and temporal parameters of the multi-channel distribution network equipment video data based on the perspective differences and the synchronization timestamp; The video fusion and splicing module is used to extract features from multiple video frame images based on the aligned multi-channel power distribution network equipment video data and perform matching to obtain the splicing relationship between the multiple video frame images; it is also used to determine the splicing seam between two video frame images to be spliced ​​based on the splicing relationship; and it is also used to fuse and splice the multiple video frame images based on the splicing relationship and the splicing seam to obtain fused video data. The hazard detection module is used to detect hazards in the fused video data using a pre-trained hazard detection model, and obtain the hazard detection results for power distribution network equipment. The stitching seam is the optimal stitching seam, which is determined by a dynamic programming method with the objective of minimizing image differences. The determination process includes: Determine a pixel-wise energy summation function for two video frame images to be stitched together, wherein the energy summation function is used to quantify the degree of image difference in the overlapping area of ​​the two video frame images; Initialize the seam path, taking the first row of the overlapping area as the minimum energy and path for the first step; iterate based on the minimum energy and path, traversing each row in turn to obtain the position of the minimum energy and path source, and at the same time record the minimum energy and vector reaching each row and each column, as well as the position of the path source reaching each row and each column. The splicing seams are traced back from the last row by recording the position matrix of the source, and the splicing seam with the minimum total energy is selected as the best splicing seam; The process of fusing and stitching the multiple video frame images includes: All video frame images to be stitched are subjected to wavelet transform and decomposed using the Mallat algorithm to obtain low-frequency and high-frequency components. A weighted average fusion method is applied to the low-frequency components to obtain the fused low-frequency components; The high-frequency components are fused using a method based on maximizing the absolute value of the window coefficient to obtain the fused high-frequency components; The fused low-frequency component and the high-frequency component are subjected to inverse wavelet transform to obtain the fused image.

7. A computer device, characterized in that, The device includes a processor and a memory: The memory is used to store computer programs and send the instructions of the computer programs to the processor; The processor executes, according to the instructions of the computer program, a method for detecting potential hazards in power distribution equipment based on large field-of-view stitched video data as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for detecting potential hazards in power distribution equipment based on large field-of-view stitched video data as described in any one of claims 1-5.

Images