Substation defect data enhancement method and system based on head and tail frame video generation

Abstract

The invention relates to the technical field of intelligent monitoring of power equipment, provides a transformer substation defect data enhancement method and system based on video generation of head and tail frames, and solves the problems of sample scarcity and insufficient dynamic evolution simulation. According to the main scheme, the method comprises the following steps: acquiring a substation equipment defect tail frame image, and extracting a depth map, an edge contour map and defect mask labels; based on a diffusion model of a ControlNet multi-constraint mechanism, fusing the depth map, the edge contour and the defect mask to carry out first-frame repair on the tail-frame image, and generating a defect-free first-frame image with consistent visual angle, illumination and background; inputting the repaired first frame and the original tail frame into a time sequence generation model, and generating a dynamic defect evolution video sequence in combination with a text prompt constraint defect evolution path and apparent characteristics; and time sequence coherent fragments are screened through optical flow consistency detection, frame insertion is carried out on the key stage to prolong the video duration, and finally high-quality defect enhancement data are generated. According to the scheme, the problems of sample scarcity and insufficient dynamic evolution simulation are effectively solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent monitoring of power equipment, and provides a substation defect data enhancement method and system based on head and tail frame video generation. BACKGROUND

[0002] In the field of intelligent monitoring of power equipment, substation defect detection technology based on computer vision is becoming a core means to ensure the safe operation of the power grid. With the rapid development of artificial intelligence technology, using advanced models such as YOLO and Transformer to train high-precision defect recognition algorithms has become the mainstream solution in the industry. However, in actual application scenarios, the collection of substation equipment defect samples faces severe challenges: on the one hand, the occurrence probability of key equipment defects (such as transformer oil leakage and insulator crack) is extremely low and evolves slowly, making it difficult for maintenance personnel to capture the complete defect evolution process; on the other hand, the existing inspection system does not obtain real-time video streams, but discrete photos taken at intervals of several weeks or even months, resulting in significant perspective deviation, light difference, and equipment state change between images across time.

[0003] In the face of this core bottleneck of sample shortage, the power industry attempts to introduce image generation and data enhancement technology. Current mainstream solutions include: static defect generation based on GAN, Stable Diffusion model repair, and expansion of the data set by generating static oil leakage images. However, these technologies still cannot simulate the time continuity and physical causality of defect evolution, resulting in poor performance of the trained detection model in real video streams and high dynamic defect omission rate. How to generate defect evolution data that conforms to the physical laws of equipment and has temporal coherence has become a key problem restricting the intelligent inspection of substations.

[0004] Based on the current situation of sample shortage and the difficulty of existing generation technologies to simulate the temporal evolution of dynamic defects, the present application aims to provide a substation defect data enhancement method and system based on head and tail frame video generation. This method and system introduce precise first frame repair technology guided by multi-modal conditions and temporal generation strategies controlled by physical properties, reconstruct a defect-free first frame with strictly aligned viewpoints using real tail frames, and drive the synthesis of dynamic defect video sequences that conform to the physical evolution laws of equipment. This method fully utilizes the perspective, structural information, and physical evolution law description contained in the tail frame images, and can effectively generate defect evolution video data with temporal coherence and physical rationality, thereby solving the problem of sample scarcity and improving the detection accuracy and robustness of the defect recognition model for dynamic defects. This has important practical significance for improving the level of intelligent monitoring of power grid equipment and ensuring the safe and stable operation of the power system. SUMMARY

[0005] The purpose of the present application is to solve the problems of substation equipment defect sample scarcity and dynamic evolution process difficulty to simulate, through first and last frame image repair and multi-modal constraint time sequence generation technology, realizing high consistency, physical reasonable defect evolution video data enhancement.

[0006] In order to achieve the above purpose, the present application adopts the following technical means:

[0007] The substation defect data enhancement method based on first and last frame video generation, characterized in that it comprises the following steps:

[0008] Step S1: obtaining the defect tail frame image of the substation equipment, and extracting the depth map, edge contour map and defect mask annotation of the tail frame image;

[0009] In the general substation intelligent inspection system, it is necessary to optimize the abnormality and defect recognition ability and recall ability of the analysis model. Generally, the missed report data fed back online is directly collected and added to the training set of the model for optimization, but the data quality is not considered, which may not be effective or may negatively optimize the model. Considering the difficulty of substation defect recognition and the ambiguity in some scenes, the tail frame picture screening standard in step S1 is adopted, and data with clear defect semantics and good image quality are used as the starting point. And the spatial features (depth map), detail features (edge contour map) and defect abnormal area are obtained and labeled, preparing for subsequent generation.

[0010] Step S2: based on the depth map, edge contour map and defect mask annotation of the tail frame image, the first frame of the tail frame image is repaired by fusing the diffusion model of the ControlNet multi-constraint mechanism, and a defect-free first frame image with strict view angle, illumination and background consistency is generated;

[0011] Based on AIGC data generation technology, generally, the lora model fine-tuning or feature migration is used after data preprocessing, so as to generate a batch of data, but since a single discrete picture is used, the features will be relatively single, for example, the generated defects are consistent, and the diversity is poor. Considering that the defect may have multiple intermediate states, such as from few to many or from small to large, in step S2, based on the depth map, contour map and mask region condition guided by controlnet obtained in step S1, the defect picture is repaired to a normal picture, which can be used as background data of the recognition model, or normal-defect picture pairs can be used to construct multi-modal model training data, or the first and last frame pictures generated by video in this paper.

[0012] Step S3: inputting the repaired first frame image and the original tail frame image into the time sequence generation model, constraining the spatial path and apparent features of the defect evolution through text prompts, and generating a dynamic defect evolution video sequence;

[0013] With the rapid development of AIGC technology, image generation is gradually being applied to the generation of training data for anomaly and defect recognition models. However, current technology struggles to generate coherent, progressively changing defect data, posing a challenge to the generalization optimization of the model. This case innovatively uses a video generation model to generate substation defect data. However, existing video generation models, such as WAN and LTX models, lack sufficient understanding of substation scenarios due to the confidentiality of substation data. Therefore, step S3 uses the first and last frame images composed of the normal-defect image from step S2 as a powerful control method, employing models like WAN for controllable generation. By using depth, contour, and mask region control on the last frame, a very high-quality first frame image is obtained, resulting in highly coherent video data that conforms to physical laws.

[0014] Step S4: Perform optical flow consistency detection on the dynamic defect evolution video sequence, select temporally coherent video segments, and perform frame interpolation on key evolution stages to extend the video duration and generate enhanced defect data.

[0015] Due to resource limitations of video generation models, such as the significant hardware resource consumption required to generate a video clip of ten seconds or more, and considering that easily missed defects generally have small pixel ranges, a video model can be used to generate videos at 16 FPS with a duration of 3-5 seconds, which can fully demonstrate the defect formation process. Since the model's generation results for different seed numbers are random, there may be unreasonable effects, such as oil stains suddenly and rapidly spreading within a very short interval. The optical flow consistency detection method in step S4 can largely filter out videos with discontinuous or unreasonable lighting changes. For types of anomalies with longer formation cycles, such as oil stains which typically form slowly dripping, the key stage of a shorter video generation (oil stain spreading) can be processed using frame interpolation in step S4, thereby improving the reasonableness of this anomaly type. This can be used for training or testing time-series defect recognition models, or frame interpolation can increase the amount of data.

[0016] In the above scheme, step S1 includes:

[0017] Step S1.1: Defect tail frame image acquisition and screening:

[0018] Discrete images of substation equipment are acquired from the existing inspection system, and images containing complete defect states are manually selected as tail frame samples. The selection criteria include: the main body of the equipment completely covers ≥30% of the image, uniform lighting, histogram variance <0.1, clearly visible defect areas, and contrast >40%.

[0019] Step S1.2: Precise labeling of defect masks:

[0020] A pixel-level annotation tool is used to annotate the defective areas in the tail frame image to generate a binary mask image; wherein, in the binary mask image, pixels marking the repaired areas that need to be modified are represented by 1 (white), and pixels marking the preserved areas are represented by 0 (black);

[0021] Step S1.3: Extraction of multimodal structure guidance information:

[0022] A depth map is extracted from the tail frame image using a depth estimation model, and an edge contour map (Canny Edge) is extracted from the tail frame image using an adaptive threshold Canny algorithm; wherein the mathematical process of the adaptive threshold Canny algorithm includes:

[0023] Gaussian filtering:

[0024]

[0025]

[0026] Gradient calculation:

[0027]

[0028]

[0029] Nonmaximum suppression:

[0030]

[0031] Dual threshold filtering:

[0032]

[0033] in, The standard deviation of the Gaussian kernel. , Adaptive calculation is used to determine high and low thresholds. get, This represents the multiplication operation.

[0034] In the above scheme, step S2 includes:

[0035] Step S2.1: Multi-source conditional input construction: The tail frame image, defect mask, depth map and edge map are input into the diffusion model of ControlNet multi-constraint mechanism;

[0036] Step S2.2: Backdiffusion Process Control: In the backdiffusion denoising process of the basic diffusion model, a depth map, a Canny edge map, and text prompts are introduced as joint constraints; wherein, the backdiffusion formula is:

[0037]

[0038] This is the restored image after backdiffusion. For time step Noise attenuation coefficient, defined as , Indicates time step Predefined noise attenuation coefficient, This represents the noise predicted by the U-Net network, whose input includes time steps. Latent variables and constraint-condition depth map features Edge map features Text prompt features , This is the noise scaling factor. , This is standard normally distributed noise;

[0039] Step S2.3: Mask Region Repair Control: Repair is performed within the region of the binary mask image, using the following formula:

[0040]

[0041] in: Represents a global latent variable. For noise in the mask area, This represents global noise in the non-masked region. For binary mask images, This represents element-wise multiplication;

[0042] Step S2.4: Repair result generation: The first frame image output by the above process presents a defect-free clean equipment surface, while maintaining the same viewing angle, lighting and background environment as the last frame image.

[0043] In the above scheme, step S3 includes:

[0044] Step S3.1: Input the reconstructed first frame image and the original last frame image into the temporal generation model, wherein the temporal generation model is Wan2.1;

[0045] Step S3.2: Constrain the spatial path of defect evolution through text prompts, the text prompts describing the physical evolution law of the defect, such as "oil droplets slowly seep out from the weld and meander downwards along the pipeline";

[0046] Step S3.3: Based on the text prompt-guided timing generation model, generate a dynamic defect evolution video sequence that conforms to physical laws. For example, for transformer oil leakage, generate a continuous frame sequence of oil droplets slowly seeping out from the weld and meandering downwards along the pipeline.

[0047] In the above scheme, step S4 includes:

[0048] Step S4.1: Perform optical flow consistency detection on the synthesized video and filter out video segments with temporal coherence; wherein, the optical flow consistency detection includes:

[0049] Calculate the forward optical flow: the optical flow from the current frame t to the next frame t+1, representing the displacement of each pixel in the current frame:

[0050]

[0051] It is the position of the pixel in the current frame. and It represents the horizontal and vertical displacement of the pixel.

[0052] Calculate the reverse optical flow: the optical flow from the next frame t+1 to the current frame t, representing the displacement of each pixel in the next frame:

[0053]

[0054] and These represent the horizontal and vertical displacements of the pixel in the next frame, respectively.

[0055] Calculate the consistency error: for each pixel The consistency error is calculated using forward and reverse optical flow.

[0056]

[0057] This means that the result of the forward optical flow is traced in reverse using the reverse optical flow, and the resulting optical flow should match the forward optical flow. Represents the Euclidean norm, used to calculate the distance between the forward and reverse optical flows;

[0058] Calculate the overall consistency error: a comprehensive evaluation of the optical flow consistency across the entire video, calculated as follows:

[0059]

[0060] in, It is the total number of pixels in the image. It is the optical flow consistency error of each pixel;

[0061] Consistency judgment: based on overall consistency error Set a threshold To determine whether a video has continuity, if If the motion is below a certain threshold, the video is considered to have consistency; otherwise, it is considered to have disjointed motion.

[0062] Step S4.2: Perform frame interpolation on the selected sequential video segments to increase the video duration; in particular, use repeating frames or frame interpolation algorithms to process the "drop → convergence" stage, and reduce the playback speed of the segment to 1 / 8 to 1 / 4 of the original speed by time stretching, so that the total duration is extended to 30 to 120 seconds.

[0063] Step S4.3: Extract frames from the processed video to obtain continuous oil leakage image data, which is then used for testing and training the defect detection model.

[0064] Optical flow consistency testing is integrated here as a quality screening step to ensure the physical rationality of the synthesized video and to serve the accurate enhancement of substation defect data.

[0065] This invention also provides a substation defect data enhancement system based on first and last frame video generation, comprising:

[0066] Image acquisition module: Acquires defect tail frame images of substation equipment, and extracts the depth map, edge contour map and defect mask annotation of the tail frame images;

[0067] First frame repair module: Based on the depth map, edge contour map and defect mask annotation of the last frame image, the first frame repair module performs first frame repair on the last frame image by using a diffusion model that integrates ControlNet multi-constraint mechanism to generate a defect-free first frame image with strict consistency in viewpoint, lighting and background with the last frame image.

[0068] The timing generation module inputs the repaired first frame image and the original last frame image into the timing generation model, and generates a dynamic defect evolution video sequence by constraining the spatial path and appearance features of defect evolution through text prompts.

[0069] Data enhancement processing module: Performs optical flow consistency detection on the dynamic defect evolution video sequence, filters out temporally coherent video segments, and performs frame interpolation on key evolution stages to extend the video duration, generating enhanced defect data.

[0070] In the above scheme, the implementation of the image acquisition module includes:

[0071] Step S1.1: Defect tail frame image acquisition and screening:

[0072] Discrete images of substation equipment are acquired from the existing inspection system, and images containing complete defect states are manually selected as tail frame samples. The selection criteria include: the main body of the equipment completely covers ≥30% of the image, uniform lighting, histogram variance <0.1, clearly visible defect areas, and contrast >40%.

[0073] Step S1.2: Precise labeling of defect masks:

[0074] A pixel-level annotation tool is used to annotate the defective areas in the tail frame image to generate a binary mask image; wherein, in the binary mask image, pixels marking the repaired areas that need to be modified are represented by 1 (white), and pixels marking the preserved areas are represented by 0 (black);

[0075] Step S1.3: Extraction of multimodal structure guidance information:

[0076] A depth map is extracted from the tail frame image using a depth estimation model, and an edge contour map (Canny Edge) is extracted from the tail frame image using an adaptive threshold Canny algorithm; wherein the mathematical process of the adaptive threshold Canny algorithm includes:

[0077] Gaussian filtering:

[0078]

[0079]

[0080] in The standard deviation of the Gaussian kernel is represented. Represents the input image matrix. The asterisk (*) represents the natural constant, and the asterisk (*) represents the convolution operation.

[0081] Gradient calculation:

[0082]

[0083]

[0084] in This represents the image matrix after Gaussian filtering. Represents the gradient magnitude matrix. Represents the gradient direction matrix (edge ​​angle);

[0085] Nonmaximum suppression:

[0086]

[0087] Dual threshold filtering:

[0088]

[0089] in, The standard deviation of the Gaussian kernel. , To adapt to high and low thresholds, adaptive calculation is used. get, It represents the multiplication operation.

[0090] In the above scheme, the implementation of the first frame repair module includes:

[0091] Step S2.1: Multi-source conditional input construction: The tail frame image, defect mask, depth map and edge map are input into the diffusion model of ControlNet multi-constraint mechanism;

[0092] Step S2.2: Backdiffusion Process Control: In the backdiffusion denoising process of the basic diffusion model, a depth map, a Canny edge map, and text prompts are introduced as joint constraints; wherein, the backdiffusion formula is:

[0093]

[0094] This is the restored image after backdiffusion. For time step Noise attenuation coefficient, defined as , Indicates time step Predefined noise attenuation coefficient, This represents the noise predicted by the U-Net network, whose input includes time steps. Latent variables and constraint-condition depth map features Edge map features Text prompt features , This is the noise scaling factor. , This is standard normally distributed noise;

[0095] Step S2.3: Mask Region Repair Control: Repair is performed within the region of the binary mask image, using the following formula:

[0096]

[0097] in: Represents a global latent variable. For noise in the mask area, This represents global noise in the non-masked region. For binary mask images, This represents element-wise multiplication;

[0098] Step S2.4: Repair result generation: The first frame image output by the above process presents a defect-free clean equipment surface, while maintaining the same viewing angle, lighting and background environment as the last frame image.

[0099] In the above scheme, the implementation of the timing generation module includes:

[0100] Step S3.1: Input the reconstructed first frame image and the original last frame image into the temporal generation model, wherein the temporal generation model is Wan2.1;

[0101] Step S3.2: Constrain the spatial path of defect evolution through text prompts, the text prompts describing the physical evolution law of the defect, such as "oil droplets slowly seep out from the weld and meander downwards along the pipeline";

[0102] Step S3.3: Based on the text prompt-guided timing generation model, generate a dynamic defect evolution video sequence that conforms to physical laws. For example, for transformer oil leakage, generate a continuous frame sequence of oil droplets slowly seeping out from the weld and meandering downwards along the pipeline.

[0103] In the above scheme, the implementation of the data augmentation processing module includes:

[0104] Step S4.1: Perform optical flow consistency detection on the synthesized video and filter out video segments with temporal coherence; wherein, the optical flow consistency detection includes:

[0105] Calculate the forward optical flow: the optical flow from the current frame t to the next frame t+1, representing the displacement of each pixel in the current frame:

[0106]

[0107] It is the position of the pixel in the current frame. and It represents the horizontal and vertical displacement of the pixel.

[0108] Calculate the reverse optical flow: the optical flow from the next frame t+1 to the current frame t, representing the displacement of each pixel in the next frame:

[0109]

[0110] and These represent the horizontal and vertical displacements of the pixel in the next frame, respectively.

[0111] Calculate the consistency error: for each pixel The consistency error is calculated using forward and reverse optical flow.

[0112]

[0113] This means that the result of the forward optical flow is traced in reverse using the reverse optical flow, and the resulting optical flow should match the forward optical flow. Represents the Euclidean norm, used to calculate the distance between the forward and reverse optical flows;

[0114] Calculate the overall consistency error: a comprehensive evaluation of the optical flow consistency across the entire video, calculated as follows:

[0115]

[0116] in, It is the total number of pixels in the image. It is the optical flow consistency error of each pixel;

[0117] Consistency judgment: based on overall consistency error Set a threshold To determine whether a video has continuity, if If the motion is below a certain threshold, the video is considered to have consistency; otherwise, it is considered to have disjointed motion.

[0118] Step S4.2: Perform frame interpolation on the selected sequential video segments to increase the video duration; in particular, use repeating frames or frame interpolation algorithms to process the "drop → convergence" stage, and reduce the playback speed of the segment to 1 / 8 to 1 / 4 of the original speed by time stretching, so that the total duration is extended to 30 to 120 seconds.

[0119] Step S4.3: Extract frames from the processed video to obtain continuous oil leakage image data, which is then used for testing and training the defect detection model.

[0120] Based on the technical solution provided by this invention, the core problem in substation defect data enhancement is solved through the following innovative methods, and significant results are achieved. Specific beneficial effects are analyzed as follows:

[0121] 1. The present invention solves the technical problem in the prior art that the generated image does not match the real scene due to the inconsistency of viewpoint, lighting and background by using the technical means described in step S2 (i.e., the first frame repair is performed by using the diffusion model that integrates the ControlNet multi-constraint mechanism based on the depth map, edge contour map and defect mask annotation of the tail frame image). It achieves the effect of generating a defect-free first frame image that is strictly aligned with the tail frame image.

[0122] In existing technologies, static image restoration methods (such as single-frame generation based on GANs or diffusion models) often lack multimodal structural constraints, leading to discontinuities in viewpoint, lighting, or texture between the restored area and the original background. This invention introduces depth maps and edge contour maps as structural guidance information (step S1.3), combines this with defect masking to accurately locate the restoration area (step S1.2), and jointly constrains depth features, edge features, and text prompt features during the reverse denoising process of the diffusion model (step S2.2), ensuring the restoration process adheres to the spatial structure and semantic consistency of the original image. Specifically, the depth map preserves the three-dimensional geometric relationships of the device, the edge contour map maintains the continuity of surface details, and the text prompt (such as "clean the device surface") further enhances semantic rationality. Through a masked region restoration control mechanism (step S2.3), noise optimization is performed only within the masked area, while global noise is retained in non-masked areas. This ensures that the restored first frame image (step S2.4) seamlessly inherits the viewpoint, lighting, and background environment of the last frame while eliminating defects, thus providing a strictly aligned input basis for subsequent temporal generation.

[0123] 2. The present invention solves the technical problem that existing technologies are unable to simulate the physical evolution law and temporal continuity of dynamic defects by means of the technical means described in step S3 (that is, inputting the repaired first frame image and the original last frame image into the time sequence generation model, and constraining the spatial path and appearance features of defect evolution through text prompts), and achieves the effect of generating a dynamic defect evolution video sequence that conforms to the physical causality of the equipment.

[0124] Traditional data augmentation methods (such as static image generation) often produce video sequences with abrupt motion changes or unreasonable paths because they ignore the temporal dimension and physical laws of defect evolution. This invention utilizes the strict alignment of the first and last frames (achieved in step S2) to construct an evolutionary path from a defect-free state to a complete defect state in the temporal generation model (step S3.1). Text prompts (step S3.2) describe specific physical laws (e.g., "oil droplets slowly seep from the weld and meander downwards along the pipe"), transforming the abstract evolutionary process into explicit constraints on spatial paths and apparent features. During inter-frame interpolation, the text prompt guides the model to prioritize matching the described spatial trajectory (e.g., the directionality of oil droplet diffusion along the pipe) and apparent changes (e.g., the gradual characteristics of oil stain diffusion), enabling the generated video sequence (step S3.3) to simulate the causal development logic of real defects in the temporal dimension (e.g., the continuity of oil leakage spreading from the starting point to the convergence point), thereby solving the problem of generated content deviating from physical reality.

[0125] 3. The present invention solves the technical problem of low data usability caused by non-continuous motion in the generated video by means of the technical means described in step S4 (i.e., optical flow consistency detection of dynamic defect evolution video sequence, screening of temporally coherent video segments, and frame interpolation processing of key evolution stages), and achieves the effect of outputting high-quality, temporally smooth enhanced defect data.

[0126] Synthetic videos often suffer from frame jumps or distortions, reducing their effectiveness in training defect detection models. This invention quantifies the temporal coherence of video through optical flow consistency detection (step S4.1): forward optical flow calculates pixel displacement paths, backward optical flow verifies displacement reversibility, and consistency error reflects the stability of the motion trajectory. The selection process is based on overall error assessment, retaining only segments with low errors to ensure that defect evolution (such as oil droplet movement) has no abrupt changes on the timeline. Further, frame interpolation (step S4.2) adds transition frames at key evolution stages (such as "droplet → convergence"), combined with time stretching to extend the total duration, enhancing the visual details of defect development with a slow-motion effect. This process not only eliminates temporal discontinuities but also improves data resolvability, allowing the final enhanced data (step S4.3) to be directly used to train the defect detection model, improving its robustness in identifying dynamic defects.

[0127] In summary, this invention systematically solves the industry problem of scarce and discontinuous substation defect data by using a collaborative mechanism of multimodal constraint repair, property-guided time series generation, and optical flow optimization screening, providing a high-fidelity and physically reasonable training data foundation for intelligent monitoring models. Attached Figure Description

[0128] Figure 1 This is the overall flowchart;

[0129] Figure 2 To generate an example diagram. Detailed Implementation

[0130] The embodiments of the present invention will be described in detail below. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, any modifications or equivalent substitutions made to the present invention should be covered within the scope of the claims of the present invention.

[0131] Furthermore, to better illustrate the present invention, numerous specific details are set forth in the following detailed embodiments. Those skilled in the art will understand that the present invention can be practiced without these specific details.

[0132] This invention provides a method and system for enhancing substation defect data based on first-frame restoration and video generation. This technical solution aims to address the problems of scarce substation defect data and poor temporal continuity of defect evolution in existing technologies. By accurately restoring key frames (first and last frames) of defect evolution and constructing a dynamic defect video with temporal coherence, the quality of training data for the substation defect identification system is greatly improved, thereby enhancing the accuracy and robustness of defect identification.

[0133] Example 1

[0134] The present invention consists of the following steps, and the detection flowchart is as follows: Figure 1 As shown:

[0135] Step S1: Obtain high-quality tail frame image, depth map, edge map, and defect mask annotation.

[0136] Step S1.1: Defect tail frame image acquisition and screening

[0137] Discrete images of substation equipment are acquired from the existing inspection system, and images containing complete defect states (such as large-area oil leakage in transformers) are manually selected as tail frame samples. The selection criteria need to be determined according to the actual inspection requirements, such as the main body of the equipment completely covering ≥30% of the image, uniform lighting (histogram variance <0.1), and clearly visible defect areas (contrast >40%).

[0138] Step S1.2: Precise labeling of defect masks

[0139] Pixel-level annotation tools (such as labelme and CVAT) are used to annotate the defective areas (oil leakage areas) in the last frame as masks for the areas to be repaired, generating a binary mask image. In the binary mask image, the pixels that mark the repaired areas that need to be modified are represented by 1 (white), and the pixels that retain the areas are represented by 0 (black).

[0140] Step S1.3: Extraction of multimodal structure guidance information

[0141] A depth map is extracted from the last frame image using a depth estimation model (such as DepthAnything), and an edge contour map (Canny Edge) is extracted from the last frame image using an adaptive threshold Canny algorithm. Both the depth map and edge map are then scaled to the same resolution as the last frame. The edge map extraction uses the adaptive threshold Canny algorithm, and its mathematical formula is as follows:

[0142] Gaussian filtering:

[0143]

[0144]

[0145] Gradient calculation:

[0146]

[0147]

[0148] Nonmaximum suppression:

[0149]

[0150] Dual threshold filtering:

[0151]

[0152] in, The standard deviation of the Gaussian kernel (default value σ=1.4). , Adaptive calculation is used to determine high and low thresholds. get, It represents the multiplication operation.

[0153] Step S2: Precise first-frame restoration based on multi-source condition control

[0154] The tail frame (base image to be repaired), defect mask (binary mask image), depth map, and edge map obtained in step S1 are input into a diffusion model that integrates ControlNet multi-constraint mechanisms (such as Stable Diffusion or Flux models). The mask area is the repair target, and the depth map and edge map are the structural constraints. The final output first frame image presents a defect-free clean equipment surface—such as a reflective metal substrate after oil stains have been removed—while maintaining the same viewpoint, lighting, and background environment as the tail frame. Note that since only the mask area needs to be repaired, the area to be repaired and a certain range of pixels around it in the tail frame will be cut out. This step will generate a depth map and a Canny contour map from the cut-out image, and after the repair is completed, the restored cut-out image will be restored (image replacement) to the original tail frame, thus obtaining the repaired first frame image.

[0155] In the reverse denoising process of the basic diffusion model, a depth map, a Canny edge map, and a text prompt are introduced as joint constraints. The text prompt can use broad content, such as "clean device surface." The reverse diffusion formula is as follows:

[0156]

[0157] Repair within the area of ​​the binary mask image:

[0158]

[0159] in: Representing the global latent variable, it is the noise state of the binary mask image composed of the masked regions at time step t. This represents the noise predicted by the U-Net network. This represents the predefined noise attenuation coefficient at time step t. , Representing depth map features, Representing edge map features, This indicates text prompt features. This represents standard normally distributed noise. This represents the noise scaling factor, which controls the intensity of random noise, and is typically set to a value of [value missing]. , Indicates the use of binary mask maps noise, Except for the masked area to be repaired, retain the global noise of the original image. , This indicates the base map to be repaired. By adding noise to time step t through the forward noise addition process of the diffusion model, the global noise is obtained. Standard Gaussian noise is retained outside the mask of the area to be repaired. , This represents the initial standard Gaussian noise. This indicates the image has been repaired.

[0160] Step S3: Synthesis of dynamic defect evolution video

[0161] The reconstructed first frame and the original last frame are input into the time-series generation model (such as Wan2.1). The model is guided to generate an evolution process that approximates the physical laws by using text prompts to constrain the spatial path of defect evolution (such as "oil droplets slowly seep out from the weld and meander downwards along the pipeline"). For example, for transformer oil leakage, a continuous frame sequence of oil droplets slowly seeping out from the weld and meandering downwards along the pipeline can be generated.

[0162] Step S4: Video optimization and enhancement data generation, such as... Figure 2 As shown;

[0163] Step S4.1: Perform optical flow consistency detection on the synthesized video and select consistent videos with temporal coherence.

[0164] For the generated video, optical flow is the motion vector of each pixel in the image, representing the displacement of each pixel over time. This method requires calculating the optical flow for each pair of adjacent frames, specifically forward optical flow, backward optical flow, consistency error, and overall consistency error. Based on a preset error threshold, it quickly determines whether the generated video achieves the expected smoothness and realism in motion. The calculation formula is as follows:

[0165] 1) Forward optical flow: The optical flow from the current frame t to the next frame t+1, representing the displacement of each pixel in the current frame.

[0166]

[0167] in, It is the position of the pixel in the current frame. and It represents the displacement of the pixel in the horizontal and vertical directions.

[0168] 2) Reverse optical flow: The optical flow from the next frame t+1 to the current frame t, representing the displacement of each pixel in the next frame.

[0169]

[0170] same, and These represent the horizontal and vertical displacements of the pixel in the next frame, respectively.

[0171] Calculate the consistency error: for each pixel The consistency error is calculated using forward and reverse optical flow.

[0172]

[0173] here, This means that the result of the forward optical flow is traced in reverse through the reverse optical flow, and the resulting optical flow should match the forward optical flow. This represents the Euclidean norm, used to calculate the distance between the forward and reverse optical flows. A smaller value indicates better consistency between the forward and reverse optical flows.

[0174] If consistency error A larger value indicates inconsistent motion at that pixel.

[0175] The overall consistency error is calculated as a comprehensive assessment of the optical flow consistency of the entire video. It can be calculated in the following ways:

[0176]

[0177] in, It is the total number of pixels in the image. It is the optical flow consistency error of each pixel.

[0178] Consistency judgment: based on overall consistency error A threshold can be set. To determine whether a video has continuity. If If the motion is below a certain threshold, the video is considered consistent; otherwise, it is considered to contain disjointed motion. This applies to general video generation tasks. A certain degree of optical flow error is allowed. By adjusting the threshold, the sensitivity of consistency detection can be controlled, ensuring that the generated video achieves the expected results in terms of motion smoothness and realism.

[0179] Step S4.2: Add video duration by interpolating video frames.

[0180] The “drip → convergence” stage of the video obtained in step S4.1 is processed by using repeated frames or frame interpolation algorithms (such as RIFE, DAIN). The playback speed of this segment is reduced to 1 / 8 to 1 / 4 of the original speed by time stretching, to 30 to 120 seconds, so that the total duration is extended to 30 to 120 seconds. The slow motion visual effect (non-physical process simulation) is achieved by combining frame interpolation and time stretching.

[0181] Step S4.3: Defect Detection Model Validation and Data Output

[0182] By performing frame-by-frame processing on the video obtained in step S4.2, coherent and consistent oil leakage image data can be obtained. This frame-by-frame image data and video footage can be used to test, train, and optimize existing oil leakage algorithm models (such as YOLO and background modeling), significantly increasing the amount of substation oil leakage data.

[0183] Example 2

[0184] This invention also provides a substation defect data enhancement system based on first and last frame video generation, comprising:

[0185] Image acquisition module: Acquires defect tail frame images of substation equipment, and extracts the depth map, edge contour map and defect mask annotation of the tail frame images;

[0186] First frame repair module: Based on the depth map, edge contour map and defect mask annotation of the last frame image, the first frame repair module performs first frame repair on the last frame image by using a diffusion model that integrates ControlNet multi-constraint mechanism to generate a defect-free first frame image with strict consistency in viewpoint, lighting and background with the last frame image.

[0187] The timing generation module inputs the repaired first frame image and the original last frame image into the timing generation model, and generates a dynamic defect evolution video sequence by constraining the spatial path and appearance features of defect evolution through text prompts.

[0188] Data enhancement processing module: Performs optical flow consistency detection on the dynamic defect evolution video sequence, filters out temporally coherent video segments, and performs frame interpolation on key evolution stages to extend the video duration, generating enhanced defect data.

[0189] In the above scheme, the implementation of the image acquisition module includes:

[0190] Step S1.1: Defect tail frame image acquisition and screening:

[0191] Discrete images of substation equipment are acquired from the existing inspection system, and images containing complete defect states are manually selected as tail frame samples. The selection criteria include: the main body of the equipment completely covers ≥30% of the image, uniform lighting, histogram variance <0.1, clearly visible defect areas, and contrast >40%.

[0192] Step S1.2: Precise labeling of defect masks:

[0193] A pixel-level annotation tool is used to annotate the defective areas in the tail frame image to generate a binary mask image; wherein, in the binary mask image, pixels marking the repaired areas that need to be modified are represented by 1 (white), and pixels marking the preserved areas are represented by 0 (black);

[0194] Step S1.3: Extraction of multimodal structure guidance information:

[0195] A depth map is extracted from the tail frame image using a depth estimation model, and an edge contour map (Canny Edge) is extracted from the tail frame image using an adaptive threshold Canny algorithm; wherein the mathematical process of the adaptive threshold Canny algorithm includes:

[0196] Gaussian filtering:

[0197]

[0198]

[0199] in The standard deviation of the Gaussian kernel is represented. Represents the input image matrix. The asterisk (*) represents the natural constant, and the asterisk (*) represents the convolution operation.

[0200] Gradient calculation:

[0201]

[0202]

[0203] in This represents the image matrix after Gaussian filtering. Represents the gradient magnitude matrix. Represents the gradient direction matrix (edge ​​angle);

[0204] Nonmaximum suppression:

[0205]

[0206] Dual threshold filtering:

[0207]

[0208] in, The standard deviation of the Gaussian kernel. , To adapt to high and low thresholds, adaptive calculation is used. get, It represents the multiplication operation.

[0209] In the above scheme, the implementation of the first frame repair module includes:

[0210] Step S2.1: Multi-source conditional input construction: The tail frame image, defect mask, depth map and edge map are input into the diffusion model of ControlNet multi-constraint mechanism;

[0211] Step S2.2: Backdiffusion Process Control: In the backdiffusion denoising process of the basic diffusion model, a depth map, a Canny edge map, and text prompts are introduced as joint constraints; wherein, the backdiffusion formula is:

[0212]

[0213] This is the restored image after backdiffusion. For time step Noise attenuation coefficient, defined as , Indicates time step Predefined noise attenuation coefficient, This represents the noise predicted by the U-Net network, whose input includes time steps. Latent variables and constraint-condition depth map features Edge map features Text prompt features , This is the noise scaling factor. , This is standard normally distributed noise;

[0214] Step S2.3: Mask Region Repair Control: Repair is performed within the region of the binary mask image, using the following formula:

[0215]

[0216] in: Represents a global latent variable. For noise in the mask area, This represents global noise in the non-masked region. For binary mask images, This represents element-wise multiplication;

[0217] Step S2.4: Repair result generation: The first frame image output by the above process presents a defect-free clean equipment surface, while maintaining the same viewing angle, lighting and background environment as the last frame image.

[0218] In the above scheme, the implementation of the timing generation module includes:

[0219] Step S3.1: Input the reconstructed first frame image and the original last frame image into the temporal generation model, wherein the temporal generation model is Wan2.1;

[0220] Step S3.2: Constrain the spatial path of defect evolution through text prompts, the text prompts describing the physical evolution law of the defect, such as "oil droplets slowly seep out from the weld and meander downwards along the pipeline";

[0221] Step S3.3: Based on the text prompt-guided timing generation model, generate a dynamic defect evolution video sequence that conforms to physical laws. For example, for transformer oil leakage, generate a continuous frame sequence of oil droplets slowly seeping out from the weld and meandering downwards along the pipeline.

[0222] In the above scheme, the implementation of the data augmentation processing module includes:

[0223] Step S4.1: Perform optical flow consistency detection on the synthesized video and filter out video segments with temporal coherence; wherein, the optical flow consistency detection includes:

[0224] Calculate the forward optical flow: the optical flow from the current frame t to the next frame t+1, representing the displacement of each pixel in the current frame:

[0225]

[0226] It is the position of the pixel in the current frame. and It represents the horizontal and vertical displacement of the pixel.

[0227] Calculate the reverse optical flow: the optical flow from the next frame t+1 to the current frame t, representing the displacement of each pixel in the next frame:

[0228]

[0229] and These represent the horizontal and vertical displacements of the pixel in the next frame, respectively.

[0230] Calculate the consistency error: for each pixel The consistency error is calculated using forward and reverse optical flow.

[0231]

[0232] This means that the result of the forward optical flow is traced in reverse using the reverse optical flow, and the resulting optical flow should match the forward optical flow. Represents the Euclidean norm, used to calculate the distance between the forward and reverse optical flows;

[0233] Calculate the overall consistency error: a comprehensive evaluation of the optical flow consistency across the entire video, calculated as follows:

[0234]

[0235] in, It is the total number of pixels in the image. It is the optical flow consistency error of each pixel;

[0236] Consistency judgment: based on overall consistency error Set a threshold To determine whether a video has continuity, if If the motion is below a certain threshold, the video is considered to have consistency; otherwise, it is considered to have disjointed motion.

[0237] Step S4.2: Perform frame interpolation on the selected sequential video segments to increase the video duration; in particular, use repeating frames or frame interpolation algorithms to process the "drop → convergence" stage, and reduce the playback speed of the segment to 1 / 8 to 1 / 4 of the original speed by time stretching, so that the total duration is extended to 30 to 120 seconds.

[0238] Step S4.3: Extract frames from the processed video to obtain continuous oil leakage image data, which is then used for testing and training the defect detection model.

Claims

1. A method for enhancing substation defect data based on video generated from the first and last frames, characterized in that, Includes the following steps: Step S1: Obtain the defect tail frame image of the substation equipment, and extract the depth map, edge contour map and defect mask annotation of the tail frame image; Step S2: Based on the depth map, edge contour map and defect mask annotation of the tail frame image, the first frame repair is performed on the tail frame image by a diffusion model that integrates ControlNet multi-constraint mechanism to generate a defect-free first frame image with strict consistency in viewpoint, lighting and background with the tail frame image. Step S3: Input the repaired first frame image and the original last frame image into the time-series generation model, and generate a dynamic defect evolution video sequence by constraining the spatial path and appearance features of defect evolution through text prompts; Step S4: Perform optical flow consistency detection on the dynamic defect evolution video sequence, select temporally coherent video segments, and perform frame interpolation on key evolution stages to extend the video duration and generate enhanced defect data.

2. The method according to claim 1, characterized in that, Step S1 includes: Step S1.1: Defect tail frame image acquisition and screening: Discrete images of substation equipment are acquired from the existing inspection system, and images containing complete defect states are manually selected as tail frame samples. The selection criteria include: the main body of the equipment completely covers ≥30% of the image, uniform lighting, histogram variance <0.1, clearly visible defect areas, and contrast >40%. Step S1.2: Precise labeling of defect masks: A pixel-level annotation tool is used to annotate the defective areas in the tail frame image to generate a binary mask image; wherein, in the binary mask image, pixels marking the repaired areas that need to be modified are represented by 1 (white), and pixels marking the preserved areas are represented by 0 (black); Step S1.3: Extraction of multimodal structure guidance information: A depth estimation model is used to extract a depth map from the tail frame image, and an adaptive threshold Canny algorithm is used to extract an edge contour map from the tail frame image. The mathematical process of the adaptive threshold Canny algorithm includes: Gaussian filtering: in The standard deviation of the Gaussian kernel is represented. Represents the input image matrix. The asterisk (*) represents the natural constant, and the asterisk (*) represents the convolution operation. Gradient calculation: in This represents the image matrix after Gaussian filtering. Represents the gradient magnitude matrix. Represents the gradient direction matrix; Nonmaximum suppression: Dual threshold filtering: in, The standard deviation of the Gaussian kernel. , To adapt to high and low thresholds, adaptive calculation is used. get, It represents the multiplication operation.

3. The method according to claim 1, characterized in that, Step S2 includes: Step S2.1: Multi-source conditional input construction: The tail frame image, defect mask, depth map and edge map are input into the diffusion model of ControlNet multi-constraint mechanism; Step S2.2: Backdiffusion Process Control: In the backdiffusion denoising process of the basic diffusion model, a depth map, a Canny edge map, and text prompts are introduced as joint constraints; wherein, the backdiffusion formula is: This is the restored image after backdiffusion. For time step Noise attenuation coefficient, defined as , Indicates time step Predefined noise attenuation coefficient, This represents the noise predicted by the U-Net network, whose input includes time steps. Latent variables and constraint-condition depth map features Edge map features Text prompt features , This is the noise scaling factor. , This is standard normally distributed noise; Step S2.3: Mask Region Repair Control: Repair is performed within the region of the binary mask image, using the following formula: in: Represents a global latent variable. For noise in the mask area, This represents global noise in the non-masked region. For binary mask images, This represents element-wise multiplication; Step S2.4: Repair result generation: The first frame image output by the above process presents a defect-free clean equipment surface, while maintaining the same viewing angle, lighting and background environment as the last frame image.

4. The method according to claim 1, characterized in that, Step S3 includes: Step S3.1: Input the reconstructed first frame image and the original last frame image into the temporal generation model, wherein the temporal generation model is Wan2.1; Step S3.2: Constrain the spatial path of defect evolution through text prompts, wherein the text prompts describe the physical evolution law of the defect; Step S3.3: Based on the text prompt-guided timing generation model, generate a dynamic defect evolution video sequence that conforms to physical laws.

5. The method according to claim 1, characterized in that, Step S4 includes: Step S4.1: Perform optical flow consistency detection on the synthesized video and filter out video segments with temporal coherence; wherein, the optical flow consistency detection includes: Calculate the forward optical flow: the optical flow from the current frame t to the next frame t+1, representing the displacement of each pixel in the current frame: It is the position of the pixel in the current frame. and It represents the horizontal and vertical displacement of the pixel. Calculate the reverse optical flow: the optical flow from the next frame t+1 to the current frame t, representing the displacement of each pixel in the next frame: and These represent the horizontal and vertical displacements of the pixel in the next frame, respectively. Calculate the consistency error: for each pixel The consistency error is calculated using forward and reverse optical flow. This means that the result of the forward optical flow is traced in reverse using the reverse optical flow, and the resulting optical flow should match the forward optical flow. Represents the Euclidean norm, used to calculate the distance between the forward and reverse optical flows; Calculate the overall consistency error: a comprehensive evaluation of the optical flow consistency across the entire video, calculated as follows: in, It is the total number of pixels in the image. It is the optical flow consistency error of each pixel; Consistency judgment: based on overall consistency error Set a threshold To determine whether a video has continuity, if If the motion is below a certain threshold, the video is considered to have consistency; otherwise, the video is considered to have disjointed motion. Step S4.2: Perform frame interpolation on the selected sequential video segments to increase the video duration; in particular, use repeating frames or frame interpolation algorithms to process the "drop → convergence" stage, and reduce the playback speed of the segment to 1 / 8 to 1 / 4 of the original speed by time stretching, so that the total duration is extended to 30 to 120 seconds. Step S4.3: Extract frames from the processed video to obtain continuous oil leakage image data, which is then used for testing and training the defect detection model.

6. A substation defect data enhancement system based on first and last frame video generation, characterized in that, include: Image acquisition module: Acquires defect tail frame images of substation equipment, and extracts the depth map, edge contour map and defect mask annotation of the tail frame images; First frame repair module: Based on the depth map, edge contour map and defect mask annotation of the last frame image, the first frame repair module performs first frame repair on the last frame image by using a diffusion model that integrates ControlNet multi-constraint mechanism to generate a defect-free first frame image with strict consistency in viewpoint, lighting and background with the last frame image. The timing generation module inputs the repaired first frame image and the original last frame image into the timing generation model, and generates a dynamic defect evolution video sequence by constraining the spatial path and appearance features of defect evolution through text prompts. Data enhancement processing module: Performs optical flow consistency detection on the dynamic defect evolution video sequence, filters out temporally coherent video segments, and performs frame interpolation on key evolution stages to extend the video duration, generating enhanced defect data.

7. The system according to claim 6, characterized in that, The implementation of the image acquisition module includes: Step S1.1: Defect tail frame image acquisition and screening: Discrete images of substation equipment are acquired from the existing inspection system, and images containing complete defect states are manually selected as tail frame samples. The selection criteria include: the main body of the equipment completely covers ≥30% of the image, uniform lighting, histogram variance <0.1, clearly visible defect areas, and contrast >40%. Step S1.2: Precise labeling of defect masks: A pixel-level annotation tool is used to annotate the defective areas in the tail frame image to generate a binary mask image; wherein, in the binary mask image, pixels marking the repaired areas that need to be modified are represented by 1 (white), and pixels marking the preserved areas are represented by 0 (black); Step S1.3: Extraction of multimodal structure guidance information: A depth estimation model is used to extract a depth map from the tail frame image, and an adaptive threshold Canny algorithm is used to extract an edge contour map from the tail frame image; wherein, the mathematical process of the adaptive threshold Canny algorithm includes: Gaussian filtering: in The standard deviation of the Gaussian kernel is represented. Represents the input image matrix. The asterisk (*) represents the natural constant, and the asterisk (*) represents the convolution operation. Gradient calculation: in This represents the image matrix after Gaussian filtering. Represents the gradient magnitude matrix. Represents the gradient direction matrix; Nonmaximum suppression: Dual threshold filtering: in, The standard deviation of the Gaussian kernel. , To adapt to high and low thresholds, adaptive calculation is used. get, It represents the multiplication operation.

8. The system according to claim 6, characterized in that, The implementation of the first frame repair module includes: Step S2.1: Multi-source conditional input construction: The tail frame image, defect mask, depth map and edge map are input into the diffusion model of ControlNet multi-constraint mechanism; Step S2.2: Backdiffusion Process Control: In the backdiffusion denoising process of the basic diffusion model, a depth map, a Canny edge map, and text prompts are introduced as joint constraints; wherein, the backdiffusion formula is: This is the restored image after backdiffusion. For time step Noise attenuation coefficient, defined as , Indicates time step Predefined noise attenuation coefficient, This represents the noise predicted by the U-Net network, whose input includes time steps. Latent variables and constraint-condition depth map features Edge map features Text prompt features , This is the noise scaling factor. , This is standard normally distributed noise; Step S2.3: Mask Region Repair Control: Repair is performed within the region of the binary mask image, using the following formula: in: Represents a global latent variable. For noise in the mask area, This represents global noise in the non-masked region. For binary mask images, This represents element-wise multiplication; Step S2.4: Repair result generation: The first frame image output by the above process presents a defect-free clean equipment surface, while maintaining the same viewing angle, lighting and background environment as the last frame image.

9. The system according to claim 6, characterized in that, The implementation of the timing generation module includes: Step S3.1: Input the reconstructed first frame image and the original last frame image into the temporal generation model, wherein the temporal generation model is Wan2.1; Step S3.2: Constrain the spatial path of defect evolution through text prompts, wherein the text prompts describe the physical evolution law of the defect; Step S3.3: Based on the text prompt-guided timing generation model, generate a dynamic defect evolution video sequence that conforms to physical laws.

10. The system according to claim 6, characterized in that, The implementation of the data augmentation processing module includes: Step S4.1: Perform optical flow consistency detection on the synthesized video and filter out video segments with temporal coherence; wherein, the optical flow consistency detection includes: Calculate the forward optical flow: the optical flow from the current frame t to the next frame t+1, representing the displacement of each pixel in the current frame: It is the position of the pixel in the current frame. and It represents the horizontal and vertical displacement of the pixel. Calculate the reverse optical flow: the optical flow from the next frame t+1 to the current frame t, representing the displacement of each pixel in the next frame: and These represent the horizontal and vertical displacements of the pixel in the next frame, respectively. Calculate the consistency error: for each pixel The consistency error is calculated using forward and reverse optical flow. This means that the result of the forward optical flow is traced in reverse using the reverse optical flow, and the resulting optical flow should match the forward optical flow. Represents the Euclidean norm, used to calculate the distance between the forward and reverse optical flows; Calculate the overall consistency error: a comprehensive evaluation of the optical flow consistency across the entire video, calculated as follows: in, It is the total number of pixels in the image. It is the optical flow consistency error of each pixel; Consistency judgment: based on overall consistency error Set a threshold To determine whether a video has continuity, if If the motion is below a certain threshold, the video is considered to have consistency; otherwise, the video is considered to have disjointed motion. Step S4.2: Perform frame interpolation on the selected sequential video segments to increase the video duration; in particular, use repeating frames or frame interpolation algorithms to process the "drop → convergence" stage, and reduce the playback speed of the segment to 1 / 8 to 1 / 4 of the original speed by time stretching, so that the total duration is extended to 30 to 120 seconds. Step S4.3: Extract frames from the processed video to obtain continuous oil leakage image data, which is then used for testing and training the defect detection model.

Images