Intelligent agent visual perception completion and denoising method and system for new energy station in extreme weather

Abstract

The application provides a kind of intelligent visual perception completion denoising method and system for new energy station under extreme weather, it is related to intelligent inspection visual enhancement technical field, by carrying out light path degradation separation processing to original visual data stream, each frame of defective visual frame is decomposed into light path degradation trace distribution graph and light path disturbance residual component graph, and according to this, adjacent frame light path degradation trace offset field is constructed, combined with pose record to generate light path degradation compensation frame sequence, with residual component graph is overlaid frame by frame to obtain preliminary visual repair frame sequence;Again, based on preliminary repair frame, construct interframe spatial context constraint field to carry out structure texture collaborative repair, generate complete visual repair frame sequence;Finally, according to the scene semantic coherence of adjacent repair frame, construct time domain consistency constraint flow, output final visual perception frame sequence, significantly improve the perception reliability and decision accuracy of new energy station intelligent inspection body in harsh environment.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection visual enhancement technology, and more specifically, to a method and system for intelligent agent visual perception completion and noise reduction under extreme weather conditions at new energy power stations. Background Technology

[0002] New energy power plants, such as photovoltaic power stations and wind farms, are typically deployed in open areas and are exposed to extreme weather conditions such as sandstorms, heavy rain, dense fog, and snow for extended periods. To ensure the safe operation of the equipment, these plants generally employ intelligent inspection systems equipped with visual sensors for automated inspections. However, under extreme weather conditions, suspended particles in the atmosphere, water vapor condensation, and drastic fluctuations in light intensity can cause problems such as large-area information loss, sharp drops in contrast, and noise coverage in the images captured by the visual sensors, severely limiting the reliability of subsequent target identification and fault diagnosis.

[0003] In existing technologies, visual enhancement processing under adverse weather conditions typically employs image dehazing algorithms or single-frame denoising models to independently repair each frame of the acquired images. These methods treat each frame as an independent sample, relying solely on pixel statistical features within the current frame for restoration, without considering the motion trajectory information of the agent during continuous inspection or the temporal correlations between frames. Furthermore, when dealing with image defects, existing methods often model degradation factors and noise factors together, resulting in blurred structural edges, loss of texture details, and flickering and semantic jumps during frame transitions, failing to meet the practical requirements of intelligent inspection agents for continuous and accurate visual perception. Summary of the Invention

[0004] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a method for intelligent agent visual perception completion and noise reduction under extreme weather conditions at new energy power stations, the method comprising: Receive raw visual data streams collected by the inspection agent in extreme weather conditions. The raw visual data streams contain a sequence of missing visual frames with consecutive frame numbers. Each missing visual frame in the sequence of missing visual frames carries the agent's pose record at the time of collection. The original visual data stream is subjected to optical path degradation separation processing, and each frame of the defective visual frame is decomposed into an optical path degradation trace distribution map and an optical path disturbance residual component map, and the intelligent agent pose recording sequence corresponding to the defective visual frame sequence is retained. Based on the optical path degradation trace distribution map, an inter-frame optical path degradation trace offset field is constructed. Based on the inter-frame optical path degradation trace offset field and the agent pose recording sequence, an optical path degradation compensation frame sequence is generated. The optical path degradation compensation frame sequence is superimposed frame by frame with the frame sequence corresponding to the optical path disturbance residual component map to obtain a preliminary visual repair frame sequence. Based on the preliminary visual restoration frame sequence, an intra-frame spatial context constraint field is constructed, and the intra-frame spatial context constraint field is used to perform structural and texture co-restoration on the preliminary visual restoration frame sequence to generate a complete visual restoration frame sequence. Based on the scene semantic coherence between adjacent repair frames in the complete visual repair frame sequence, a temporal consistency constraint flow is constructed. The temporal consistency constraint flow is then used to eliminate inter-frame flicker in the complete visual repair frame sequence, generating the final visual perception output frame sequence.

[0005] Furthermore, embodiments of the present invention also provide an intelligent agent visual perception completion and noise reduction system for new energy power stations under extreme weather conditions, comprising: A processor; a machine-readable storage medium for storing machine-executable instructions of the processor; wherein the processor is configured to execute the above-described intelligent agent visual perception completion and denoising method for new energy power stations under extreme weather conditions by executing the machine-executable instructions.

[0006] Based on the above, by explicitly separating optical path degradation factors and noise disturbance factors in the raw visual data stream collected by the agent under extreme weather conditions, and constructing an optical path degradation trace offset field between adjacent frames using the agent's pose recording sequence, the generation of the optical path degradation compensation frame sequence can accurately compensate for the loss of structured information caused by atmospheric scattering and suspended particles, rather than relying on blind estimation based on statistical assumptions. This improves the realism and credibility of the repaired frames at the physical level. Furthermore, this invention constructs an intra-frame spatial context constraint field based on the preliminary visual repair frame sequence, realizing the collaborative repair of structural information and texture details, avoiding the edge blurring and detail distortion problems caused by the separation of structure restoration and texture restoration in traditional methods. At the same time, this invention introduces a temporal consistency constraint stream to eliminate inter-frame flicker in the fully visual repair frame sequence, and uses the scene semantic coherence between adjacent repair frames to constrain the inter-frame transition, solving the inherent inter-frame inconsistency problem of independent frame-by-frame repair. As a result, the final output visual perception frame sequence can still maintain structural integrity, clear texture, and coherent timing even under extreme weather conditions, significantly improving the perception reliability and decision-making accuracy of the intelligent inspection system for new energy power stations in harsh environments. Attached Figure Description

[0007] Figure 1 This is a schematic diagram of the execution flow of the intelligent agent visual perception completion and noise reduction method for new energy power stations under extreme weather conditions provided in an embodiment of the present invention.

[0008] Figure 2 This is a schematic diagram of the simulation results of the intelligent agent visual perception completion and denoising method for new energy power stations under extreme weather conditions provided in the embodiments of the present invention.

[0009] Figure 3 This is a schematic diagram of the main visual processing of the intelligent agent visual perception completion and noise reduction method for new energy power stations under extreme weather conditions provided in this embodiment of the invention. Detailed Implementation

[0010] Figure 1 This is a flowchart illustrating a method for visual perception completion and denoising of intelligent agents for new energy power stations under extreme weather conditions, provided by an embodiment of the present invention. It is applied to repairing degraded visual data collected by UAV intelligent agents performing inspection tasks at wind power stations under various extreme weather conditions, such as sandstorms. The method separates the degradation process into optical path degradation and random perturbation. Through multi-frame degradation trace tracking and intra-frame structural texture collaborative repair, a clear and temporally consistent visual perception output frame sequence is reconstructed. In this embodiment, the visual data acquisition is completed by legally authorized inspection UAVs performing routine inspection tasks within designated new energy power station areas, strictly adhering to airspace management regulations and data security laws. The acquired images do not contain any personally identifiable privacy information.

[0011] Step S110: Receive the raw visual data stream collected by the inspection agent in extreme weather conditions. The raw visual data stream contains a sequence of missing visual frames with consecutive frame numbers. Each missing visual frame in the sequence carries the agent's pose record at the time of collection.

[0012] In this embodiment, the inspection agent is a multi-rotor UAV equipped with an industrial camera, an inertial measurement unit, and a global positioning system (GPS) integrated navigation module. The UAV flies along a predetermined inspection route of the wind power station during a sandstorm. The industrial camera continuously acquires visual data at a fixed frame rate, simultaneously recording the UAV's pose at the moment of each acquired frame. The raw visual data stream is input as a sequence of time-series data packets. Each time-series data packet contains a missing visual frame image and a corresponding set of agent pose recording data. The missing visual frame image data is a two-dimensional pixel matrix, with each element storing the light intensity value at the corresponding pixel location. The agent pose recording data is an array containing spatial coordinates and attitude quaternions. All time-series data packets are arranged in ascending order of frame number to form a complete raw visual data stream.

[0013] Step S120: Perform optical path degradation separation processing on the original visual data stream, decompose each defective visual frame into an optical path degradation trace distribution map and an optical path disturbance residual component map, and retain the intelligent agent pose recording sequence corresponding to the defective visual frame sequence.

[0014] Step S121: extracting all frames of the defective visual frame sequence frame by frame from the original visual data stream, performing pixel brightness level decomposition on each defective visual frame to generate a brightness level component sequence, wherein the brightness level component sequence consists of a rough brightness base layer and progressive detailed brightness residual layers.

[0015] Traversing the defective visual frame sequence in the original visual data stream frame by frame, converting each defective visual frame to the YCbCr color space and extracting a luminance channel component, denoted as Y. Performing Laplacian pyramid decomposition on Y, with the number of pyramid layers preset as N, the topmost layer being the rough brightness base layer denoted as B0, and the remaining layers being progressive detailed brightness residual layers from coarse to fine, denoted as L1 to L N-1 . B0 reflects the global illumination distribution and large-scale brightness changes caused by dust scattering, and L1 to L N-1 retain multi-scale surface texture and local detail information.

[0016] Step S122: performing gradient direction histogram statistics on the rough brightness base layer to obtain a gradient direction distribution of the rough brightness base layer, and marking pixel regions with gradient amplitude lower than the median gradient amplitude in the gradient direction distribution of the rough brightness base layer as optical path degradation candidate regions.

[0017] Calculating a horizontal gradient G1 and a vertical gradient G2 for each pixel position of B0. Gradient amplitude , gradient direction angle . Counting the gradient direction distribution histogram of all pixel positions, wherein the total number of direction intervals of the histogram is 36, and each interval spans 10 degrees. Calculating the median A0 of the gradient amplitude of the whole image. Traversing each pixel position, marking pixels satisfying A < A0 as optical path degradation candidate region pixels to form a candidate region R1. Suspended particles in the sandstorm medium form degradation traces with low contrast and blurred edges in the image, and their gradient amplitude is low. Using the median gradient amplitude as an adaptive threshold can effectively screen out degradation candidate regions.

[0018] Step S123: performing optical path degradation direction field estimation on the optical path degradation candidate region, calculating the main optical path degradation direction angle at each pixel position within the optical path degradation candidate region, and forming an optical path degradation direction map.

[0019] For each pixel position in R1, taking a square local neighborhood window with side length W centered on the pixel position. Calculating the weighted main direction of the gradient direction distribution in the window as the main optical path degradation direction angle D1 of the pixel position, wherein the weight is the gradient amplitude A of each pixel. The weighted main direction is calculated by weighted summing the gradient vectors [G1, G2] of each pixel in the window according to A to obtain a weighted average gradient vector [U1, U2], then calculating . D1 of all pixels within R1 form an optical path degradation direction map M1. Driven by wind force, dust particles have a dominant movement direction, which is reflected in images as directionally distributed optical path degradation traces; this dominant direction can be estimated through weighted statistics of local neighborhood gradients.

[0020] Step S124: perform layer-by-layer direction consistency comparison between the optical path degradation direction map and the progressive detailed brightness residual layers, and extract detailed brightness residual components whose direction angle with the optical path degradation direction map is smaller than a direction angle threshold as fine-grained optical path degradation traces.

[0021] For each layer Lk in the progressive detailed brightness residual layer sequence, calculate the local gradient direction angle D2 at each pixel position in Lk. Calculate the included angle Δ=|D2-D1| between D2 and D1 at the corresponding pixel position in M1. If Δ<T1, where T1 is a preset direction angle threshold, retain the residual value at that pixel position in Lk as a fine-grained optical path degradation trace, denoted as H1. H1 retains the high-frequency texture degradation components that are consistent with the main direction of optical path degradation in the progressive detailed brightness residual layers, and achieves directional separation from randomly scattered dust particle noise that is inconsistent with the direction.

[0022] Step S125: perform component aggregation on the coarse brightness base layer component corresponding to the optical path degradation candidate region and the fine-grained optical path degradation traces to generate the optical path degradation trace distribution map.

[0023] Extract the brightness value of the region corresponding to R1 from B0, denoted as B1. Superimpose the brightness value of B1 and H1 at the corresponding spatial position pixel by pixel, and the superposition result constitutes the optical path degradation trace distribution map P1. P1 contains both large-scale brightness attenuation caused by dust scattering and fine-grained texture blurring consistent with the degradation direction, which completely characterizes the degradation effect caused by sandstorms on the imaging optical path.

[0024] Step S126: separate remaining brightness residual components that have not been aggregated into the optical path degradation trace distribution map from the brightness hierarchical component sequence, and reconstruct the remaining brightness residual components into the optical path disturbance residual component map.

[0025] Retain the region not marked as R1 in B0 as the normal brightness base layer B2. Retain residual components that have not been aggregated into H1 from L1 to LN-1 as the remaining detailed residual layers, denoted as L1r to LN-1r. Perform inverse transformation on B2 and L1r to LN-1r according to the Laplacian pyramid reconstruction rule to obtain the optical path disturbance residual component map P2 through reconstruction. P2 contains non-directional disturbance components such as normal scene textures, random scattering noise of dust particles and sensor thermal noise, and forms a complete decomposition with P1 that characterizes the directional degradation effect of sandstorms.

[0026] Step S127: Establish a mapping association between the frame number of each defective visual frame and the corresponding frame number of the intelligent agent pose record in the intelligent agent pose record sequence, and retain the intelligent agent pose record sequence.

[0027] For each frame in the defective visual frame sequence, using the frame number as the index key, the corresponding agent pose record with that frame number is searched in the agent pose record sequence to establish a mapping relationship and store it. This mapping relationship is used to provide inter-frame camera motion information during subsequent inter-frame optical path degradation compensation.

[0028] Step S130: Construct an inter-frame optical path degradation trace offset field based on the optical path degradation trace distribution map, generate an optical path degradation compensation frame sequence based on the inter-frame optical path degradation trace offset field and the agent pose recording sequence, and superimpose the optical path degradation compensation frame sequence with the frame sequence corresponding to the optical path disturbance residual component map frame by frame to obtain a preliminary visual repair frame sequence.

[0029] Step S131: Extract optical path degradation feature points from each frame of the optical path degradation trace distribution map sequence to obtain a set of optical path degradation feature points. Each optical path degradation feature point in the set carries the spatial coordinate position of the feature point in the optical path degradation trace distribution map.

[0030] For each frame P1 in the optical path degradation trace distribution map sequence, an accelerated segment test feature point detection algorithm is used to extract optical path degradation feature points. For each pixel position in P1, a circular neighborhood with radius r is taken centered on that pixel position. The brightness difference between each pixel on the circumference and the central pixel is compared. If the brightness difference of m consecutive pixels on the circumference exceeds the threshold T2, then the central pixel is marked as an optical path degradation feature point. Each optical path degradation feature point carries its pixel x-coordinate u and y-coordinate v in P1, forming the optical path degradation feature point set S1.

[0031] Step S132: Perform cross-frame feature point pairing on the optical path degradation feature point set of the distribution map of optical path degradation traces in two adjacent frames, calculate the coordinate displacement vector of the successfully paired cross-frame optical path degradation feature point pair, and construct the optical path degradation trace offset field between adjacent frames based on the coordinate displacement vector.

[0032] For two adjacent frames of optical path degradation trace distribution maps P1t and P1t+1, corresponding sets of optical path degradation feature points S1t and S1t+1 are selected. For each feature point in S1t, a fixed-size feature descriptor block centered on that feature point is extracted, and the gradient direction histogram of the feature descriptor block is calculated as the feature descriptor vector. In S1t+1, the feature point with the smallest Euclidean distance to the feature descriptor vector of each feature point in S1t is searched as a candidate matching point. If the ratio of the nearest distance to the second nearest distance is less than the threshold T3, the pairing is confirmed as successful. For each successfully paired feature point pair, the coordinate displacement vectors Δu=u2-u1 and Δv=v2-v1 are calculated. All successfully paired displacement vectors constitute the optical path degradation trace offset field V1 between adjacent frames. V1 records the image planar motion of the dust degradation traces between adjacent frames.

[0033] Step S133: Extract the relative pose change of the agent corresponding to adjacent frames from the agent pose recording sequence, and project the relative pose change of the agent onto the image plane of the defective visual frame to obtain the image plane projection pose change vector.

[0034] Extract pose records corresponding to adjacent frames t and t+1 from the agent's pose recording sequence, and calculate the rotation matrix R and translation vector T from frame t to frame t+1. Project the relative motion in 3D space onto the image plane using the camera intrinsic matrix K. For each pixel position in frame t, the projected pose change vector V2 = K·(R·X+T)-K·X, where X is the 3D spatial coordinate of the pixel in the camera coordinate system of frame t, calculated from the pixel's depth value and pixel coordinates through back projection. In the absence of depth values, estimate the depth value by interpolating the depth values ​​of neighboring pixels or by assuming the scene is a local plane.

[0035] Step S134: The optical path degradation trace offset field between adjacent frames is fused with the image plane projection pose change vector to obtain the fused optical path degradation compensation vector field, and the optical path degradation compensation intensity field is determined according to the amplitude of the fused optical path degradation compensation vector field.

[0036] For each pixel location, V1 is weighted and fused with V2 at that pixel location. The fused optical path degradation compensation vector V3 = w1·V1 + w2·V2, where w1 and w2 are preset fusion weights, and w1 + w2 = 1. The V3 values ​​at all pixel locations constitute the fused optical path degradation compensation vector field M2. The magnitude of the vector at each pixel location in M2 is then calculated. A1 is used as the optical path degradation compensation intensity at the pixel location to form the optical path degradation compensation intensity field M3.

[0037] Step S135: For each frame of the optical path degradation trace distribution map in the optical path degradation trace distribution map sequence, perform pixel optical path reverse migration along the compensation direction indicated by the fused optical path degradation compensation vector field and according to the compensation intensity indicated by the optical path degradation compensation intensity field to generate the optical path degradation compensation frame sequence.

[0038] For each frame P1t in the optical path degradation trace distribution map sequence, the spatial position of each pixel in P1t is reverse-shifted along the compensation direction indicated by M2 and according to the compensation intensity indicated by M3. The reverse shifting is achieved through inverse bilinear interpolation, and the shifted pixel value is taken from the pixel value at the corresponding shifted coordinate position in P1t. After all frames undergo pixel optical path reverse shifting, an optical path degradation compensation frame sequence is generated, denoted as Q1.

[0039] Step S136: Superimpose the pixel-by-pixel brightness values ​​of each optical path degradation compensation frame in the optical path degradation compensation frame sequence with the frame with the same frame number in the frame sequence corresponding to the optical path disturbance residual component map to obtain the corresponding frame in the preliminary visual restoration frame sequence.

[0040] For each frame Q1t in Q1, Q1t is superimposed with the corresponding frame number's optical path perturbation residual component map P2t, pixel by pixel brightness value. The superposition result is the corresponding frame R1t in the preliminary visual restoration frame sequence. R1t = Q1t + P2t. The R1t of all frames are arranged according to the frame number to form the preliminary visual restoration frame sequence.

[0041] Step S140: Construct an intra-frame spatial context constraint field based on the preliminary visual restoration frame sequence, and use the intra-frame spatial context constraint field to perform structural and texture co-restoration on the preliminary visual restoration frame sequence to generate a complete visual restoration frame sequence.

[0042] Step S141: Perform spatial texture structure decomposition on each preliminary visual restoration frame in the preliminary visual restoration frame sequence, and extract the texture layer and structure layer of the preliminary visual restoration frame. The texture layer contains the micro-texture information of the preliminary visual restoration frame, and the structure layer contains the macro-geometric boundary information of the preliminary visual restoration frame.

[0043] For each frame R1 in the initial visual restoration frame sequence, a total variational image decomposition algorithm is used to decompose the spatial texture structure. The target energy function of the decomposition includes a data fidelity term and a regularization term. The data fidelity term constrains the sum of the decomposed texture layer T1 and structure layer S1 to approximate the original R1, while the regularization term imposes a total variational constraint on the structure layer to maintain the sharpness of the boundaries. The minimization problem of this energy function is solved iteratively using the alternating direction multiplier method to obtain the texture layer T1 and the structure layer S1. T1 contains microscopic texture details, and S1 contains macroscopic geometric boundaries and segmented smooth regions.

[0044] Step S142: Perform geometric boundary coherence analysis on the structural layer, identify the geometric boundary fracture locations caused by extreme weather in the structural layer, and perform boundary breakpoint pairing at the geometric boundary fracture locations to obtain a set of fracture boundary pairing points.

[0045] Edge detection is performed on structural layer S1, using the Canni edge detection algorithm to extract edge pixel positions. All edge pixel positions are traversed to identify edge endpoints, which are edge pixels with only one or zero adjacent edge pixels in their 8-neighborhood. For each edge endpoint, the search area is extended along the direction of the edge containing that endpoint to find the nearest other edge endpoint whose directional difference is less than a threshold T4. These two endpoints are paired as break boundary pairing points. All break boundary pairing points constitute set S2.

[0046] Step S143: Based on the spatial distance and local boundary orientation of each pair of fracture boundary pairing points in the set of fracture boundary pairing points, construct a boundary coherence constraint field. The boundary coherence constraint field is used to indicate the boundary connection direction that should be restored at the fracture location of the geometric boundary.

[0047] For each pair of fracture boundary points in S2, calculate the spatial Euclidean distance d1 between the two points and the mean α1 of the local boundary orientation angles at the two points. Using the midpoint between the two points as the center, generate a boundary connectivity guiding vector V4 along the direction α1, with the magnitude of V4 being d1. Interpolate and diffuse V4 for all fracture boundary pairing points in the image space to generate a boundary coherence constraint field M4 covering all fracture locations.

[0048] Step S144: Perform texture block similarity search on the texture layer, search for similar texture blocks that match the damaged texture region in the undamaged texture region of the texture layer, and construct a texture block migration constraint field based on the spatial offset of the similar texture blocks.

[0049] For texture layer T1, mark the texture loss areas caused by boundary breaks. A texture block of size p×p, centered on the pixel position of the edge of the texture loss area, is selected as the block to be repaired. Within the undamaged texture area of ​​T1, a search window is slidable with a step size s. The sum of squared pixel brightness differences between the texture block and the block to be repaired at each search position is calculated. The texture block with the smallest sum of squared pixel brightness differences is selected as a similar texture block, and the spatial offset [Δu1, Δv1] of this similar texture block relative to the block to be repaired is recorded. The offsets of all blocks to be repaired constitute the texture block migration constraint field M5.

[0050] Step S145: Perform constraint fusion of the boundary coherence constraint field and the texture block migration constraint field to form the intra-frame spatial context constraint field.

[0051] Constraint fusion is performed on M4 and M5. The fusion method is as follows: for each pixel position, if the position belongs to a boundary break region, the constraint vector of M4 is used; if it belongs to a texture missing region, the constraint vector of M5 is used; if it belongs to both types of regions, the average of the vectors of M4 and M5 is taken. The fused field is the intra-frame spatial context constraint field M6.

[0052] Step S146: Using the intra-frame spatial context constraint field, the broken boundaries and damaged texture regions of each preliminary visual repair frame in the preliminary visual repair frame sequence are synchronously repaired, and the boundary connection direction and similar texture blocks are migrated and filled to the corresponding positions to generate the complete visual repair frame sequence.

[0053] For each frame R1, based on the constraint vector of each pixel position in M6, boundary pixel interpolation is performed on the broken boundary region along the boundary connection guidance direction, and texture content is copied from similar texture block positions to fill the texture missing region. After the repair is completed, the corresponding frame F1 in the complete visual repair frame sequence is obtained, and the F1 of all frames constitutes the complete visual repair frame sequence.

[0054] Step S150: Construct a temporal consistency constraint flow based on the scene semantic coherence between adjacent repair frames in the complete visual repair frame sequence, and use the temporal consistency constraint flow to perform inter-frame flicker elimination on the complete visual repair frame sequence to generate the final visual perception output frame sequence.

[0055] Step S151: Divide each full visual restoration frame in the full visual restoration frame sequence into scene semantic regions, and divide each full visual restoration frame into multiple semantically coherent region units. Each semantically coherent region unit corresponds to a visual element block with independent scene semantics in that full visual restoration frame.

[0056] For each frame F1 in the fully visually restored frame sequence, a pre-trained semantic segmentation model is invoked to divide the scene into semantic regions. This semantic segmentation model adopts the DeepLabV3+ architecture, with a ResNet-101 backbone network as the encoder and a decoder that fuses multi-scale semantic features through dilated spatial pyramid pooling, outputting a pixel-level semantic category label map with the same resolution as the input image. Based on the semantic category label map, each frame is divided into multiple semantically coherent region units (SCUs). Each SCU consists of a set of spatially connected pixels belonging to the same semantic category, denoted as E1 to EK. In the wind power station scene, semantic categories include wind turbine tower, blades, nacelle, transmission lines, sky, and ground.

[0057] Step S152: Perform cross-frame semantic matching on the semantically coherent region units of two adjacent fully visually repaired frames, establish the corresponding association relationship between semantically coherent region units between adjacent frames, and form a set of semantically coherent region unit matching pairs.

[0058] For two adjacent frames F1t and F1t+1, cross-frame semantic matching is performed on their respective semantically coherent region units E1t to EKt and E1t+1 to EKt+1. The matching criterion is that the semantic category labels of the two semantically coherent region units are the same, and their spatial intersection-union ratio is greater than the threshold T5. Corresponding associations are established for semantically coherent region units that meet the matching criteria, forming a set of semantically coherent region unit matching pairs M7.

[0059] Step S153: Calculate the brightness fluctuation amplitude and color fluctuation amplitude of each pair of matched semantic coherent region units in the set of matching semantic coherent region units between adjacent frames, and generate the temporal flicker intensity distribution of the semantic coherent region unit based on the brightness fluctuation amplitude and color fluctuation amplitude.

[0060] For each pair of matched semantically coherent region units Et and Et+1 in M7, calculate the average pixel value differences ΔRt, ΔGt, and ΔBt across the three RGB color channels. (Brightness fluctuation amplitude) Color fluctuation range L1 and C1 are used as the temporal flicker intensity values ​​of the semantically coherent region unit and assigned to all pixel positions contained in the semantically coherent region unit to form the temporal flicker intensity distribution of the pair of matching units.

[0061] Step S154: Spatially aggregate the temporal flicker intensity distribution of all matching pairs in the semantically coherent region unit matching pair set to construct a temporal consistency constraint flow covering the entire frame. The temporal consistency constraint flow records the brightness and color adjustment vector of each pixel position between adjacent frames based on the pixel position index.

[0062] The L1 and C1 values ​​of all matching pairs in M7 are aggregated in the entire frame image space. For pixel locations where different semantically coherent region units overlap spatially, the average flicker intensity value of each overlapping unit is taken as the flicker intensity value of that pixel location. For each pixel location, a brightness and color adjustment vector V5 = [-L1, -C1] is constructed, which indicates the amount of inverse compensation to be applied to eliminate inter-frame flicker. The V5 values ​​of all pixel locations constitute the temporal consistency constraint flow M8.

[0063] Step S155: Use the temporal consistency constraint flow to adjust the brightness and color of each pixel position in each fully visually restored frame in the fully visually restored frame sequence to obtain a temporally consistent visually restored frame sequence.

[0064] For each pixel position in F1t+1, adjust the brightness and color according to V5 in M8 for that pixel position. The adjusted pixel value = original pixel value + V5. After all frames are adjusted, a temporally consistent visual restoration frame sequence is obtained, denoted as F2.

[0065] Step S156: Arrange all frames in the temporally consistent visual repair frame sequence according to the frame number to generate the final visual perception output frame sequence.

[0066] Arranging all frames in F2 in frame number order yields the final visual perception output frame sequence, denoted as Z1. Z1, while restoring the degradation traces of the sandstorm, eliminates brightness and color flicker between adjacent frames after restoration, and can be directly used for subsequent visual perception tasks such as defect detection in wind power plant equipment.

[0067] Step S210: Perform optical path degradation source inversion and tracing processing on the optical path degradation trace distribution map, map each frame of the optical path degradation trace distribution map to the visual sensing optical path model of the inspection agent, and analyze the optical path degradation cause type of the optical path degradation trace on a cross-section of the optical path within the visual sensing optical path model. The optical path degradation cause type includes optical path medium scattering degradation type and optical path medium refraction offset degradation type.

[0068] Obtain the optical path degradation trace distribution map P1 generated in step S125 and the pre-calibrated inspection agent visual sensing optical path model parameters. This optical path model describes the complete optical path geometric mapping relationship from 3D scene points to the camera sensor plane. For each pixel position marked as a degradation trace in P1, trace the incident ray path in 3D space in reverse through the optical path model, and divide the incident ray into K1 optical path sections at equal intervals. Calculate the local orientation consistency feature of the degradation trace on each section: take a micro-element centered at the intersection of the section and the ray, and calculate the structure tensor of the degradation trace gradient direction within the micro-element. Perform eigenvalue decomposition on this structure tensor to obtain two eigenvalues ​​λ1 and λ2. When λ1 is much larger than λ2, it indicates that the degradation trace on this section has strong orientation consistency, and is determined to be of the optical path medium refraction offset degradation type, marked with label 2. When λ1 and λ2 are close and small, it indicates that the degradation trace orientation is dispersed, and is determined to be of the optical path medium scattering degradation type, marked with label 1. The analysis results of all cross sections are traversed, and the type with the largest proportion is taken as the optical path degradation cause type at that pixel location. A cause type label map E1 with the same size as P1 is generated, and the value of each pixel is 1 or 2.

[0069] Step S220: According to the optical path degradation cause type, the optical path degradation trace distribution map is labeled with degradation trace attributes. Each optical path degradation pixel unit in the optical path degradation trace distribution map is marked as an optical path medium scattering degradation pixel unit or an optical path medium refraction offset degradation pixel unit, and a degradation cause attribute labeled optical path degradation trace distribution map is generated.

[0070] For each pixel position in P1, read the corresponding cause type label in E1. If the label value is 1, write the optical path medium scattering degradation pixel unit marker into the metadata channel of the corresponding pixel position in the P1 copy. If the label value is 2, write the optical path medium refraction offset degradation pixel unit marker. For non-degradation trace pixels in P1, write 0 into the metadata channel to indicate no degradation. Finally, generate a degradation cause attribute-annotated optical path degradation trace distribution map P3, with the same image size as P1. Each pixel contains the original brightness value and an integral cause type attribute label, forming a two-channel data structure.

[0071] Step S230: Perform pixel unit clustering on the optical path degradation trace distribution map labeled with degradation cause attributes, and aggregate the optical path degradation pixel units that are spatially continuous and have the same degradation cause type into optical path degradation homogeneous blocks to obtain a set of optical path degradation homogeneous blocks. Each optical path degradation homogeneous block in the set of optical path degradation homogeneous blocks has a unified degradation cause type and continuous spatial boundary.

[0072] Create an access marker array of the same size as P3, initializing all elements to 0. Scan P3 row by row; when encountering a pixel with an access marker of 0 and a non-zero causation type attribute, use that pixel as the seed point to initiate region growing. The similarity criterion for region growing is that the causation type attribute of neighboring pixels is the same as that of the seed point. Using an 8-neighborhood connection method, aggregate all spatially connected pixels that satisfy the similarity criterion into a single optical path degeneration homogeneous block. Assign a unique identifier to this block, record its causation type label, and extract the coordinates of the top-left and bottom-right corners of the smallest bounding rectangle as the spatial boundary. The output is a list-structured set of optical path degeneration homogeneous blocks S3, where each element contains a block ID, a causation type label, a list of pixel coordinates, and spatial boundary coordinates.

[0073] Step S240: Quantify the degradation degree of each optical path degradation homogeneous block in the optical path degradation homogeneous block set, extract the optical path degradation intensity value of all optical path degradation pixel units in the optical path degradation homogeneous block, calculate the average optical path degradation intensity of the optical path degradation homogeneous block, and assign the average optical path degradation intensity as the degradation degree value of the optical path degradation homogeneous block.

[0074] For each homogeneous block of optical path degradation in S3, iterate through its pixel coordinate list, read the optical path degradation intensity value at the corresponding coordinate position from P1, and store the optical path degradation intensity values ​​of all pixels into a one-dimensional array. Calculate the arithmetic mean E1 of this array; E1 is the degradation level value of the block, and it is appended to the attributes of that block in S3. The larger the E1 value, the more severe the optical path degradation of the block, and the greater the processing intensity required for compensation.

[0075] Step S250: Based on the degradation cause type and degradation degree value of the optical path degradation homogeneous block, perform degradation cause differentiation compensation strategy for optical path degradation homogeneous blocks with different degradation cause types, perform optical path scattering inverse filtering compensation for the optical path degradation homogeneous block where the optical path medium scattering degradation pixel unit is located, and perform optical path refraction offset reverse mapping compensation for the optical path degradation homogeneous block where the optical path medium refraction offset degradation pixel unit is located, thereby generating a refined optical path degradation compensation frame sequence.

[0076] Each block in S3 is processed according to its causal type label. For a scattering degradation block with a causal type label of 1, the physical model for scattering degradation is a point spread function convolution, and inverse filtering compensation is achieved through Wiener filtering. The Gaussian kernel radius σ1 of the scattering degradation point spread function is estimated based on the degradation degree E1 of the block, where σ1 is proportional to E1. A Gaussian kernel is constructed, and the degradation transfer function H is calculated in the frequency domain using Fourier transform. Then, the Wiener filtering formula is applied. Where K2 is the ratio constant of the noise power spectrum to the signal power spectrum, and conj(H) is the conjugate of H. Pixels within the block are frequency-domain filtered and then inversely transformed back to the spatial domain to obtain the scattering compensation result. For refraction-offset degradation blocks with a cause type label of 2, the refraction-offset model is a pixel spatial position offset, with the offset Δp proportional to E1. For each pixel within the block, its reverse offset target position (x-Δpx, y-Δpy) is calculated, and bilinear interpolation is used to sample pixel values ​​from this position and fill them into the current position. After all blocks are processed, a refined optical path degradation compensation frame sequence Q2 is generated.

[0077] Step S260: The refined optical path degradation compensation frame sequence is superimposed frame by frame with the frame sequence corresponding to the optical path disturbance residual component map to update the preliminary visual repair frame sequence.

[0078] For frames Q2 and P2 with the same frame number, pixel-by-pixel brightness values ​​are superimposed, with the superposition method being R1new(x,y)=Q2(x,y)+P2(x,y). After processing all frames, an updated preliminary visual restoration frame sequence R1new is generated, which replaces the original R1 to complete the update.

[0079] Step S310: Perform spectral analysis of the optical path disturbance residual component map, transform each frame of the optical path disturbance residual component map from the spatial domain to the frequency domain to obtain the optical path disturbance residual spectrum map, and identify the spectral abnormal peak clusters corresponding to the dynamic medium disturbance of extreme weather in the optical path disturbance residual spectrum map.

[0080] A two-dimensional fast Fourier transform is performed on the residual component image P2 of the optical path disturbance to obtain the complex spectrum F. The amplitude spectrum M = |F| is calculated. After shifting the origin of the spectral coordinates to the center, the spectrum is divided into three annular regions of low frequency, mid frequency, and high frequency with frequency radii r1 and r2. Local maxima are searched in the mid frequency region using a 3×3 neighborhood centered on each frequency point. If the amplitude of a point is greater than all other points in the neighborhood and exceeds C1 times the mean amplitude of the entire spectrum, it is marked as a local maximum. Spatially adjacent local maxima are clustered into a cluster of spectral anomaly peaks, and the center frequency coordinates and cluster radius of each cluster are recorded. These clusters of spectral anomaly peaks correspond to quasi-periodic noise caused by the dynamic medium disturbance of sand and dust.

[0081] Step S320: Based on the frequency distribution range and amplitude distribution characteristics of the spectrum abnormal peak cluster, construct an extreme weather disturbance frequency domain filter. The extreme weather disturbance frequency domain filter is used to suppress the frequency components corresponding to the spectrum abnormal peak cluster from the optical path disturbance residual spectrum.

[0082] Create a matrix of all ones with the same size as the complex spectrum F as the filter H2. Iterate through each cluster of spectral anomalies, using the center frequency coordinates of the cluster as the center and the radius of the cluster multiplied by the spread factor e1 as the suppression radius. Set all frequency points within this suppression radius to 0 in H2. After processing all clusters, H2 becomes the frequency domain filter for extreme weather disturbances, which is a binary mask matrix.

[0083] Step S330: Use the extreme weather disturbance frequency domain filter to perform frequency domain filtering on the optical path disturbance residual spectrum, filter out the frequency components corresponding to the abnormal peak clusters of the spectrum, and obtain the filtered optical path disturbance residual spectrum.

[0084] The filtering operation is a frequency-by-frequency element-wise multiplication, F4 = F × H2. F4 is the residual spectrum of optical path disturbance after filtering, and the frequency components corresponding to the abnormal peak clusters in the spectrum are suppressed.

[0085] Step S340: The filtered optical path disturbance residual spectrum is inversely transformed from the frequency domain back to the spatial domain to obtain the de-disturbed optical path residual component map. Compared with the original optical path disturbance residual component map, the de-disturbed optical path residual component map removes the random noise disturbance component caused by dynamic medium disturbance in extreme weather.

[0086] A two-dimensional inverse fast Fourier transform is performed on F4, and the real part of the result is taken as the residual component image P4 of the de-perturbed optical path. Compared with P2, P4 effectively suppresses the quasi-periodic noise caused by the rapid movement of sand particles, while preserving the real texture of the scene.

[0087] Step S350: Perform residual detail enhancement processing on the residual component map of the de-disturbed optical path, extract the weakened visual detail weak signal in the residual component map of the de-disturbed optical path, and amplify the signal gain of the weak visual detail signal to obtain the detail-enhanced residual component map of the optical path.

[0088] High-frequency detail signals were extracted by convolving P4 with the Laplacian operator, where the Laplacian kernel K was a standard 3×3 kernel. The extracted high-frequency details... ,in This represents the convolution operation. After multiplying Lap by the gain factor g1 and then superimposing it back into P4, P5 = P4 + g1 × Lap. P5 is the detail-enhanced optical path residual component map, whose texture details are clearer than P4.

[0089] Step S360: Superimpose the optical path degradation compensation frame sequence with the frame sequence corresponding to the detail enhancement optical path residual component map frame by frame to update the preliminary visual repair frame sequence.

[0090] The optical path degradation compensation frame sequence Q1 and P5 are superimposed frame by frame, with the superposition method being the addition of pixel brightness values, to generate an updated preliminary visual restoration frame sequence.

[0091] Step S410: Perform inter-frame optical flow continuity analysis on the fully visually restored frame sequence, extract the dense optical flow vector field between two adjacent fully visually restored frames, and mark the regions in the dense optical flow vector field whose optical flow vector direction deviates from the direction deviation threshold of the neighboring optical flow vector direction as suspected restoration artifact regions.

[0092] For two adjacent frames F1t and F1t+1 in the fully visually restored frame sequence F1, the Farneback dense optical flow algorithm is used to calculate the dense optical flow vector field. This algorithm approximates the local neighborhood of each pixel through polynomial expansion, solves for the optical flow displacement using polynomial coefficients, and outputs a two-dimensional optical flow vector for each pixel. For each pixel position in the optical flow field, the direction angles of all optical flow vectors in its 8-neighborhood are taken, and the circular mean of these direction angles is calculated. Then, the absolute angle difference between the pixel's optical flow direction angle and the circular mean is calculated. Pixels with angle differences exceeding a preset threshold T7 are marked as abnormal optical flow points. Morphological closing operations are performed on the abnormal optical flow points, and abnormal optical flow points with a spatial distance of less than d1 are clustered into a connected region. Each connected region is a suspected restoration artifact region, denoted as set R2.

[0093] Step S420: Perform temporal backtracking verification on the suspected repair artifact region, select verification pixels within the suspected repair artifact region, and trace the pixel value evolution trajectory of the verification pixels in the defective visual frame sequence and the preliminary visual repair frame sequence back along the time axis.

[0094] For each suspicious region in R2, one verification pixel is selected at its center and one at its boundary. For each verification pixel, its pixel value in each frame within a continuous time window is recorded. The time window covers the frame backward from the current frame (L1 frames) and the frame forward (L2 frames). At each time point, the original defective pixel value Iorig is read from the same pixel coordinates of the corresponding frame in the defective visual frame sequence, and the preliminary repaired pixel value Irep is read from the same pixel coordinates of the corresponding frame in the preliminary visual repair frame sequence R1. Iorig and Irep are arranged in chronological order to form the pixel value evolution trajectory X1 of the verification pixel. X1 is a two-dimensional array with the number of rows equal to the total number of frames in the time window. The first column stores Iorig, and the second column stores Irep.

[0095] Step S430: Determine whether the suspected repair artifact region is a false texture region based on the pixel value evolution trajectory. If the pixel value evolution trajectory shows a non-continuous jump pattern in the time domain, then the suspected repair artifact region is determined to be a false texture region.

[0096] For the Irep column of X1, calculate the variance of the inter-frame difference V1 = Var(Irep(i+1)-Irep(i)), and simultaneously calculate the variance of the inter-frame difference V0 = Var(Iorig(i+1)-Iorig(i)) for the Iorig column. V1 in a normal texture restoration region should be close to V0, while V1 in a false texture region will be significantly greater than V0. If V1 / V0 > T8, then the suspected restoration artifact region containing the verified pixel is determined to be a false texture region, and the set of false texture regions R3 is output.

[0097] Step S440: Perform texture realism regeneration processing on the false texture region, extract the spatial neighborhood real texture samples of the false texture region in the complete visual restoration frame sequence, and regenerate the texture content in the false texture region according to the texture primitive arrangement rules of the spatial neighborhood real texture samples to obtain the artifact elimination visual restoration frame sequence.

[0098] For each false texture region in R3, its smallest bounding rectangle is taken and expanded outward by r3 pixels to form a context region containing the neighboring real texture. A block-matching-based texture synthesis algorithm is used, starting with the boundary pixels of the false texture region and filling layer by layer from the outside in. The filling unit is an s1×s1 pixel block. For each block to be filled, the most similar texture block in its already filled neighborhood is searched in the context region. The similarity is measured by the sum of squared pixel brightness differences. The content of the most similar texture block is copied to the location of the block to be filled. After all false texture regions have been processed, the artifact-removed visual restoration frame sequence F5 is generated.

[0099] Step S450: Perform inter-frame texture consistency verification on the artifact elimination visual restoration frame sequence, extract the texture primitive arrangement rules at the same spatial position between adjacent frames, mark the spatial positions with inconsistent texture primitive arrangement rules as residual inconsistent texture regions, and perform local texture remapping on the residual inconsistent texture regions to generate a consistency-enhanced visual restoration frame sequence.

[0100] For the same spatial coordinates of two adjacent frames F5t and F5t+1, a texture block pair of size s2×s2 is taken, and the gray-level co-occurrence matrix of the two texture blocks is calculated. Four feature values—contrast, energy, homogeneity, and correlation—are extracted to form a feature vector. The Euclidean distance D1 between the two feature vectors is calculated as a measure of the difference in texture primitive arrangement rules. The center pixel of the texture block whose D1 exceeds the threshold T9 is marked as a pixel of the residual inconsistent texture region. For the residual inconsistent texture region, using F5t as the reference frame, an optical flow-guided texture remapping is used to map the texture of F5t to the corresponding position of F5t+1. After processing, a consistency-enhanced visual restoration frame sequence F6 is generated.

[0101] For example, step S510: Perform visual perception task adaptability analysis on the final visual perception output frame sequence to obtain the target visual perception task type to be performed by the inspection agent under extreme weather conditions at the new energy power station. The target visual perception task type includes new energy equipment appearance defect detection task and new energy power station perimeter foreign object intrusion detection task.

[0102] Read the integer value of the task type field from the task planning file in the UAV flight control system. If it is 1, the target visual perception task type is new energy equipment appearance defect detection task; if it is 2, it is new energy station perimeter foreign object intrusion detection task. Output the task type identifier TASK.

[0103] Step S520: Determine the visual perception sensitive feature dimensions according to the target visual perception task type. If the target visual perception task type is a new energy equipment appearance defect detection task, then the visual perception sensitive feature dimensions are determined to be the equipment edge contour sharpness feature and the equipment surface texture contrast feature. If the target visual perception task type is a new energy station perimeter foreign object intrusion detection task, then the visual perception sensitive feature dimensions are determined to be the moving target salience feature and the moving target contour integrity feature.

[0104] If TASK is 1, the feature dimension list is set to {device edge contour sharpness, device surface texture contrast}. If TASK is 2, the feature dimension list is set to {moving target salience, moving target contour integrity}.

[0105] Step S530: For each final visual perception output frame in the final visual perception output frame sequence, perform feature intensity evaluation on the visual perception sensitive feature dimension, calculate the feature response intensity value of the final visual perception output frame on the visual perception sensitive feature dimension, and obtain the feature response intensity frame sequence.

[0106] Each frame in the final visual perception output frame sequence Z1 is evaluated item by item according to the feature dimension list. If the feature is the sharpness of the device edge contour, the edge pixels of the device region are first extracted using Canny edge detection, and then the average gradient magnitude of all edge pixels is calculated as the feature response intensity value C2. If the feature is the saliency of a moving target, the absolute difference map between the current frame and the previous frame is first calculated, the moving target region is automatically segmented using the Otsu method, and the average gray value of the pixels in that region is taken as C3. The C2 or C3 of all frames are arranged according to the frame number to form the feature response intensity frame sequence X2.

[0107] Step S540: Perform temporal characteristic response fluctuation analysis on the characteristic response intensity frame sequence, identify frame segments in the characteristic response intensity frame sequence that exhibit abnormal attenuation of characteristic response intensity, and mark the frame segments with abnormal attenuation of characteristic response intensity as weak perception frame segments.

[0108] Calculate the inter-frame difference sequence ΔC(i) = X2(i+1) - X2(i) for X2. Identify consecutive negative subsequences from the ΔC sequence, corresponding to time periods where the feature response intensity continuously decreases. For each consecutive negative subsequence, calculate the ratio r of the cumulative decrease between the start and end frames to the initial decrease. If r > T10, mark the frame segment corresponding to that subsequence as a perceptually weak frame segment R5.

[0109] Step S550: Perform visual perception sensitive feature enhancement processing on the perception-weak frame segment. Perform feature response compensation on each frame in the perception-weak frame segment in the visual perception sensitive feature dimension to increase the feature response intensity value of the frame in the visual perception sensitive feature dimension, and obtain a perception-enhanced visual perception output frame sequence.

[0110] For each frame in R5, the difference ΔF between its current feature response intensity value and the normal level reference value is calculated. If it is a device edge contour sharpness feature, an unsharpened mask enhancement algorithm is used for compensation, with the enhancement amount proportional to ΔF. If it is a moving target saliency feature, adaptive histogram equalization is used to stretch the contrast of the moving region, with the stretching intensity proportional to ΔF. After processing, the perception-enhanced visual perception output frame sequence Z2 is obtained.

[0111] Step S560: The perception-enhanced visual perception output frame sequence is used as a new final visual perception output frame sequence and provided to the target visual perception task module of the inspection agent.

[0112] The update is completed by assigning Z2 to Z1. The updated Z1 is then transmitted to the airborne target visual perception task module via the UAV data link for use by subsequent defect detection or foreign object intrusion detection algorithms.

[0113] Step S610: Input the original visual data stream into the pre-constructed inter-frame optical path degradation evolution prediction model. The inter-frame optical path degradation evolution prediction model is used to predict the optical path degradation trend map of the next frame of the missing visual frame based on the optical path degradation traces and the pose changes of the agent in consecutive missing visual frames.

[0114] The pre-built inter-frame optical path degradation evolution prediction model employs a cascaded architecture of a 3D convolutional neural network and a long short-term memory network. The 3D convolutional neural network consists of three 3D convolutional layers, each with a kernel size of 3×3×3 and a stride of 1, producing 32, 64, and 128 output channels respectively. Each convolutional layer is followed by a ReLU activation function and a 3D max-pooling layer to extract spatiotemporal features from K1 consecutive defective visual frames, outputting a four-dimensional tensor of 128×4×H×W. This tensor is unfolded along the time dimension into a sequence of length 4, with each time step's feature vector having a dimension of 128×H×W. This feature vector is then input into a two-layer long short-term memory network, with each layer containing a hidden state of 256 dimensions. The hidden state of the last time step in the long short-term memory network is mapped through a fully connected layer to the predicted optical path degradation trend map for the next frame, with the same dimensions as the input frame. The model was trained using a series of continuous video sequences collected during a sandstorm. The loss function was the mean square error between the predicted optical path degradation trend map and the actual optical path degradation trace distribution map of the next frame.

[0115] Step S620: Extract intermediate layer representation features from the inter-frame optical path degradation evolution prediction model. The intermediate layer representation features are used to characterize the implicit evolution state of the extreme weather degradation process. The intermediate layer representation features are used as the external guiding signal for the optical path degradation separation process.

[0116] The raw visual data stream is input into the trained inter-frame optical path degradation evolution prediction model. During the model's forward propagation, the hidden state vector H3, with a dimension of 256, is extracted from the last time step of the Long Short-Term Memory network. H3 encodes the implicit state of the dynamic evolution of dust degradation between consecutive frames, and H3 is output as an external guiding signal.

[0117] Step S630: The external guiding signal is conditionally fused with the pixel brightness level decomposition process of each frame of the defective visual frame. When generating the brightness level component sequence, the external guiding signal is used as a conditional constraint to adjust the separation threshold between the coarse brightness base layer and the progressive detail brightness residual layer, so that the coarse brightness base layer contains more complete extreme weather optical path degradation traces.

[0118] H3 is input into a conditional parameter generation network consisting of two fully connected layers: the first layer has 128 neurons, and the second layer has 1 neuron. The output is an adaptive separation threshold T11. T11 replaces the original fixed threshold and controls the allocation of residual components to the coarse brightness base layer or the progressive detail brightness residual layer during Laplacian pyramid decomposition. T11 dynamically adapts to the intensity of dust degradation, allowing the coarse brightness base layer to retain more complete traces of optical path degradation.

[0119] Step S640: Guide and correct the optical path degradation direction field estimation process by injecting the prior information of optical path degradation direction implied in the external guidance signal into the calculation of the main optical path degradation direction angle of each pixel position in the optical path degradation candidate region, thereby correcting the optical path degradation direction estimation deviation caused by dynamic changes in extreme weather.

[0120] Input H3 into another conditional parameter generation network with the same structure, and output the orientation correction angle ΔD. Correct the optical path degradation principal orientation angle D1 calculated in step S123; the corrected D1' = D1 + ΔD. The corrected orientation angle reconstructs the optical path degradation pattern M1. ΔD is derived from the dust movement trend learned from consecutive frames by the inter-frame optical path degradation evolution prediction model, and can correct the deviation in the single-frame orientation field estimation.

[0121] Step S650: The extraction process of fine-grained traces of optical path degradation is guided and constrained. The significant distribution information of optical path degradation traces hidden in the external guidance signal is fused with the layer-by-layer directional consistency comparison process. The directional angle threshold is adjusted so that the directional angle threshold adaptively matches the extreme weather degradation intensity of the current frame.

[0122] Input H3 into the third conditional parameter generation network to output an adaptive orientation angle threshold T12. Replace the fixed threshold T1 in step S124 with T12 to determine whether the residual values ​​of each pixel in the progressive detail brightness residual layer are retained as fine-grained traces of optical path degradation. The more severe the degradation, the larger the value of T12, in order to retain more fine-grained degradation traces with similar orientations.

[0123] Step S660: Apply global consistency constraints to the generation process of the optical path degradation trace distribution map, using the continuity information of optical path degradation evolution between adjacent frames provided by the external guidance signal as a constraint condition to correct the inter-frame inconsistency of optical path degradation traces in the optical path degradation trace distribution map, thereby obtaining an optical path degradation trace distribution map with enhanced temporal consistency.

[0124] Input H3 into the fourth conditional parameter to generate the network, which outputs the inter-frame consistency constraint coefficient λ1, with a value ranging from 0 to 1. For the optical path degradation trace distribution map P1t of the current frame, take P1t-1 and P1t+1 of the previous and next frames, and perform weighted fusion. The closer λ1 is to 0, the greater the impact of dynamic disturbances on the current frame, and the more it should rely on information from neighboring frames for smoothing correction. The corrected P1new is smoother and more consistent in the time dimension.

[0125] Step S710: Perform compensation boundary fusion processing on the optical path degradation compensation frame sequence, and extract the boundary transition zone between the compensation area that has completed optical path degradation compensation and the normal area that is not affected by optical path degradation in each optical path degradation compensation frame. The boundary transition zone is a strip-shaped set of pixels where the compensation area and the normal area are spatially adjacent and the pixel brightness difference exceeds the fusion threshold.

[0126] In the optical path degradation compensation frame sequence Q1, the compensation region mask for each frame is formed by the pixel positions of the optical path degradation candidate region R1 marked in step S125. Morphological dilation and erosion operations are performed on this mask, with both the dilation radius and erosion radius being r4. The outer boundary is the difference region between the dilation result and the erosion result, and the strip between the outer boundary and the eroded region is the boundary transition zone R6. Within R6, the brightness difference between the compensation region side and the normal region side is calculated pixel by pixel, and pixels with a brightness difference exceeding the threshold T13 are retained as the final boundary transition zone pixels.

[0127] Step S720: Measure the width of the boundary transition zone, calculate the pixel distance from the edge of the compensation area to the edge of the normal area pixel by pixel along the direction perpendicular to the boundary transition zone, and obtain the transition zone width distribution sequence. Determine the transition zone type based on the change pattern of the transition zone width in the transition zone width distribution sequence. The transition zone type includes uniform width transition zone and non-uniform width transition zone.

[0128] For each pixel position on the R6 boundary line, calculate the pixel step d2 from the edge of the compensation region to the edge of the normal region along the boundary normal direction. Statistically calculate the d2 values ​​of all boundary pixels to form the transition band width distribution sequence X3. Calculate the mean and standard deviation of X3. If the ratio of the standard deviation to the mean is less than T14, it is determined to be a uniform width transition band; otherwise, it is a non-uniform width transition band.

[0129] Step S730: Perform gradient brightness blending processing on the uniform width transition band. Establish a brightness gradient curve in the uniform width transition band along a direction perpendicular to the direction of the transition band. The starting brightness value of the brightness gradient curve is the brightness value of the edge pixel of the compensation area, and the ending brightness value of the brightness gradient curve is the brightness value of the edge pixel of the normal area. Reassign the brightness value of each pixel in the uniform width transition band according to the brightness gradient curve to eliminate the abrupt boundary between the compensation area and the normal area.

[0130] For a uniform width transition zone, the average width of the transition zone is taken as the effective width davg. A brightness gradient function f(s) is established along the boundary normal direction, where s is the distance variable in the normal direction, ranging from 0 to davg. f(0) is equal to the brightness value A2 at the edge of the compensation area, and f(davg) is equal to the brightness value B2 at the edge of the normal area. Each pixel on the normal line within the transition zone is reassigned a brightness value by f(s) according to its s value, eliminating abrupt brightness changes.

[0131] Step S740: Perform adaptive width equalization processing on the non-uniform width transition band, extract the local maximum and local minimum positions of the transition band width within the non-uniform width transition band, expand the extension range of the normal region texture towards the compensation region at the local maximum position, and shrink the extension range of the compensation region texture towards the normal region at the local minimum position, so that the width of the non-uniform width transition band tends to be uniform, thus obtaining a width equalization transition band, and perform the same gradient brightness fusion processing on the width equalization transition band as on the uniform width transition band.

[0132] A sliding window is used to detect the locations of local maxima and minima in the transition band width of X3. At the local maxima, the normal region texture is expanded towards the compensation region along the normal direction, with the expansion step being the rounded difference between the local maxima and the mean of X3. At the local minima, the compensation region texture is contracted towards the normal region along the normal direction, with the contraction step being the rounded difference between the mean of X3 and the local minima. After processing, the variance of X3 decreases, and the transition band width tends to be uniform. Then, the same gradient brightness blending process as in step S730 is performed.

[0133] Step S750: Perform edge texture rematching on the compensation area that has completed the gradient brightness fusion processing, extract the texture primitive features of the edge pixels of the compensation area and the texture primitive features of the edge pixels of the normal area, calculate the texture primitive feature difference between the edge texture primitive features of the compensation area and the edge texture primitive features of the normal area, and perform micro-deformation adjustment on the texture primitives of the compensation area edge according to the texture primitive feature difference, so that the texture primitives of the compensation area edge and the texture primitives of the normal area edge form a natural continuation of texture primitives at the boundary.

[0134] Take strips of width w2 at the edges of both the compensated and normal regions. Calculate the local binary pattern feature value for each pixel in each strip, and then plot the local binary pattern feature histograms of the two strips, denoted as Hc and Hn. Calculate the chi-square distance between Hc and Hn as the texture primitive feature difference. If the difference exceeds T15, for each pixel in the compensated region edge strip, search for the pixel with the closest local binary pattern feature value in the normal region edge strip. Mix this pixel value with the current pixel value using a weighted average α1, ensuring a natural transition between the compensated region edge texture and the normal region edge texture at their boundaries.

[0135] Step S760: Merge the compensation area after edge texture rematching with the normal area into a seamless fusion visual frame, combine all the seamless fusion visual frames with all frame numbers into a seamless fusion optical path degradation compensation frame sequence, and superimpose the seamless fusion optical path degradation compensation frame sequence with the frame sequence corresponding to the optical path disturbance residual component map frame by frame to update the preliminary visual repair frame sequence.

[0136] The compensated area processed in step S750 is merged with the normal area into a seamless fused visual frame. All seamless fused visual frames constitute the seamless fused optical path degradation compensation frame sequence Q3. Q3 is then superimposed on the optical path perturbation residual component map P2 frame by frame, with the superposition method being the addition of pixel-by-pixel brightness values, to generate an updated preliminary visual repair frame sequence.

[0137] Step S810: Perform repair trace invisibility processing on the fully visually repaired frame sequence, and extract the spatial position mask of the repair filling area after structural texture co-repair in each fully visually repaired frame. The repair filling area is the area in the initial visually repaired frame sequence that originally had geometric boundary breaks or texture missing and was filled and repaired by the intra-frame spatial context constraint field.

[0138] The positions of the filled pixels in each frame are obtained during the repair process in step S146. The filled pixel positions are used to construct a binary mask M10, with the filled pixels marked as 1 and the rest as 0.

[0139] Step S820: Perform a naturalness analysis of the filling texture in the repaired filling area, extract the texture primitive arrangement rules of the filling texture in the repaired filling area, and simultaneously extract the texture primitive arrangement rules of the original texture in the spatial neighborhood of the repaired filling area. Compare the rule differences between the texture primitive arrangement rules of the filling texture and the texture primitive arrangement rules of the original texture to obtain a texture primitive arrangement rule difference distribution map.

[0140] For each filled region with a value of 1 in M10, calculate its gray-level co-occurrence matrix (GLCM), and extract four features: contrast, energy, homogeneity, and correlation to form a feature vector Vfill. Take the spatial neighborhood of the filled region extending by p2 pixels, and similarly calculate the GLCM feature vector Vorig. Calculate the Euclidean distance between the two feature vectors as the texture primitive arrangement rule difference, and assign this difference to all pixels within the filled region, forming the texture primitive arrangement rule difference distribution map M11.

[0141] Step S830: Locate the misaligned positions of texture primitives where the difference in texture primitive arrangement rules exceeds the rule difference tolerance according to the texture primitive arrangement rule difference distribution map. Perform forced alignment processing of texture primitive arrangement rules at the misaligned positions. Using the texture primitive arrangement rules of the original texture as the alignment reference, rearrange the spatial positions of the filling texture primitives at the misaligned positions so that the arrangement rules of the filling texture primitives are consistent with the arrangement rules of the original texture primitives.

[0142] Pixel positions in M11 where the difference exceeds the threshold T16 are marked as texture primitive misalignment positions. Using the eigenvector Vorig of the gray-level co-occurrence matrix of the original texture in the spatial neighborhood as the target, an iterative optimization method is employed to adjust the spatial arrangement of the filling texture pixels. In each iteration, the Euclidean distance between the eigenvector of the current texture block's gray-level co-occurrence matrix and Vorig is calculated, and the pixel position is updated along the gradient descent direction until the distance no longer decreases.

[0143] Step S840: Perform chromaticity coordination processing on the repaired filling area after the forced alignment of texture primitive arrangement rules. Extract the chromaticity statistical distribution features of the filling texture in the repaired filling area and the chromaticity statistical distribution features of the original texture in the spatial neighborhood. Calculate the chromaticity offset between the mean chromaticity of the filling texture and the mean chromaticity of the original texture. Based on the chromaticity offset, perform an overall translation of the chromaticity components of each pixel in the repaired filling area so that the chromaticity statistical distribution features of the filling texture are close to the chromaticity statistical distribution features of the original texture.

[0144] The aligned filled region and its spatial neighborhood are converted to the YCbCr color space. The mean values ​​of the Cb and Cr channels in the filled region, as well as the mean values ​​of the Cb and Cr channels in the neighborhood, are calculated, and the chromaticity offsets ΔCb and ΔCr for each channel are calculated respectively. ΔCb and ΔCr are added to the Cb and Cr values ​​of each pixel in the filled region to complete the overall translation.

[0145] Step S850: Perform fill boundary trace removal processing on the repaired fill area after the fill texture color coordination processing is completed. Extract the repair boundary line between the repaired fill area and the spatial neighborhood. Take a pixel band of the width of the boundary removal band on both sides of the repair boundary line along the normal direction of the repair boundary line as the boundary removal area. Perform spatial domain Gaussian weighted mixing in the boundary removal area. The fill texture pixel value in the repaired fill area and the original texture pixel value in the spatial neighborhood are weighted and mixed according to the Gaussian weight coefficient to eliminate the artificial traces at the repair boundary line.

[0146] The boundary between 1 and 0 values ​​in M10 is extracted as the repair boundary line. A boundary hidden area with a width of w3 pixels is formed on both sides along the boundary normal direction. The final pixel value in the boundary hidden area is obtained by Gaussian weighted mixing. The weight decreases as the normal distance from the pixel to the boundary increases. At the boundary, the repair side weight is 1 and the original side weight is 0. Further away from the boundary, the repair side weight decreases and the original side weight increases.

[0147] Step S860: Merge the repaired filling area that has undergone forced alignment of texture primitive arrangement rules, color coordination of filling texture, and hidden mark removal of filling boundary with the unrepaired area in the complete visual repair frame sequence into an invisible repair visual frame. Combine the invisible repair visual frames with all frame numbers into an invisible repair complete visual repair frame sequence. Update the complete visual repair frame sequence with the invisible repair complete visual repair frame sequence.

[0148] Replace the pixel values ​​of the filled region after completing all three steps of processing with the corresponding positions in F1 to obtain the invisible repair visual frame F7. The repair traces in F7 are difficult to detect visually. Combine the F7 values ​​of all frames into a sequence of completely visually repaired frames with invisible repair, and assign the values ​​to F1 to complete the update.

[0149] Step S910: Perform scene semantic stability preservation processing on the final visual perception output frame sequence, and extract the scene semantic label map of each final visual perception output frame in the final visual perception output frame sequence. The scene semantic label map is a pixel-level scene semantic category annotation map obtained by scene semantic segmentation of the final visual perception output frame.

[0150] The same DeepLabV3+ semantic segmentation model as in step S151 is used to perform semantic segmentation on each frame in the final visual perception output frame sequence Z1, outputting a semantic label map M12 of the same size as the input frame, with each pixel value being a semantic category identifier. In the wind power station scenario, semantic categories include wind turbine tower, blades, nacelle, transmission lines, sky, and ground.

[0151] Step S920: Perform semantic boundary displacement analysis on the scene semantic label map of two adjacent final visual perception output frames, extract the semantic region boundary of the same scene semantic category in the scene semantic label map of two adjacent frames, calculate the boundary displacement vector field of the semantic region boundary between two adjacent frames, and the boundary displacement vector field describes the position change of each boundary pixel on the semantic region boundary between adjacent frames.

[0152] For two adjacent frames M12t and M12t+1, extract the coordinates of boundary pixels of the same semantic category. For each boundary pixel in M12t, search for the nearest boundary pixel of the same semantic category in M12t+1; the difference between their coordinates is the displacement vector V6 of that boundary pixel. The V6 of all boundary pixels constitute the boundary displacement vector field M13.

[0153] Step S930: Perform semantic boundary displacement rationality judgment on the boundary displacement vector field, convert the physical allowable displacement range corresponding to the scene semantic category to which the boundary pixel belongs into the image allowable displacement range through the visual sensing optical path parameters of the inspection agent, and then compare the boundary displacement vector of each boundary pixel in the boundary displacement vector field with the corresponding image allowable displacement range, and mark the boundary pixel whose boundary displacement vector exceeds the image allowable displacement range as semantic boundary oscillation point.

[0154] The disparity range between adjacent frames is calculated based on the drone's flight speed and frame rate. Combined with the physical dimensions corresponding to the scene's semantic category, the permissible displacement range of the image for each semantic category is determined. The modulus of V6 is compared with this range, and boundary pixels exceeding the range are marked as semantic boundary oscillation points J1.

[0155] Step S940: Perform semantic boundary stabilization processing on the semantic boundary oscillation point, extract the semantic boundary position of the scene semantic category to which the semantic boundary oscillation point belongs in the final visual perception output frames of the preceding and following frames, perform boundary trajectory smoothing fitting based on the time series of the semantic boundary positions in the final visual perception output frames of the preceding and following frames to obtain the smoothed semantic boundary trajectory, and correct the boundary position of the semantic boundary oscillation point to the smoothed semantic boundary trajectory.

[0156] For each oscillation point in J1, take the corresponding position of the same semantic category boundary in each preceding and following m1 frames, and use a quadratic polynomial to perform least squares fitting on the coordinates of these positions to correct the coordinates of J1 onto the fitted curve.

[0157] Step S950: Perform semantic region internal texture solidification processing on the two adjacent final visual perception output frames after semantic boundary stabilization processing, extract the semantic region internal texture features of the same scene semantic category in the two adjacent frames, perform temporal low-pass filtering on the semantic region internal texture features, filter out the temporal high-frequency fluctuation components of texture features introduced by inter-frame flicker elimination processing, and retain the temporal low-frequency stable components of texture features.

[0158] For each semantic region in each frame, extract the sequence of texture feature values. Apply mean filtering to the time dimension using a sliding window of length m2. The filtering result is the time-domain low-frequency stable component. Replace the pixel value corresponding to the original texture feature value with the filtering result.

[0159] Step S960: Combine the frame sequences after semantic boundary stabilization and semantic region internal texture solidification into a semantically stable final visual perception output frame sequence, and update the final visual perception output frame sequence with the semantically stable final visual perception output frame sequence.

[0160] The processed frame sequence is combined into Z3 and assigned to Z1 to complete the update of the final visual perception output frame sequence.

[0161] Step S1010: Perform extreme weather medium concentration field inversion on the defective visual frame sequence, extract the spatial distribution of the attenuation degree of extreme weather medium on scene light transmission in each defective visual frame, and invert and calculate the medium concentration field distribution of extreme weather medium in the three-dimensional scene space based on the attenuation degree spatial distribution. The medium concentration field distribution describes the spatial concentration change of extreme weather medium within the three-dimensional space covered by the visual sensing optical path of the inspection agent.

[0162] The dark channel prior algorithm is used to estimate the medium transmission graph of each missing visual frame. Where Ic is the pixel color channel value, Ac is the atmospheric light value, and ω is the retention coefficient. The medium transport map t and scattering coefficient β are used to determine the medium concentration value. By combining the UAV pose and multi-view geometric relationships, the two-dimensional medium concentration value is back-projected onto a three-dimensional voxel mesh to form the medium concentration field distribution V7.

[0163] Step S1020: Convert the medium concentration field distribution into a priori map of optical path degradation intensity, and perform integral projection of the medium concentration field distribution in the three-dimensional scene space along the visual sensing optical path direction of the inspection agent to obtain a two-dimensional priori map of optical path degradation intensity corresponding to each frame of defective visual frame. The two-dimensional priori map of optical path degradation intensity represents the spatial distribution of the expected optical path degradation intensity caused by extreme weather medium concentration on the image plane.

[0164] Line integration is performed on the medium concentration field distribution V7 along the incident light direction of each pixel. The integration result is the optical path degradation intensity prior map M14. The value of each pixel in M14 is the integral value of the medium concentration σ along the incident light path of that pixel.

[0165] Step S1030: The two-dimensional optical path degradation intensity prior map is introduced as an optical path degradation separation guide map into the optical path degradation separation process. In the process of decomposing each frame of the defective visual frame into an optical path degradation trace distribution map, the two-dimensional optical path degradation intensity prior map is used as the initial estimate for optical path degradation trace separation. The expected spatial distribution of optical path degradation intensity indicated by the two-dimensional optical path degradation intensity prior map is used to constrain the generation range of the optical path degradation trace distribution map, so that the spatial distribution of the optical path degradation trace distribution map is consistent with the spatial distribution of the two-dimensional optical path degradation intensity prior map.

[0166] In the optical path degradation candidate region determination in step S122, M14 is normalized and added as an additional weight term to the determination condition. The modified determination condition is that pixels with a gradient magnitude less than the median gradient magnitude and a normalized M14 value greater than T17 are marked as optical path degradation candidate regions. This makes the selection of degradation candidate regions not only dependent on gradient features, but also subject to the prior constraint of degradation intensity obtained from physical inversion.

[0167] Step S1040: Use the medium concentration field distribution to impose a three-dimensional motion consistency constraint on the construction of the optical path degradation trace offset field between adjacent frames. Based on the three-dimensional spatial displacement of the medium concentration field distribution between adjacent frames, derive the expected offset vector field of the optical path degradation trace on the image plane. Use the expected offset vector field as the initial value of the optical path degradation trace offset field between adjacent frames to correct the estimation deviation of the optical path degradation trace offset field caused by the two-dimensional image feature matching error.

[0168] A 3D scene flow estimation is performed on V7 between frame t and frame t+1 to obtain the displacement field of the medium concentration field in 3D space. This 3D displacement field is then projected onto the image plane to obtain the expected offset vector field V8. In step S134, the original optical path degradation trace offset field V1 and V8 are weighted and fused. The fused result replaces the original V1, reducing the offset field estimation deviation caused by feature matching error.

[0169] Step S1050: Use the medium concentration field distribution to spatially adaptively adjust the constraint strength of the intra-frame spatial context constraint field. In the spatial region where the extreme weather medium concentration is higher in the medium concentration field distribution, enhance the constraint strength of the intra-frame spatial context constraint field. In the spatial region where the extreme weather medium concentration is lower, weaken the constraint strength of the intra-frame spatial context constraint field, so that the repair strength of the structure and texture co-repair matches the severity of extreme weather degradation.

[0170] The normalized value of M14 is used as the constraint strength adjustment factor. After generating the intra-frame spatial context constraint field M6 in step S145, the amplitude of M6 is multiplied by the adjustment coefficient (1 + γ × normalized value of M14), where γ is the adjustment coefficient. The higher the medium concentration, the greater the amplitude enhancement of the constraint field, so that the repair strength matches the severity of degradation.

[0171] The method of this invention is used to process a continuous 60-frame visual data sequence under heavy rain conditions (rainfall exceeding 30 mm / h). After independent frame-by-frame repair of raindrop streaks and dynamic water mist in the heavy rain, noticeable brightness and color flickering occurs between adjacent frames. For example... Figure 2 As shown, Figure 2 The red curve represents the temporal variation of flicker intensity in the fully visually repaired frame sequence F1 before step S150. The flicker intensity fluctuates drastically within the range of 0.15 to 0.35, with a variance of 184.6. The curve exhibits obvious high-frequency sawtooth patterns, with two distinct flicker peak clusters appearing at frames 15-20 and 35-42, corresponding to the periods of intensified rainstorm intensity. This reflects the inconsistency in brightness and color between adjacent frames caused by independent frame-by-frame repair. After performing the temporal consistency constraint processing in steps S151 to S155, the green curve in the figure represents the temporal variation of flicker intensity in the temporally consistent visually repaired frame sequence F2. The flicker intensity stabilizes within the range of 0.02 to 0.06, the variance of fluctuation decreases to 24.3, the curve becomes smoother, and the high-frequency sawtooth fluctuations are effectively suppressed. Step S151 divides each frame into multiple semantically coherent region units using the DeepLabV3+ semantic segmentation model. Step S152 establishes the corresponding association relationship between semantically coherent region units in adjacent frames. Step S153 calculates the brightness fluctuation amplitude and color fluctuation amplitude of each pair of matching units. Step S154 spatially aggregates the temporal flicker intensity distribution of all matching pairs to construct a temporal consistency constraint flow M8 covering the entire frame. Step S155 uses M8 to adjust the brightness and color of each pixel position. Statistically, after processing, the inter-frame flicker index decreased from 0.247 to 0.032, an improvement of 87.0%; the brightness fluctuation variance decreased from 184.6 to 24.3, an improvement of 86.8%; the color shift ΔE decreased from 9.8 to 1.2, an improvement of 87.8%; and the semantic coherence increased from 68.4% to 96.8%.

[0172] Furthermore, such as Figure 3 The image shown is the main visual display area of ​​the intelligent agent visual perception completion and noise reduction system for new energy power stations under extreme weather conditions, provided in an embodiment of the present invention.

[0173] The main visual display area is divided into two rows of windows. The upper row of windows is used to display intermediate processing results: the "Original Frame" window displays the original defective image input by the agent in real time, corresponding to the original defective visual frame in step S110; the "Degradation Trace Map" window displays the distribution of separated optical path degradation traces in the form of a heat map, corresponding to the optical path degradation trace distribution map generated in step S125; the "Residual Component Map" window displays the residual component image after removing the degradation traces, corresponding to the optical path disturbance residual component map generated in step S126; and the "Preliminary Repair Frame" window displays the preliminary repair results after optical path compensation and superposition, corresponding to the preliminary visual repair frame sequence generated in step S136. The bottom row of windows displays the core and final results: the "Structure Layer / Texture Layer" window can be switched via tabs to display the structure layer or texture layer decomposed in the spatial co-repair process, corresponding to the texture layer and structure layer extracted in step S141; the "Fully Repaired Frame" window displays the result after spatial co-repair, corresponding to the fully visually repaired frame sequence generated in step S146; the "Temporal Constraint Flow" window visualizes the temporal consistency constraint flow in vector field form, corresponding to the temporal consistency constraint flow constructed in step S154; the "Final Output Frame" window highlights the final output image after all processing, corresponding to the final visually perceived output frame sequence generated in step S156. Each window has a frame number display and playback control buttons below it for easy review of the processing effect frame by frame.

[0174] In an exemplary embodiment, a smart agent visual perception completion and denoising system for new energy power plants under extreme weather conditions is provided. This system can be a terminal, server, etc., and its internal structure includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, near-field communication, or other technologies. When the computer program is executed by the processor, it implements a smart agent visual perception completion and denoising method for new energy power plants under extreme weather conditions. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, or a button, trackball, or touchpad set on the shell of the intelligent agent visual perception completion and noise reduction system for extreme weather conditions in new energy power stations, or an external keyboard, touchpad, or mouse, etc.

[0175] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A method for intelligent agent visual perception completion and noise reduction for new energy power stations under extreme weather conditions, characterized in that, The method includes: Receive raw visual data streams collected by the inspection agent in extreme weather conditions. The raw visual data streams contain a sequence of missing visual frames with consecutive frame numbers. Each missing visual frame in the sequence of missing visual frames carries the agent's pose record at the time of collection. The original visual data stream is subjected to optical path degradation separation processing, and each frame of the defective visual frame is decomposed into an optical path degradation trace distribution map and an optical path disturbance residual component map, and the intelligent agent pose recording sequence corresponding to the defective visual frame sequence is retained. Based on the optical path degradation trace distribution map, an inter-frame optical path degradation trace offset field is constructed. Based on the inter-frame optical path degradation trace offset field and the agent pose recording sequence, an optical path degradation compensation frame sequence is generated. The optical path degradation compensation frame sequence is superimposed frame by frame with the frame sequence corresponding to the optical path disturbance residual component map to obtain a preliminary visual repair frame sequence. Based on the preliminary visual restoration frame sequence, an intra-frame spatial context constraint field is constructed, and the intra-frame spatial context constraint field is used to perform structural and texture co-restoration on the preliminary visual restoration frame sequence to generate a complete visual restoration frame sequence. Based on the scene semantic coherence between adjacent repair frames in the complete visual repair frame sequence, a temporal consistency constraint flow is constructed. The temporal consistency constraint flow is then used to eliminate inter-frame flicker in the complete visual repair frame sequence, generating the final visual perception output frame sequence.

2. The method for intelligent agent visual perception completion and noise reduction for new energy power stations under extreme weather conditions as described in claim 1, characterized in that, The process of performing optical path degradation separation on the original visual data stream decomposes each defective visual frame into an optical path degradation trace distribution map and an optical path perturbation residual component map, and retains the agent pose recording sequence corresponding to the defective visual frame sequence, including: All frames of the defective visual frame sequence are extracted frame by frame from the original visual data stream. For each defective visual frame, pixel brightness level decomposition is performed to generate a brightness level component sequence. The brightness level component sequence consists of a coarse brightness base layer and a progressive detail brightness residual layer. Gradient direction histogram statistics are performed on the rough brightness base layer to obtain the gradient direction distribution of the rough brightness base layer, and pixel regions whose gradient magnitude is lower than the median gradient magnitude in the gradient direction distribution of the rough brightness base layer are marked as candidate regions for optical path degradation. Optical path degradation direction field estimation is performed on the candidate region of optical path degradation, and the principal direction angle of optical path degradation at each pixel position is calculated within the candidate region of optical path degradation to form an optical path degradation direction map; The optical path degradation pattern is compared with the progressive detail brightness residual layer layer by layer to ensure consistent orientation. The detail brightness residual components whose orientation angle with the optical path degradation pattern is less than the orientation angle threshold are extracted as fine-grained traces of optical path degradation. The rough brightness base layer component corresponding to the optical path degradation candidate region is aggregated with the fine-grained traces of optical path degradation to generate the distribution map of the optical path degradation traces. The remaining brightness residual components that were not aggregated into the optical path degradation trace distribution map are separated from the brightness level component sequence, and the remaining brightness residual components are reconstructed into the optical path disturbance residual component map; A mapping association is established between the frame number of each defective visual frame and the corresponding intelligent agent pose record in the intelligent agent pose record sequence, and the intelligent agent pose record sequence is retained.

3. The method for intelligent agent visual perception completion and noise reduction for new energy power stations under extreme weather conditions, as described in claim 1, is characterized in that... The process involves constructing an inter-frame optical path degradation trace offset field based on the optical path degradation trace distribution map, generating an optical path degradation compensation frame sequence based on the inter-frame optical path degradation trace offset field and the agent pose recording sequence, and then superimposing the optical path degradation compensation frame sequence with the frame sequence corresponding to the optical path disturbance residual component map frame by frame to obtain a preliminary visual restoration frame sequence, including: Optical path degradation feature points are extracted from each frame of the optical path degradation trace distribution map sequence to obtain a set of optical path degradation feature points. Each optical path degradation feature point in the set carries the spatial coordinate position of the feature point in the optical path degradation trace distribution map. Cross-frame feature point pairing is performed on the optical path degradation trace distribution map of two adjacent frames. The coordinate displacement vector of the successfully paired cross-frame optical path degradation feature point pair is calculated. The optical path degradation trace offset field between adjacent frames is constructed based on the coordinate displacement vector. Extract the relative pose change of the agent corresponding to adjacent frames from the agent pose recording sequence, and project the relative pose change of the agent onto the image plane of the defective visual frame to obtain the image plane projection pose change vector. The optical path degradation trace offset field between adjacent frames is fused with the image plane projection pose change vector to obtain a fused optical path degradation compensation vector field, and the optical path degradation compensation intensity field is determined according to the amplitude of the fused optical path degradation compensation vector field. For each frame of the optical path degradation trace distribution map in the optical path degradation trace distribution map sequence, the pixel optical path is reverse-migrated along the compensation direction indicated by the fused optical path degradation compensation vector field and according to the compensation intensity indicated by the optical path degradation compensation intensity field to generate the optical path degradation compensation frame sequence. Each optical path degradation compensation frame in the optical path degradation compensation frame sequence is superimposed with the frame with the same frame number in the frame sequence corresponding to the optical path disturbance residual component map, pixel-by-pixel brightness values ​​are superimposed to obtain the corresponding frame in the preliminary visual restoration frame sequence.

4. The method for intelligent agent visual perception completion and noise reduction for new energy power stations under extreme weather conditions, as described in claim 1, is characterized in that... The step of constructing an intra-frame spatial context constraint field based on the preliminary visual restoration frame sequence, and using the intra-frame spatial context constraint field to perform structure-texture co-restoration on the preliminary visual restoration frame sequence to generate a complete visual restoration frame sequence includes: Spatial texture structure decomposition is performed on each preliminary visual restoration frame in the preliminary visual restoration frame sequence to extract the texture layer and structure layer of the preliminary visual restoration frame. The texture layer contains the micro-texture information of the preliminary visual restoration frame, and the structure layer contains the macro-geometric boundary information of the preliminary visual restoration frame. A geometric boundary coherence analysis is performed on the structural layer to identify the locations of geometric boundary breaks caused by extreme weather. Boundary breakpoints are then paired at these locations to obtain a set of paired breakpoints. Based on the spatial distance and local boundary orientation of each pair of fracture boundary pairing points in the set of fracture boundary pairing points, a boundary coherence constraint field is constructed. The boundary coherence constraint field is used to indicate the boundary connection direction that should be restored at the location of the geometric boundary fracture. A texture block similarity search is performed on the texture layer. Similar texture blocks that match the damaged texture regions are searched within the undamaged texture regions of the texture layer, and a texture block migration constraint field is constructed based on the spatial offset of the similar texture blocks. The boundary coherence constraint field and the texture block migration constraint field are constrained and fused to form the intra-frame spatial context constraint field; The broken boundaries and damaged texture regions of each preliminary visual restoration frame in the preliminary visual restoration frame sequence are synchronously repaired using the intra-frame spatial context constraint field. The boundary connection direction and similar texture blocks are migrated and filled to the corresponding positions to generate the complete visual restoration frame sequence.

5. The method for intelligent agent visual perception completion and noise reduction for new energy power stations under extreme weather conditions according to claim 1, characterized in that, The step of constructing a temporal consistency constraint flow based on the scene semantic coherence between adjacent repair frames in the fully visually repaired frame sequence, and using the temporal consistency constraint flow to perform inter-frame flicker elimination on the fully visually repaired frame sequence to generate the final visual perception output frame sequence includes: Each frame of the complete visual restoration frame sequence is divided into scene semantic regions, and each frame of the complete visual restoration frame is divided into multiple semantically coherent region units. Each semantically coherent region unit corresponds to a visual element block with independent scene semantics in that frame of the complete visual restoration frame. Cross-frame semantic matching is performed on the semantically coherent region units of two adjacent fully visually restored frames to establish the corresponding association relationship between the semantically coherent region units between adjacent frames, forming a set of semantically coherent region unit matching pairs. Calculate the brightness fluctuation amplitude and color fluctuation amplitude of each pair of matched semantic coherent region units in the set of matching semantic coherent region units between adjacent frames, and generate the temporal flicker intensity distribution of the semantic coherent region unit based on the brightness fluctuation amplitude and color fluctuation amplitude. Spatially aggregate the temporal flicker intensity distribution of all matching pairs in the semantically coherent region unit matching pair set to construct a temporal consistency constraint flow covering the entire frame. The temporal consistency constraint flow records the brightness and color adjustment vector of each pixel position between adjacent frames based on the pixel position index. The brightness and color of each pixel position in each fully visually restored frame in the fully visually restored frame sequence are adjusted using the temporal consistency constraint flow to obtain a temporally consistent visually restored frame sequence. All frames in the temporally consistent visual restoration frame sequence are arranged in order of frame number to generate the final visual perception output frame sequence.

6. The method for intelligent agent visual perception completion and noise reduction for new energy power stations under extreme weather conditions according to claim 1, characterized in that, After performing optical path degradation separation processing on the original visual data stream, decomposing each defective visual frame into an optical path degradation trace distribution map and an optical path perturbation residual component map, the method further includes: The optical path degradation trace distribution map is subjected to optical path degradation source inversion and tracking processing. Each frame of the optical path degradation trace distribution map is mapped to the visual sensing optical path model of the inspection agent. The optical path degradation cause type of the optical path degradation trace is analyzed on an optical path cross section by optical path within the visual sensing optical path model. The optical path degradation cause type includes optical path medium scattering degradation type and optical path medium refraction offset degradation type. According to the type of optical path degradation, the optical path degradation trace distribution map is labeled with degradation trace attributes. Each optical path degradation pixel unit in the optical path degradation trace distribution map is marked as an optical path medium scattering degradation pixel unit or an optical path medium refraction offset degradation pixel unit, thereby generating an optical path degradation trace distribution map with degradation cause attribute annotation. The distribution map of optical path degradation traces labeled with degradation cause attributes is clustered into pixel units. The optical path degradation pixel units that are spatially continuous and have the same degradation cause type are aggregated into optical path degradation homogeneous blocks, resulting in a set of optical path degradation homogeneous blocks. Each optical path degradation homogeneous block in the set of optical path degradation homogeneous blocks has a unified degradation cause type and continuous spatial boundary. For each homogeneous optical path degradation block in the set of homogeneous optical path degradation blocks, the degradation degree is quantified, the optical path degradation intensity value of all optical path degradation pixel units in the homogeneous optical path degradation block is extracted, the average optical path degradation intensity of the homogeneous optical path degradation block is calculated, and the average optical path degradation intensity is assigned as the degradation degree value of the homogeneous optical path degradation block; Based on the degradation cause type and degradation degree value of the optical path degradation homogeneous block, a degradation cause differentiation compensation strategy is executed for optical path degradation homogeneous blocks with different degradation cause types. Optical path scattering inverse filtering compensation is executed for the optical path degradation homogeneous block where the optical path medium scattering degradation pixel unit is located. Optical path refraction offset reverse mapping compensation is executed for the optical path degradation homogeneous block where the optical path medium refraction offset degradation pixel unit is located, generating a refined optical path degradation compensation frame sequence. The refined optical path degradation compensation frame sequence is superimposed frame by frame with the frame sequence corresponding to the optical path disturbance residual component map to update the preliminary visual repair frame sequence.

7. The method for intelligent agent visual perception completion and noise reduction for new energy power stations under extreme weather conditions according to claim 1, characterized in that, After performing optical path degradation separation processing on the original visual data stream, decomposing each defective visual frame into an optical path degradation trace distribution map and an optical path perturbation residual component map, the method further includes: The optical path disturbance residual component map is subjected to disturbance component spectrum analysis. Each frame of the optical path disturbance residual component map is transformed from the spatial domain to the frequency domain to obtain the optical path disturbance residual spectrum map. In the optical path disturbance residual spectrum map, the cluster of abnormal spectral peaks corresponding to the dynamic medium disturbance of extreme weather is identified. Based on the frequency distribution range and amplitude distribution characteristics of the spectral anomaly peak cluster, an extreme weather disturbance frequency domain filter is constructed. The extreme weather disturbance frequency domain filter is used to suppress the frequency components corresponding to the spectral anomaly peak cluster from the optical path disturbance residual spectrum. The extreme weather disturbance frequency domain filter is used to perform frequency domain filtering on the optical path disturbance residual spectrum, filtering out the frequency components corresponding to the abnormal peak clusters in the spectrum, and obtaining the filtered optical path disturbance residual spectrum. The filtered optical path disturbance residual spectrum is inversely transformed from the frequency domain back to the spatial domain to obtain the de-disturbed optical path residual component map. Compared with the original optical path disturbance residual component map, the de-disturbed optical path residual component map removes the random noise disturbance component caused by dynamic medium disturbance in extreme weather. The residual detail enhancement process is performed on the residual component map of the de-disturbed optical path to extract the weak visual detail signals that are weakened in the residual component map of the de-disturbed optical path. The weak visual detail signals are then amplified by signal gain to obtain the residual component map of the detail-enhanced optical path. The initial visual repair frame sequence is updated by superimposing the optical path degradation compensation frame sequence with the frame sequence corresponding to the detail enhancement optical path residual component map frame by frame.

8. The method for intelligent agent visual perception completion and noise reduction for new energy power stations under extreme weather conditions, as described in claim 1, is characterized in that... After constructing an intra-frame spatial context constraint field based on the preliminary visual restoration frame sequence, and using the intra-frame spatial context constraint field to perform structure-texture co-restoration on the preliminary visual restoration frame sequence to generate a complete visual restoration frame sequence, the method further includes: Inter-frame optical flow continuity analysis is performed on the fully visually restored frame sequence, and dense optical flow vector fields between two adjacent fully visually restored frames are extracted. Regions in the dense optical flow vector field whose optical flow vector direction deviates from the direction deviation threshold of the neighboring optical flow vector direction are marked as suspected restoration artifact regions. Temporal backtracking verification is performed on the suspected repair artifact region. Verification pixels are selected within the suspected repair artifact region, and the pixel value evolution trajectory of the verification pixels in the defective visual frame sequence and the preliminary visual repair frame sequence is traced back along the time axis. Based on the pixel value evolution trajectory, it is determined whether the suspected repair artifact region is a false texture region. If the pixel value evolution trajectory shows a non-continuous jump pattern in the time domain, the suspected repair artifact region is determined to be a false texture region. The false texture region is subjected to texture realism regeneration processing. The real texture samples of the spatial neighborhood of the false texture region in the complete visual restoration frame sequence are extracted. The texture content in the false texture region is regenerated according to the texture primitive arrangement rules of the real texture samples of the spatial neighborhood, and the artifact elimination visual restoration frame sequence is obtained. The artifact-eliminating visual restoration frame sequence is subjected to inter-frame texture consistency verification. The texture primitive arrangement rules at the same spatial location between adjacent frames are extracted. Spatial locations with inconsistent texture primitive arrangement rules are marked as residual inconsistent texture regions. Local texture remapping is performed on the residual inconsistent texture regions to generate a consistency-enhanced visual restoration frame sequence.

9. The method for intelligent agent visual perception completion and noise reduction for new energy power stations under extreme weather conditions according to claim 2, characterized in that, The method further includes receiving the raw visual data stream collected by the inspection agent in extreme weather conditions, wherein the raw visual data stream contains a sequence of missing visual frames with consecutive frame numbers, and each missing visual frame in the sequence carries the agent's pose record at the time of acquisition. The original visual data stream is input into a pre-built inter-frame optical path degradation evolution prediction model, which is used to predict the optical path degradation trend map of the next frame of the defective visual frame based on the optical path degradation traces and the pose changes of the agent in multiple consecutive defective visual frames. Intermediate layer representation features are extracted from the inter-frame optical path degradation evolution prediction model. These intermediate layer representation features are used to characterize the implicit evolution state of the extreme weather degradation process. The intermediate layer representation features are used as the external guiding signal for the optical path degradation separation process. The external guiding signal is conditionally fused with the pixel brightness level decomposition process of each frame of the defective visual frame. When generating the brightness level component sequence, the external guiding signal is used as a conditional constraint to adjust the separation threshold between the coarse brightness base layer and the progressive detail brightness residual layer, so that the coarse brightness base layer contains more complete extreme weather optical path degradation traces. The optical path degradation direction field estimation process is guided and corrected by injecting the prior information of optical path degradation direction implied in the external guidance signal into the calculation of the main optical path degradation direction angle of each pixel position in the optical path degradation candidate region, thereby correcting the optical path degradation direction estimation deviation caused by dynamic changes in extreme weather. The extraction process of fine-grained traces of optical path degradation is guided and constrained. The significant distribution information of optical path degradation traces hidden in the external guidance signal is fused with the layer-by-layer directional consistency comparison process. The directional angle threshold is adjusted so that the directional angle threshold adaptively matches the extreme weather degradation intensity of the current frame. A global consistency constraint is applied to the generation process of the optical path degradation trace distribution map. The continuity information of optical path degradation evolution between adjacent frames provided by the external guidance signal is used as a constraint condition to correct the inter-frame inconsistency of optical path degradation traces in the optical path degradation trace distribution map, thereby obtaining an optical path degradation trace distribution map with enhanced temporal consistency.

10. A visual perception completion and noise reduction system for intelligent agents used in extreme weather conditions at new energy power stations, characterized in that, include: processor; A machine-readable storage medium for storing machine-executable instructions of the processor; The processor is configured to execute the intelligent agent visual perception completion and denoising method for new energy power stations under extreme weather conditions according to any one of claims 1 to 9 by executing the machine-executable instructions.

Images