An image data missing frame completion method based on a conditional diffusion model

Abstract

The application discloses an image data missing frame completion method based on a conditional diffusion model and belongs to the field of remote sensing big data. The method takes a diffusion model under a mask mechanism as a generation framework, first extracts pixel-level motion features between effective frames by using inter-frame attention, then constructs a forward and reverse double-path feature extraction link in the time dimension, and realizes bidirectional time sequence completion of the full-sequence basic motion features through time step scaling, combines a nonlinear correction module, corrects spatial isolated deviation and time sequence rate deviation through spatial neighborhood attention and time window attention respectively, so that the complex nonlinear motion law of a cloud body can be adapted, and finally the motion features of fine modeling are taken as conditional information to be injected into the diffusion model, so that the missing frames of the full sequence are generated at one time.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing big data, and in particular relates to a method for completing missing frames in image data based on a conditional diffusion model. Background Technology

[0002] With the rapid development of satellite remote sensing technology and the continuous improvement of the space observation system, high spatiotemporal resolution remote sensing cloud images have become a core data source for capturing dynamic changes in the atmosphere and analyzing cloud movement patterns. They are widely used in many key fields such as ecological environment assessment, meteorological and climate research, extreme weather early warning, and battlefield environment monitoring. Satellite observation platforms such as Himawari-8 / 9 and the MODIS series satellites can achieve all-weather, routine monitoring of the global atmosphere and cloud systems, continuously generating time-series cloud image data. However, during the remote sensing data acquisition and transmission process, remote sensing cloud images are prone to data loss due to various factors such as obstruction by observation equipment, transmission link failures, and abnormal data storage. This can manifest as missing single frames or multiple consecutive missing frames in an image sequence. This directly affects the availability of downstream research data and the accuracy of downstream applications. Therefore, solving the problem of missing cloud image sequences with high quality is of great significance.

[0003] Inpainting of missing data in remote sensing cloud image sequences is essentially an interpolation and completion problem for spatiotemporal image sequences. Its core challenge lies in ensuring that the inpainted result simultaneously meets the dual requirements of pixel accuracy and structural realism. However, remote sensing cloud image sequences contain rich spatiotemporal dynamic information, encompassing the entire process of cloud formation, dissipation, drift, and evolution. Spatially, the texture and structure of the inpainted frame must be identical to that of the real cloud. Figure 1 Therefore, the timing needs to conform to the nonlinear motion law of clouds driven by the atmosphere, so that the repaired sequence presents a natural and continuous motion evolution trend.

[0004] Traditional inpainting methods primarily rely on simple spatial interpolation and basic temporal filling, depending on the temporal changes of adjacent effective pixels and preceding and following frames. They fill in the missing frames by fitting pixel values ​​of corresponding regions in the preceding and following frames. While this approach is simple in principle, its accuracy is limited. Optical flow-based interpolation is a major research direction in this field. It derives the pixel positions of missing frames by estimating pixel motion vector fields. However, when the estimation interval is large, optical flow methods are prone to cumulative errors, and they do not fully consider the spatiotemporal correlation of sequences, making them unsuitable for handling complex cloud movements and irregular missing scenes. The development of deep learning technology has provided new ideas for cloud image sequence missing inpainting methods. Some researchers rely on sequence feature encoding techniques and attention mechanisms to capture spatiotemporal sequence correlations, but limited image generation capabilities lead to problems such as blurred details and texture distortion in the inpainted frames.

[0005] In recent years, with the rapid development of generative models, related methods have also been widely applied to image sequence inpainting tasks. Diffusion models, with their powerful generative capabilities and stability, have become the mainstream method for high-fidelity image generation. However, general diffusion models lack specific modeling for cloud motion evolution, making it difficult to uncover the motion correlation patterns of clouds when directly applied to inpainting missing remote sensing cloud image sequences. Another problem is that most of the aforementioned methods have poor adaptability when facing random missing scenes, easily resulting in distortion of motion trajectory, blurring, ghosting, and artifacts in the inpainted frames. Summary of the Invention

[0006] The technical problem this invention aims to solve is to overcome the shortcomings of existing technologies. To this end, this invention explicitly models cloud motion and mines motion evolution information within the sequence context, proposing a method for completing missing frames in image data based on a conditional diffusion model.

[0007] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0008] In a first aspect, the present invention provides an image data missing frame completion method based on a conditional diffusion model, comprising the following steps: inputting a remote sensing cloud image sequence with missing frames into a trained completion model; firstly, using a temporal completion module to perform bidirectional temporal modeling and linear interpolation to complete the effective frames before and after the missing frames in the sequence, obtaining a complete motion feature set covering the entire sequence; and then performing spatiotemporal nonlinear correction through a nonlinear correction module to obtain a corrected motion feature set, which serves as a guiding condition for the diffusion model; using Gaussian noise to replace the missing frames and together with the effective frames to form a noisy sequence input into the diffusion model; and after multiple denoising steps, obtaining the completion results for each missing frame.

[0009] Based on the above scheme, each step can be implemented in the following preferred manner.

[0010] As a preferred option of the first aspect mentioned above, the specific process of the temporal completion module is as follows: two valid frames adjacent to a missing frame are used to form a valid frame pair. The feature extraction module extracts features along the forward and reverse temporal sequences respectively, forming a set of forward motion features and a set of reverse motion features covering the remote sensing cloud image sequence, which together constitute the completed motion feature set.

[0011] As a preferred embodiment of the first aspect mentioned above, the internal processing flow of the feature extraction module is as follows:

[0012] S11. High-dimensional appearance features of effective frame pairs are extracted by a multi-scale convolutional architecture and formed into feature pairs. Pixel similarity and cross-frame motion trends in the feature pairs are jointly modeled by inter-frame attention. Two attention maps, one in forward time series and one in reverse time series, are obtained and neighborhood position information is weighted and aggregated. Then, the coordinate position corresponding to each feature vector of one high-dimensional appearance feature in the same feature pair is estimated when it is mapped to another high-dimensional appearance feature.

[0013] S12. By calculating the estimated deviation of the coordinate position pixel by pixel, the displacement vectors of the forward and reverse time series are obtained and encoded by a linear layer to obtain the forward and reverse motion features of the effective frame pairs. By scaling by time step, the forward and reverse motion features between each frame in the remote sensing cloud image sequence are obtained, which correspond to the forward motion feature set and the reverse motion feature set and form the complete motion feature set.

[0014] As a preferred embodiment of the first aspect, in the multi-scale convolutional architecture, the remote sensing cloud image sequence is first downsampled to a high-dimensional appearance feature space through multi-scale convolution to obtain the appearance features of each effective frame pair at each scale; then, appearance features are extracted and fused across scales through dilated convolution, thereby aligning the appearance features at each scale to a high-dimensional appearance feature with uniform dimensions.

[0015] As a preferred embodiment of the first aspect mentioned above, in the inter-frame attention joint modeling, when extracting forward temporal features, the preceding frame in the effective frame pair is used as the query vector, and the following frame is used as the key vector and value vector, respectively. When extracting reverse temporal features, the following frame is used as the query vector, and the preceding frame is used as the key vector and value vector, respectively.

[0016] As a preferred embodiment of the first aspect mentioned above, the specific process of obtaining the motion features between frames in the remote sensing cloud image sequence by time step scaling is as follows: For any missing frame between the preceding and following frames, the time interval between the preceding and following frames is calculated, the reciprocal of the time interval is used as the time step ratio, and the forward motion features from the preceding frame to the following frame and the reverse motion features from the following frame to the preceding frame are scaled using the time step ratio respectively, so as to obtain the motion features from the missing frame to the next frame and the motion features from the next frame to the missing frame, thereby obtaining the set of forward motion features composed of forward motion features between frames in the remote sensing cloud image sequence and the set of reverse motion features composed of reverse motion features between frames.

[0017] As a preferred embodiment of the first aspect mentioned above, the internal processing flow of the nonlinear correction module is as follows:

[0018] S21. Perform dimensional reshaping on any adjacent frame motion features in the complete motion feature set, and then input the dimensional reshaping motion features as query, key and value vectors into the spatial neighborhood attention module for spatial neighborhood attention calculation to obtain spatial attention enhancement features. These features are then fused with the dimensional reshaping motion features through residual connections to obtain spatial correction features.

[0019] S22. Using the spatially corrected features as the query vector, and the motion features corresponding to the effective frames after dimensional reshaping as the key and value vectors, input them together into the temporal window attention module to perform temporal window attention calculation, obtain the temporal attention enhanced features, and fuse them with the spatially corrected features through residual connection to obtain the spatiotemporal corrected features. Then, perform dimensional calibration and nonlinear fine-tuning to obtain the corrected motion feature set.

[0020] In a second aspect, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement an image data missing frame completion method based on a conditional diffusion model as described in any of the solutions in the first aspect above.

[0021] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for completing missing frames of image data based on a conditional diffusion model as described in any of the solutions of the first aspect above.

[0022] Fourthly, the present invention provides a computer electronic device, which includes a memory and a processor;

[0023] The memory is used to store computer programs;

[0024] The processor is configured to, when executing the computer program, implement a method for completing missing frames of image data based on a conditional diffusion model as described in any of the solutions of the first aspect above.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] This invention proposes a method for completing missing frames in image data based on a conditional diffusion model, using the concept of motion feature-constrained diffusion generation. This method achieves high-precision repair of missing frames in satellite remote sensing cloud image sequences. The method uses a diffusion model as the foundation for generating a framework. Its core function is to accurately model and conditionally fuse inter-frame motion features to precisely uncover cloud motion correlations between remote sensing cloud image sequences. This guides the diffusion model to generate missing frames consistent with the sequence's spatiotemporal characteristics, thus achieving consistent repair of remote sensing cloud image sequences.

[0027] The overall process of this invention can be summarized into three core steps: First, inter-frame motion feature extraction is performed based on inter-frame attention to capture pixel-level motion correlations between adjacent frames in the remote sensing cloud image sequence, completing the initial extraction of motion features; second, information loss due to single-direction modeling is compensated by forward and reverse dual-path feature extraction; combining linear filling and nonlinear correction strategies, temporal integrity of motion features is achieved and nonlinear motion laws are adapted; finally, high-quality modeled motion features are incorporated as conditional information into the reverse generation process of the diffusion model to constrain the generation direction and content features of missing frames, ultimately outputting a temporally coherent and spatially realistic remote sensing cloud image sequence repair result. This method, through bidirectional motion feature extraction and motion feature constraint generation, ensures the authenticity of the repaired cloud image content and strengthens the motion consistency of the sequence dimension, effectively solving the dual challenges of "spatial fidelity" and "temporal coherence" in remote sensing cloud image sequence repair. Attached Figure Description

[0028] Figure 1 This is an overall flowchart of the method of the present invention;

[0029] Figure 2 This is a schematic diagram of inter-frame motion feature extraction according to the present invention;

[0030] Figure 3 This is a schematic diagram of the inter-frame attention joint modeling of the present invention;

[0031] Figure 4 This is a schematic diagram of the local linear filling of the present invention;

[0032] Figure 5 This is a schematic diagram of the linear estimation at both ends of the present invention;

[0033] Figure 6 This is a schematic diagram of bidirectional temporal completion of motion features according to the present invention;

[0034] Figure 7 This is a schematic diagram of the spatiotemporal adaptive nonlinear correction of the present invention;

[0035] Figure 8 This is a schematic diagram of the spatial nonlinear correction of the present invention;

[0036] Figure 9 This is a schematic diagram of the time nonlinearity correction of the present invention;

[0037] Figure 10 This is a schematic diagram of the repair of the diffusion model for motion feature constraints in this invention;

[0038] Figure 11 This is a schematic diagram of the real remote sensing cloud image sequence used in this embodiment and its missing information;

[0039] Figure 12Comparison diagram of repair results using different methods provided in embodiments of the present invention;

[0040] Figure 13 This is a schematic diagram of a computer electronic device provided by the present invention. Detailed Implementation

[0041] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0042] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0043] like Figure 1 As shown, in a preferred embodiment of the present invention, the above-mentioned image data missing frame completion method based on conditional diffusion model includes the following steps.

[0044] The remote sensing cloud image sequence with missing frames is input into the trained inpainting model. First, the temporal inpainting module performs bidirectional temporal modeling and linear interpolation to fill in the valid frames before and after the missing frames in the sequence, obtaining a complete motion feature set covering the entire sequence. Then, the nonlinear correction module performs spatiotemporal nonlinear correction to obtain a corrected motion feature set, which is used as the guiding condition for the diffusion model. Gaussian noise is used to replace the missing frames and together with the valid frames, it forms a noisy sequence input into the diffusion model. After multiple denoising steps, the inpainting results for each missing frame are obtained.

[0045] The specific implementation process of each module of the repair model will be described in detail below.

[0046] While inter-frame motion feature extraction can derive motion features covering the entire sequence through effective frame pairs, the extraction process suffers from poor temporal continuity in scenarios where preceding frames are severely missing in some remote sensing cloud image sequences. Specifically, motion feature modeling in a single direction struggles to capture the trends and dynamics of continuous temporal changes in remote sensing clouds in complex missing scenarios such as severe preceding frame loss. Furthermore, inter-frame motion feature extraction in a single direction does not fully utilize subsequent effective frames in this scenario, resulting in a lack of continuous temporal change information and motion correlation information with subsequent effective frames. These issues all affect the temporal coherence and realism of the missing frame repair results generated by the subsequent diffusion model. To address this, this invention designs a temporal completion module. This module achieves temporal integrity interpolation completion of motion features through two processing flows: bidirectional temporal modeling and linear interpolation filling, providing motion constraints that combine temporal integrity and basic motion representation for subsequent processes.

[0047] The specific process of the temporal completion module is as follows: two valid frames adjacent to a missing frame are used to form a valid frame pair. The feature extraction module extracts features along the forward and reverse temporal sequences respectively, forming a set of forward motion features and a set of reverse motion features covering the remote sensing cloud image sequence, which together constitute the complete motion feature set.

[0048] It should be noted that in the bidirectional temporal modeling of this invention, a bidirectional temporal feature extraction path is constructed to compensate for information bias in a single direction. The bidirectional temporal feature extraction stage is carried out based on the high-quality inter-frame motion feature extraction capability of the feature extraction module. By adjusting the frame order logic corresponding to the query and key, symmetrical extraction of forward and reverse motion features is achieved, ensuring the consistency and complementarity of feature representation.

[0049] The aforementioned feature extraction module, as a core pre-processing step for repairing missing frames during remote sensing cloud image sampling and transmission, aims to accurately extract pixel-level displacement correlation information between frames from the retained valid frames in the remote sensing cloud image sequence. This generates structured basic motion features covering the entire sequence, providing structured input for subsequent cloud motion representation and diffusion model conditional fusion, decoupling motion and appearance. This module's core design philosophy is based on the separation of motion and appearance features and independent computational links, avoiding the fuzzy representation caused by the mixed extraction of motion and appearance features in traditional methods, or the high computational overhead caused by the separate extraction of motion and appearance features. Through high-dimensional feature extraction, inter-frame attention modeling, coordinate mapping, displacement vector calculation, and multi-time-step adaptation, combined with the continuity of remote sensing cloud motion and the task characteristics of random missing features, this module achieves efficient, accurate, and full-sequence coverage of motion features. It should be noted that the missing frames mentioned above refer to randomly missing frames in the remote sensing cloud image sequence. The remote sensing cloud image sequence may have one or more missing frames. Correspondingly, the complete frames that are not missing in the remote sensing cloud image sequence are valid frames. For example, in a remote sensing cloud image sequence with a fixed time step and a length of 6 frames, when frames 2, 3, and 5 are missing frames, frames 1, 4, and 6 are retained as valid frames.

[0050] Specifically, such as Figure 2 As shown, the internal processing flow of the feature extraction module is as follows:

[0051] S11. High-dimensional appearance features of effective frame pairs are extracted by a multi-scale convolutional architecture and formed into feature pairs. Pixel similarity and cross-frame motion trends in the feature pairs are jointly modeled by inter-frame attention. Two attention maps, one forward and one reverse, are obtained and their neighborhood position information is weighted and aggregated. Then, the most likely coordinate position of each feature vector of a high-dimensional appearance feature in the same feature pair when mapped to another high-dimensional appearance feature is estimated.

[0052] Furthermore, in the multi-scale convolutional architecture of S11 of the present invention, the remote sensing cloud image sequence is first downsampled to a high-dimensional appearance feature space through multi-scale convolution to obtain the appearance features of each effective frame pair at each scale; then, appearance features are extracted and fused across scales through dilated convolution, thereby aligning the appearance features at each scale to a high-dimensional appearance feature with unified dimensions.

[0053] In this embodiment S11, considering that directly performing inter-frame attention calculation on the original resolution remote sensing cloud image sequence would incur huge memory and computational overhead, this embodiment first uses a multi-scale convolutional architecture as the feature extractor. Considering that the subsequent process is performed between two valid frames before and after the missing frame, this invention constructs a valid frame pair from any two valid frames before and after the missing frame. In the above example, the first frame... With frame 4 Frame 4 With frame 6 Each represents a valid frame pair. For any valid frame pair... This invention defines a valid frame preceding a missing frame as a preceding frame and a valid frame following a missing frame as a following frame. In other words, it satisfies... hour, For the preceding frame, For subsequent frames, For frame index.

[0054] Then, this invention downsamples these effective frame pairs to a high-dimensional appearance feature space through multi-scale convolution, obtaining the appearance features of each effective frame pair at various scales. The appearance features at each scale are denoted as , Corresponding to the preceding frame In the Appearance features at various scales Corresponding subsequent frame In the Appearance features at various scales For scale levels, For the real number space, The height and width of the effective frame. This represents the number of feature channels. Subsequently, dilated convolution is used to extract and fuse cross-scale information, compensating for the lack of detail in single-scale features. Thus, each valid frame pair yields a feature pair with uniform dimensions, consisting of two high-dimensional appearance features. Composition, in which, For the preceding frame High-dimensional appearance features, For subsequent frames High-dimensional appearance features; For high-dimensional appearance feature dimensions, The aggregated feature dimensions.

[0055] Furthermore, in the inter-frame attention joint modeling of S11 of the present invention, when extracting forward temporal features, the preceding frame in the effective frame pair is used as the query vector, and the following frame is used as the key vector and value vector. When extracting reverse temporal features, the following frame is used as the query vector, and the preceding frame is used as the key vector and value vector.

[0056] In this embodiment, as Figure 3 As shown, taking a forward time series as an example, the method for obtaining the attention map of a forward time series is as follows:

[0057] 1) High-dimensional appearance features of preceding frames ,Will Middle pixel position corresponding feature vector Through the first linear projection matrix Mapped to query vector ;

[0058] 2) High-dimensional appearance features of subsequent frames , will be based on pixel position Centered on All eigenvectors within the spatial neighborhood are projected through the second linear projection matrix. Third linear projection matrix Mapped to key vectors and value vectors respectively:

[0059]

[0060]

[0061] in, High-dimensional appearance features In, by pixel position The first, centered and located within the spatial neighborhood 1 eigenvector; They are respectively The corresponding key vector and value vector; These are preset hyperparameters;

[0062] 3) The feature similarity is quantified by performing a dot product operation between the query vector and each key vector, and then normalized using the SoftMax function to obtain an attention map for motion feature extraction. :

[0063]

[0064] in, This is the matrix transpose. The projected feature dimension; the value at each position in the attention map. express and Pixel position in spatial neighborhood The higher the similarity value, the higher the probability that the two are corresponding cloud regions between frames, that is, the higher the probability that the same cloud structure is projected in different frames.

[0065] It should be noted that for reverse timing, the attention map is obtained in a similar way to that for forward timing. The difference is that the "high-dimensional appearance features of the preceding frame" in 1) needs to be replaced with the "high-dimensional appearance features of the following frame", and the "high-dimensional appearance features of the following frame" in 2) needs to be replaced with the "high-dimensional appearance features of the preceding frame". Other processes remain unchanged, and the attention map for reverse timing can be obtained.

[0066] Furthermore, in S11 of this invention, taking the forward time series as an example, the specific process of coordinate position estimation is as follows: in the coordinate graph In the middle, it will be combined with the high-dimensional appearance features of subsequent frames. The region corresponding to the spatial neighborhood is used as a coordinate image segment. Then, based on the attention map, the coordinate image segments are weighted pixel by pixel and summed to obtain the high-dimensional appearance features of the preceding frame. Each feature vector exist The most likely corresponding coordinate position :

[0067]

[0068] in, The pixel position in the coordinate image segment The formula uses the similarity in the attention map as weights to aggregate neighborhood location information, avoiding errors caused by matching a single location and achieving accurate estimation of the corresponding location.

[0069] In the specific process of the aforementioned coordinate position mapping, in order to accurately calculate the cloud motion displacement, this invention introduces a fixed coordinate reference template, namely a coordinate graph, the size of which is consistent with... Consistent. The location of each pixel in the coordinate graph is stored and normalized to... The relative coordinates of the interval, the coordinates of the top-left pixel coordinates of the high-dimensional appearance feature in the coordinate graph. The pixel coordinates of the lower right corner of the high-dimensional appearance feature correspond to the coordinates in the coordinate graph. The coordinates of the center pixel coordinates of the high-dimensional appearance feature correspond to the coordinates in the coordinate graph. Furthermore, it does not change with the input frame, providing a unified position reference benchmark for calculating motion displacement between adjacent frames.

[0070] It should be noted that for reverse time series, the above coordinate position estimation method is similar to that for forward time series, the difference being that the coordinate image segment needs to use the high-dimensional appearance features of the preceding frame. By keeping other processes unchanged, the high-dimensional appearance features of subsequent frames can be obtained from the region corresponding to the spatial neighborhood. Each feature vector exist The most likely corresponding coordinate position.

[0071] S12. By calculating the estimated deviation of the coordinate position pixel by pixel, the displacement vectors of the forward and reverse time series are obtained and encoded by a linear layer to obtain the forward and reverse motion features of the effective frame pairs. By scaling by time step, the forward and reverse motion features between each frame in the remote sensing cloud image sequence are obtained, which correspond to the forward motion feature set and the reverse motion feature set and form the complete motion feature set.

[0072] Furthermore, in S12 of the present invention, the displacement vector of the forward time sequence is obtained as follows: for the feature vector of the preceding frame , and put it in The most likely corresponding coordinate position As an estimated coordinate position, and then actual coordinate position With estimated coordinate position The difference is calculated to determine the estimated deviation of the coordinate position, which is then used as the pixel position. The pixel at that location is from the previous frame To the next frame displacement vector , which is the displacement vector of the positive time sequence.

[0073] It should be noted that the method for obtaining the shift vector in reverse timing is similar to the process described above, specifically: for the feature vector of the subsequent frame... , and put it in The most likely corresponding coordinate position is used as the estimated coordinate position, and then... The difference between the actual coordinate position and the estimated coordinate position is used to calculate the estimated coordinate position deviation, which is then used as the pixel position. The pixel at that location is from the subsequent frame. To the preceding frame displacement vector , which is the displacement vector in reverse time sequence.

[0074] Based on the aforementioned displacement vector, calculations are performed pixel-by-pixel and through a linear layer. Encoding is performed to obtain the encoded cloud from the preceding frame. To the next frame Positive motion characteristics of natural motion trends And the encoded cloud body from the subsequent frame To the preceding frame Reverse motion characteristics of natural motion trends ,in, This is the encoding vector dimension. This motion feature fully encodes the pixel-level motion trend of the entire remote sensing cloud image sequence, while maintaining representational separation and decoupling from appearance features.

[0075] In S12 of this invention, the specific process of obtaining the motion features between frames in the remote sensing cloud image sequence by time step scaling is as follows: For the preceding frame With subsequent frames Any missing frame between , For missing frame index, Calculate the time interval between the preceding and following frames. The reciprocal of this time interval is used as the proportion of the time step. And the forward motion features from the preceding frame to the following frame were analyzed using the time step ratio. And the reverse motion features from the later frame to the earlier frame. Scaling is performed to obtain the motion features from the missing frame to the next frame. And the motion features from the next frame to the missing frame. This allows us to obtain the set of positive motion features formed by the positive motion features between each frame in the remote sensing cloud image sequence. The set of inverse motion features consisting of inverse motion features between frames .in, These represent the positive motion features from frame 1 to frame 2, from frame 2 to frame 3, ..., from frame T-1 to frame T, respectively. These represent the reverse motion features from frame 2 to frame 1, from frame 3 to frame 2, ..., from frame T to frame T-1, respectively. This represents the length of the remote sensing cloud image sequence.

[0076] As can be seen from the above process, for missing frames in a remote sensing cloud image sequence, this invention only needs to calculate the motion features of the valid frame pair once, and can flexibly adapt the motion features by scaling the time step, naturally supporting the derivation of motion features for any intermediate frame. Specifically, as shown in the example... Figure 4 As shown, this embodiment is based on the assumption of local linear motion. For any missing intermediate frame, the positive motion features representing the motion changes in the next frame are expressed as a percentage of the time step. right The scaling process yields a final set of forward motion features covering the entire sequence, providing complete and continuous motion feature data support for subsequent work. The scaling process for reverse motion features is similar. This design fully adapts to the random missing characteristics of remote sensing cloud image sequences; regardless of the location and number of missing frames, the missing frames can be filled in by deducing motion features from valid frame pairs, ensuring the robustness and versatility of the method.

[0077] Based on the aforementioned forward and reverse motion features, although the bidirectionally acquired motion feature set can achieve full sequence feature completion using a piecewise linear interpolation filling strategy, the absence of the first and last frames of the sequence or the consecutive loss at both ends of the sequence will result in the lack of corresponding preceding or following frames for reference. To address this problem of motion feature gaps and broken effective frame pairs, this invention also estimates based on the local linear motion assumption, using the nearest effective frame pair for derivation and substitution, ultimately obtaining a complete motion feature set covering the entire sequence. .

[0078] Specifically, such as Figure 5As shown, for problems such as missing or consecutively missing first frames or missing or consecutively missing last frames in a sequence, the motion features extracted from the inter-frame motion features of the nearest valid frame pair are used to replace the missing frame. Taking the missing or consecutively missing first frames as an example, assume that the time step corresponding to the missing frame is... The forward motion feature is used to represent the direction and amplitude of pixel flow from frame 1 to frame 2. Since there is no previous frame reference, it needs to be derived based on the most recent valid frame pair. Assuming the time step of the most recent valid frame pair closest to the missing frame is... and The alternative motion feature completion method is as follows:

[0079]

[0080] in, This represents the positive motion features from frame 2 to frame 4, multiplied by a scaling factor to represent the linear interpolated frame-by-frame motion features; This represents the positive motion features from frame 1 to frame 2. When the last frame of the sequence is missing or consecutively missing, the motion features of the last valid frame pair in the entire sequence are used instead. The motion feature completion method in this scenario is the same as the process described above, so it will not be repeated here.

[0081] Therefore, through the aforementioned linear interpolation strategy, this invention ultimately forms a set of forward motion features and a set of reverse motion features covering the remote sensing cloud image sequence, such as... Figure 6 As shown. Among them, forward motion features focus on capturing motion trends under natural time sequence, while reverse motion features can effectively supplement the correlation bias caused by rapid cloud deformation and local occlusion in forward motion features. By making full use of the information of subsequent frames in the sequence missing repair scenario, the model can extract and use more sufficient contextual information. The two, as basic motion priors, work together to provide comprehensive information support for subsequent work.

[0082] However, the assumption of local linear motion is difficult to adapt to the nonlinear motion of clouds caused by turbulence and atmospheric circulation. The motion feature set under linear interpolation is grouped according to its corresponding preceding and following frames, resulting in segmented fragmentation across the entire sequence. The motion feature sets extracted and linearly estimated from the same preceding and following frames show the same trend, and the first and last frames of the sequence have natural motion feature gaps due to the lack of preceding or following reference frames. Completing the motion feature set is affected by both the local linear motion assumption and the cumulative interpolation error, resulting in problems such as segmented feature fragmentation, lack of correlation across the entire sequence, and insufficient representation of spatiotemporal nonlinear motion. The true laws of pixel flow, or cloud motion, often exhibit spatiotemporal nonlinear characteristics, such as curvilinear motion and acceleration motion. Piecewise linear interpolation cannot accurately fit the characteristics of cloud motion, resulting in two core biases: temporally uniform motion and spatially isolated motion. The former distributes the cumulative motion between effective frames evenly across all intermediate frames, resulting in a constant motion rate and amplitude. This makes it impossible to adapt to the natural transition of motion rates between effective frames and the nonlinear characteristics of clouds, such as acceleration and deceleration due to airflow. The latter assumes that pixels move end-to-end according to the starting and ending points derived from the effective frames, without considering intermediate motion states. This makes it difficult to fit the distribution and correlation of motion relationships in space, such as nonlinear trajectories like local directional deflection or curvilinear flow of clouds caused by turbulence and circulation. Ultimately, this causes a disconnect between motion characteristics and the actual motion patterns of remotely sensed clouds, leading to problems such as stiff motion and distorted trajectories in the repair frames.

[0083] To address this, this invention designs a nonlinear correction module centered on an attention mechanism. This module leverages the attention mechanism to mine prior knowledge of spatiotemporal changes in consecutive effective frames. Spatial attention is used to fit the gradual trajectory of cloud motion, and temporal attention is used to adapt the transitional trend of motion rate. This corrects the spatial and temporal correlation of motion features in the sequence, achieving spatiotemporal nonlinear correction of motion features that aligns with cloud motion characteristics. The corrected motion features retain the basic motion trend extracted from local inter-frame motion features while accurately correcting the two types of inherent biases introduced by local linear interpolation of motion features. This achieves spatiotemporal natural adjustment of motion features, providing high-quality motion constraints for subsequent diffusion models.

[0084] Specifically, such as Figure 7 As shown, the internal processing flow of the nonlinear correction module is as follows:

[0085] S21. Perform dimensional reshaping on any adjacent frame motion features in the completed motion feature set, and then input the dimensional reshaping motion features as query, key and value vectors into the spatial neighborhood attention module for spatial neighborhood attention calculation to obtain spatial attention enhanced features. These features are then fused with the dimensional reshaping motion features through residual connections to obtain spatially corrected features.

[0086] It should be noted that in S21 of this invention, the forward and reverse motion features between any adjacent frames can be nonlinearly corrected in spatial dimension according to the above process. In this embodiment, the forward motion features in the forward motion feature set are used. Taking this example, the above spatial correction process will be explained in detail. First, in order to adapt to the input format of attention encoding and to explore the spatiotemporal correlation between motion features to enhance the representation of spatiotemporal nonlinear motion patterns, this embodiment first... Dimensional reshaping is represented as:

[0087]

[0088] in, That is Motion features after dimensional reshaping; This is a dimensional reshaping operation.

[0089] Based on this, this embodiment uses a spatial neighborhood attention module to focus on the motion coordination relationship between different regions in space, capture the motion correlation between spatial dimension pixels and intra-frame motion dependence, mine the intra-frame cloud motion distribution characteristics and correlation change rules, and correct the spatial isolated motion deviation in motion features.

[0090] like Figure 8 As shown, the spatial neighborhood attention module partitions the dimension-reshaped motion features spatially and flattens them into query vectors. Key vector AND value vector By calculating the similarity of motion features between any two partitions in space, a spatial attention weight matrix is ​​generated. This weight matrix encodes the motion relationships between pixels in the partitions within the space. Attention calculation strengthens the association between core pixels and edge pixels, allowing the motion trend of motion features to transition naturally from the core to the edge, rather than isolated end-to-end motion of each pixel. This embodiment... Figure 8 In the diagram, red arrows represent the motion trend of the current frame, and yellow arrows represent the motion trends of the other two valid frames. It can be seen that after spatial nonlinear correction, the continuity of the motion trends of these three frames can be better reflected.

[0091] The calculation process for spatial neighborhood attention is as follows:

[0092]

[0093] in, This is a spatial neighborhood attention calculation function used to capture regional cooperative relationships based on motion features; This is a spatial attention enhancement feature. To preserve the basic motion trend of the motion features and avoid orientation distortion caused by over-correction, this embodiment uses residual connections to fuse the spatial attention enhancement features with the dimension-reshaped motion features, resulting in spatially corrected features. This design retains the basic movement trend, strengthens regional coordination, and better fits the spatial correlation and change trend of cloud movement.

[0094] S22. Using the spatially corrected features as the query vector, and the motion features corresponding to the effective frames after dimensional reshaping as the key and value vectors, input them together into the temporal window attention module to perform temporal window attention calculation, obtain the temporal attention enhanced features, and fuse them with the spatially corrected features through residual connection to obtain the spatiotemporal corrected features. Then, perform dimensional calibration and nonlinear fine-tuning to obtain the corrected motion feature set.

[0095] It should be noted that, similarly, in S22 of this invention, the forward and reverse motion features between any adjacent frames can be nonlinearly corrected in the time dimension according to the above process. In this embodiment, the forward motion features in the forward motion feature set are still used. Taking this as an example, the above time correction process will be explained in detail. Figure 9 As shown, firstly, this embodiment uses spatially corrected features as the query vector. The motion features corresponding to the effective frames after dimensional reshaping are key vectors. AND value vector By calculating the temporal correlation between frames, a temporal attention weight matrix is ​​generated to adapt to the rate transition trend. For example, when the preceding frame transitions from low-speed motion to high-speed motion in the following frame, the motion features of the preceding frame within the window will receive a higher correlation weight with the low-speed effective frame, while the motion features of the following frame will be more closely associated with the high-speed effective frame, thus achieving a smooth rate decay.

[0096] The calculation process for attention within a time window is as follows:

[0097]

[0098] in, A time-window attention calculation function is used to correct motion rate deviations. Then, residual connections are used to fuse the temporal attention enhancement features with the spatial correction features obtained in the previous step, resulting in spatiotemporal correction features. This design not only has regional coordination within a single frame, but also presents a natural transition in frame rate.

[0099] Therefore, the aforementioned temporal window attention module can focus on the transition trend of inter-frame rates and correct motion rate deviations. By dividing the spatial correction features into local temporal windows according to time steps, the window covers the motion features after dimensional reshaping and the effective frame motion features at both ends of the missing interval, ensuring that the attention mechanism can capture the rate change patterns between effective frames.

[0100] It should be noted that in S22 of this invention, after attention encoding correction in both the spatiotemporal dimensions, this embodiment employs a lightweight feature decoding layer to complete the final optimization of the motion feature set. Specifically, this lightweight feature decoding layer includes a single-layer linear projection, a GELU activation function, and dimensional reshaping. It only performs dimensional calibration and nonlinear fine-tuning on the spatiotemporally corrected features, eliminating feature redundancy and fitting errors in the attention mechanism, and enhancing the representation capability of effective motion information. The positive motion features in the positive motion feature set are used as the basis for this optimization. For example, the calculation process of this lightweight feature decoding layer is as follows:

[0101]

[0102]

[0103] in, The intermediate motion features are output by the GELU activation function; and These are the weights and biases of a single-layer linear projection, respectively; Use the GELU activation function; To The corrected motion characteristics obtained after spatiotemporal correction.

[0104] Therefore, by sequentially performing the above process on all forward and reverse motion features of the entire sequence, this embodiment can obtain a modified motion feature set. The nonlinear correction module boasts a simple and efficient architecture, fusing and complementing spatial neighborhood attention and temporal window attention. The former addresses the problem of fragmented regional motion within a single frame, while the latter breaks through the assumption of local linear motion, achieving the core objectives of natural spatial transitions and smooth temporal rates. The final output corrected motion features better reflect the real motion patterns of clouds, providing high-quality motion constraints for subsequent diffusion models and effectively ensuring the spatiotemporal coherence and natural motion of the repaired frames.

[0105] After inter-frame motion feature extraction, bidirectional temporal completion of motion features, and spatiotemporal adaptive nonlinear correction, a corrected motion feature set covering the entire sequence has been obtained, which extracts and encodes the motion rhythm and trajectory patterns of remotely sensed clouds. For example... Figure 10As shown, this invention is based on the full-sequence diffusion generation method and adopts a guiding condition fusion strategy that injects modified motion features into the diffusion process. That is, motion features are integrated as guiding conditions into the noise prediction process of the diffusion model to achieve one-time high-quality repair of missing frames in the full sequence. This not only preserves the stability of the original sampling logic of the diffusion model, but also ensures the spatiotemporal coherence and appearance authenticity of the generated results.

[0106] It should be noted that in the inference phase of the diffusion model of this invention, the unknown frame is first initialized. Standard Gaussian noise And load the adapted and corrected motion feature set, then press to The denoising steps are executed sequentially and iteratively, with each diffusion time step based on the current noisy sequence. diffusion time step Denoising is achieved by correcting motion feature constraints, resulting in a denoised image. It follows a standard normal distribution. It is the identity matrix; This is the preset maximum diffusion time step.

[0107] In each diffusion time step, the update rule is as follows: Noisy sequence at each diffusion time step :

[0108]

[0109]

[0110] in, The initial input is a pure noise map; Introduce noise to the nth diffusion step; Accumulated noise for diffusion in step n; This is the noise prediction result for the corresponding unknown frame in the output of the diffusion model; The standard deviation of the diffused noise in step n; Random noise is introduced into the update process to balance the diversity and stability of the generated data. Accumulated noise is diffused in step n-1. When iterates to... At that time, the full sequence repair result is obtained. This enables seamless transitions between missing and valid frames.

[0111] It should also be noted that the diffusion model framework of this invention adopts a conditional diffusion model based on sequence random masking. The core forward denoising and backward denoising processes follow the standard DDPM paradigm, and motion constraint guidance is achieved only through conditional feature modulation of the noise prediction branch. In the training phase of the diffusion model, let the input remote sensing cloud image sequence be... First, a random masking operation is performed on the sequence to divide it into a set of valid frames. With missing frame set ,satisfy and .in, The length of the input remote sensing cloud image sequence; and These are taking the intersection and taking the union, respectively. This is an empty set. Then, during the forward noise addition process of the diffusion model, the effective frame set remains in its original state and is not noise-added; it serves as an appearance reference. Noise addition is only performed on the missing frame set. Stepwise noise addition, the noise addition process satisfies:

[0112]

[0113]

[0114]

[0115] in, The set of missing frames for adding noise in step n; This is the original set of missing frames before noise was added; It is independent and identically distributed Gaussian noise; A variable whose value is between 1 and n; This represents the noise at step k. The noise scheduling function is monotonically increasing. After adding noise to the missing frame set, this invention concatenates the noisy missing frame set with the valid frame set according to their indices to form a noisy sequence. And used as input for the diffusion model, This indicates that the input is concatenated by index, ensuring that noise-free valid frames and noisy missing frames are input together. The entire input sequence lays the foundation for the fusion of diffusion generation and motion constraints.

[0116] Furthermore, this invention also injects modified motion features as guiding conditions into the diffusion model, achieving deep integration and generation guidance of motion features and the diffusion model. Specifically, this embodiment first performs effective frame pair matching on the effective frame set to extract the modified motion feature set, which is then calibrated for encoding dimension and adapted for resolution using a feature adapter. In the feature adapter, this embodiment uses convolution operations to map the number of motion feature encoding channels corresponding to each frame's cloud image to be consistent with the channel number dimension of each frame's cloud image, and adjusts the spatial size upsampling to be consistent with the spatial dimension of each frame's cloud image, thereby obtaining the adapted modified motion feature set. In the stepwise denoising process of the diffusion model, the adapted modified motion feature set is scaled according to the scale and injected into each layer of the diffusion model. At each scale layer, the modified motion feature set is compared with the current diffusion model. Layer output features Perform residual join operation:

[0117]

[0118] in, For the diffusion model The output features of the layer The layered channel and spatial adaptation function ensures that the diffusion model can perceive motion feature constraints throughout the entire diffusion generation process, so that the generated repair results not only match the real appearance of the cloud body, but also follow the preset motion law.

[0119] loss function Similarly, the calculation is performed only on the set of missing frames to avoid the cleanliness information of valid frames interfering with the loss, ensuring that the diffusion model focuses on the denoising generation of missing frames, defined as:

[0120]

[0121] in, To achieve the desired outcome; This is a clean missing frame; Original Gaussian noise; This represents the noise prediction result for the corresponding set of missing frames in the model output; This is the square of the L2 norm. This loss function enables the model to accurately recover the true distribution of missing frames under the dual constraints of motion constraints and appearance references.

[0122] The above process, by injecting modified motion features as guiding conditions into the diffusion model, achieves deep integration of motion constraints while retaining the original sampling logic of the diffusion model. The one-time generation strategy of the entire sequence avoids the cumulative error caused by inter-frame iterative generation. The random masking mechanism enables it to adaptively handle any missing patterns at any position and in any number. The continuous constraints of motion features ensure the motion coherence of the generated frames. The effective frames, as appearance references, effectively improve the authenticity of the repair results, providing an efficient and reliable solution for repairing missing patterns in remote sensing cloud image sequences.

[0123] The present invention will now demonstrate the application effect of the image data missing frame completion method based on the conditional diffusion model described in the above embodiments on a specific dataset through a specific example, so as to facilitate understanding of the essence of the present invention.

[0124] Example

[0125] The image data missing frame completion method based on the conditional diffusion model used in this embodiment is implemented as described above and will not be repeated here. The following section demonstrates part of the implementation process and results:

[0126] This embodiment uses data from the Advanced Himawari Imager (AHI) of the Himawari-8 / 9 geostationary meteorological satellites of a certain regional meteorological bureau as the data source. The target region is East Asia and the Northwest Pacific, with longitudes ranging from 110°E to 175°E and latitudes from 35°S to 35°N. All raw satellite data were uniformly reprojected to an equal latitude and longitude grid (EPSG:4326), with a spatial resolution of approximately 0.05°.

[0127] To construct data suitable for training the deep learning model, the target region was spatially divided into 256×256 pixel blocks, uniformly divided into 5×5 grids along the latitude and longitude directions, generating a total of 25 sampling areas. Each sampling area covers a geographical range of approximately 12.8°×12.8° (approximately 1300km×1300km). This embodiment uses the visible light band of AHI as the primary input channel, with the band values ​​of the original L1-level data as the main reference, combined with cloud type (CLT) detection data from L2-level cloud products as an auxiliary data source, to perform cloud masking processing. This removes cloudless or low-cloud areas, ensuring that the model learning focuses on the evolution of real cloud systems.

[0128] This embodiment uses a 30-minute time interval for downsampling. Each training sequence contains 12 frames of images, corresponding to a 6-hour time window. All methods are performed under a standard experimental setting of 12-frame sequences and 4 randomly missing frames. All data are z-score normalized. The dataset spans from January 1 to July 31, 2025, and is randomly divided into training and test sets in an 8:2 ratio.

[0129] This embodiment uses mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and learned perceptual image patch similarity (LPIPS) as evaluation metrics to assess the invention.

[0130] Mean squared error is used to calculate the error between the pixels in the predicted image (i.e., the repaired result corresponding to the missing frame) and the pixels in the real image corresponding to the missing frame. Its mathematical formula is as follows:

[0131]

[0132] in, Number of images; This represents the total number of pixels in the image. These are the image's height, width, and number of channels, respectively. For the first The first real image in One channel, Pixel value at the location; For the first Zhang predicted image in the first One channel, The pixel value at the location.

[0133] Peak signal-to-noise ratio (PSNR) is calculated based on mean squared error and is used to measure the overall reconstruction quality of the predicted image. It is defined as follows:

[0134]

[0135] in, This represents the maximum pixel value predicted for the image.

[0136] The structural similarity index comprehensively considers the brightness, contrast, and structural information of an image, and is defined as follows:

[0137]

[0138] in, Indicates the structural similarity between the real image and the predicted image; These are the average brightness values ​​of the real image and the predicted image, respectively. These represent the variances of the real image and the predicted image, respectively. and It is the stability constant; Let be the covariance between the real image and the predicted image.

[0139] Learning to perceive image patch similarity better reflects the subjective perceptual quality of human eyes regarding details, textures, and boundaries; it is defined as:

[0140]

[0141] in, The learned perceptual image patch similarity between the real image and the predicted image; The first The height and width of the layer feature map; For network layer index; For the first Learnable channel weights for layers are used to balance the importance of features at different levels; This indicates a channel-by-channel weighted average. This indicates that the network extracts the first [image] from the real image. Layer features; Representation of features exist Pixel value at the location; This indicates that the network extracts the first [item] from the predicted image. Layer features; Representation of features exist Pixel value at the location; This represents the square of the L2 norm.

[0142] To verify the effectiveness of the method of this invention, five methods were selected for comparative implementation. DAIN is a depth-information-based optical flow frame interpolation method that captures pixel-level motion correlations between frames through optical flow estimation, combines depth information to handle motion occlusion problems, and perceives hierarchical features from adjacent pixels. CAIN is a deep neural network method based on an attention mechanism, trained through an end-to-end deep neural network, combining an attention mechanism to replace optical flow calculation, and extracting pixel-level frame synthesis motion information. EDSC is a novel non-optical flow kernel-based method called Enhanced Deformable Separable Convolution. By learning adaptive kernels, offsets, masks, and biases, the network can acquire information from non-local neighborhoods, thereby estimating multiple intermediate frames. RaMViD is a video completion method based on a masked diffusion model. It simulates random missing frames through masks, utilizes 3D-UNet to capture the spatiotemporal correlation of frame sequences, and adapts to the dynamic changes of video sequences. RIFE is a real-time intermediate flow estimation frame interpolation method that designs a lightweight IFNet network to directly estimate the intermediate flow state between intermediate frames and adjacent frames, improving interpolation speed and accuracy.

[0143] Table 1 shows the performance comparison of different methods. Experiments show that the present invention outperforms the best-performing control model on the dataset in all evaluation metrics. Compared with all the comparison methods, the core advantage of the present invention lies in achieving independent modeling of motion patterns and deep integration with diffusion generation, completely breaking through the inherent limitations of traditional methods. Compared with optical flow methods such as DAIN and RIFE, the present invention gets rid of the dependence on adjacent frames and the linear motion assumption. By modeling the temporal changes of motion features, the motion trajectory of the repaired frame is made to better fit the evolution law of the cloud, thus significantly improving pixel accuracy. Compared with deep network methods such as CAIN and EDSC, the present invention avoids the blurriness and incongruity of the repaired frame through spatiotemporal adaptive nonlinear correction and diffusion model generation, significantly improving structural similarity and visual perception quality. Compared with RaMViD, which is also a diffusion model, the present invention does not only rely on 3D convolution to capture spatiotemporal correlation, but also uses deep constraints and guidance of motion features to allow diffusion generation to perceive the motion law of the cloud throughout the process, achieving a significant improvement in structural clarity and maintaining an advantage in pixel accuracy. Experiments have shown that this invention achieves the optimization goals of pixel fidelity and temporal coherence, providing an advanced and effective method for repairing missing data in remote sensing cloud image sequences.

[0144] Table 1. Comparison of experimental results using different methods

[0145] Figure 11This embodiment demonstrates the missing data in the real remote sensing cloud image sequence used. A visualization comparison of this invention with other methods is shown on this real remote sensing cloud image sequence. Figure 12 As shown, the cloud map repaired by the method of this invention visually matches the real cloud map quite well. Not only are the outlines and texture details of the clouds accurately restored, but the evolution of the cloud system is also more natural and smooth. At all missing frame locations, including discrete and continuous missing frames, the generated repaired cloud map better conforms to the motion patterns of the real clouds, with clear cloud edges, distinct internal textures, and smooth transitions between frames, essentially eliminating the blurring, distortion, and artifact problems common in other methods. The visualization results are basically consistent with the conclusions of the quantitative indicators, verifying that this invention effectively solves the core pain points of traditional methods in nonlinear motion modeling of clouds and temporal continuity, providing a method for repairing missing remote sensing cloud map sequences that combines pixel accuracy and structural quality.

[0146] It is understood that the image data missing frame completion method based on the conditional diffusion model described in the above embodiments can essentially be implemented by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer program product corresponding to the image data missing frame completion method based on the conditional diffusion model provided in the above embodiments. This product includes a computer program / instructions that, when executed by a processor, can implement the image data missing frame completion method based on the conditional diffusion model as described in the above embodiments.

[0147] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the image data missing frame completion method based on the conditional diffusion model provided in the above embodiments, such as... Figure 13 As shown, it includes a memory and a processor;

[0148] The memory is used to store computer programs;

[0149] The processor is configured to implement, when executing the computer program, a method for completing missing frames of image data based on a conditional diffusion model as described in the above embodiments.

[0150] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0151] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the image data missing frame completion method based on the conditional diffusion model provided in the above embodiment. The storage medium stores a computer program, which, when executed by a processor, can implement the image data missing frame completion method based on the conditional diffusion model in the above embodiment.

[0152] It is understood that the aforementioned storage media may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage media may also be various media capable of storing program code, such as USB flash drives, external hard drives, magnetic disks, or optical discs.

[0153] It is understood that the processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0154] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A method for completing missing frames in image data based on a conditional diffusion model, characterized in that, Includes the following steps: The remote sensing cloud image sequence with missing frames is input into the trained inpainting model. First, the temporal inpainting module performs bidirectional temporal modeling and linear interpolation to fill in the valid frames before and after the missing frames in the sequence, obtaining a complete motion feature set covering the entire sequence. Then, the nonlinear correction module performs spatiotemporal nonlinear correction to obtain a corrected motion feature set, which is used as the guiding condition for the diffusion model. Gaussian noise is used to replace the missing frames and together with the valid frames, it forms a noisy sequence input into the diffusion model. After multiple denoising steps, the inpainting results of each missing frame are obtained. The internal processing flow of the nonlinear correction module is as follows: S21. Perform dimensional reshaping on any adjacent frame motion features in the complete motion feature set, and then input the dimensional reshaping motion features as query, key and value vectors into the spatial neighborhood attention module for spatial neighborhood attention calculation to obtain spatial attention enhancement features. These features are then fused with the dimensional reshaping motion features through residual connections to obtain spatial correction features. S22. Using the spatially corrected features as the query vector and the motion features after dimensional reshaping corresponding to the effective frames as the key and value vectors, input them together into the temporal window attention module to perform temporal window attention calculation, obtain the temporal attention enhanced features, and fuse them with the spatially corrected features through residual connection to obtain the spatiotemporal corrected features. Then, perform dimensional calibration and nonlinear fine-tuning to obtain the corrected motion feature set.

2. The image data missing frame completion method based on the conditional diffusion model as described in claim 1, characterized in that, The specific process of the temporal completion module is as follows: two valid frames adjacent to a missing frame are used to form a valid frame pair. The feature extraction module extracts features along the forward and reverse temporal sequences respectively, forming a set of forward motion features and a set of reverse motion features covering the remote sensing cloud image sequence, which together constitute the complete motion feature set.

3. The image data missing frame completion method based on the conditional diffusion model as described in claim 2, characterized in that, The internal processing flow of the feature extraction module is as follows: S11. High-dimensional appearance features of effective frame pairs are extracted by a multi-scale convolutional architecture and formed into feature pairs. Pixel similarity and cross-frame motion trends in the feature pairs are jointly modeled by inter-frame attention. Two attention maps, one in forward time series and one in reverse time series, are obtained and neighborhood position information is weighted and aggregated. Then, the coordinate position corresponding to each feature vector of one high-dimensional appearance feature in the same feature pair is estimated when it is mapped to another high-dimensional appearance feature. S12. By calculating the estimated deviation of the coordinate position pixel by pixel, the displacement vectors of the forward and reverse time series are obtained and encoded by a linear layer to obtain the forward and reverse motion features of the effective frame pairs. By scaling by time step, the forward and reverse motion features between each frame in the remote sensing cloud image sequence are obtained, which correspond to the forward motion feature set and the reverse motion feature set and form the complete motion feature set.

4. The image data missing frame completion method based on the conditional diffusion model as described in claim 3, characterized in that, In the multi-scale convolutional architecture, the remote sensing cloud image sequence is first downsampled to a high-dimensional appearance feature space through multi-scale convolution to obtain the appearance features of each effective frame pair at each scale. Subsequently, appearance features are extracted and fused across scales through dilated convolution, thereby aligning the appearance features at each scale to a high-dimensional appearance feature with uniform dimensions.

5. The image data missing frame completion method based on the conditional diffusion model as described in claim 3, characterized in that, In the joint modeling of inter-frame attention, during forward temporal feature extraction, the preceding frame in the effective frame pair is used as the query vector, and the following frame is used as the key vector and value vector. During reverse temporal feature extraction, the following frame is used as the query vector, and the preceding frame is used as the key vector and value vector.

6. The image data missing frame completion method based on the conditional diffusion model as described in claim 3, characterized in that, The specific process of obtaining motion features between frames in a remote sensing cloud image sequence by time step scaling is as follows: For any missing frame between the preceding and following frames, the time interval between the preceding and following frames is calculated. The reciprocal of this time interval is used as the time step ratio. The forward motion features from the preceding frame to the following frame and the reverse motion features from the following frame to the preceding frame are scaled using the time step ratio. This yields the motion features from the missing frame to the next frame and the motion features from the next frame to the missing frame. Thus, the set of forward motion features and the set of reverse motion features between frames in the remote sensing cloud image sequence are obtained.

7. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it can implement the image data missing frame completion method based on the conditional diffusion model as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the image data missing frame completion method based on the conditional diffusion model as described in any one of claims 1 to 6.

9. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the image data missing frame completion method based on the conditional diffusion model as described in any one of claims 1 to 6.

Images