Video restoration area positioning method and related equipment

Abstract

The invention provides a video restoration area positioning method and related equipment, and relates to the technical field of video restoration, and the method comprises the steps: extracting a high-frequency residual feature from noise stream data for training, and carrying out the early fusion of the high-frequency residual feature and an original pixel feature in RGB stream data for training, so as to capture the correlation feature of content and noise; a loss function is calculated according to a set supervision mode and a dense prediction mask sequence, and network parameters of the double-flow 3D Swindow-UNet fused with a space-time attention mechanism are updated according to a loss function value, so that optical flow dependence is abandoned, and the problem that an existing video repair detection method is poor in efficiency when facing a modern deep learning repair algorithm is effectively solved. The problems that low-contrast area detection fails, small target counterfeiting traces are lost, the reasoning efficiency is low due to excessive dependence on optical flow characteristics, and full supervision training has too high dependence on massive frame-by-frame labeling data are solved, and efficient and accurate positioning under sparse time sequence supervision is achieved.

Description

Technical Field

[0001] This invention relates to the field of video restoration technology, and in particular to a method for locating video restoration areas and related equipment. Background Technology

[0002] Video inpainting technology aims to fill in missing regions in video sequences or remove specific objects while maintaining spatiotemporal consistency. With the advent of modern deep learning inpainting algorithms based on optical guides or Transformer architectures, the inpainting results have reached a level that is difficult to detect with the naked eye. However, if this technology is maliciously used to forge surveillance videos or delete critical evidence, it will pose serious security risks.

[0003] Existing video restoration and detection methods suffer from the following shortcomings: First, modern restoration algorithms generate forged regions with extremely high visual consistency in the RGB domain and smooth out optical flow anomalies, rendering traditional methods that rely on explicit optical flow calculations ineffective and computationally intensive. Second, existing dual-stream networks often employ late fusion, failing to effectively utilize the correlation features between content and noise, resulting in detection failure in low-contrast scenes (such as dark forged objects against a dark background). Furthermore, existing methods often adopt a "multi-frame input predicts a single frame" approach, ignoring the temporal dimension of the output, making it impossible to utilize temporal consistency for self-supervised learning, wasting the self-supervised signals of the temporal dimension, and requiring expensive frame-by-frame fully supervised annotation for training, thus limiting their practical application. Summary of the Invention

[0004] This invention provides a video restoration area localization method and related equipment, the purpose of which is to achieve efficient and accurate localization under sparse temporal supervision.

[0005] To achieve the above objectives, the present invention provides a method for locating a video restoration region, comprising: Step 1: Extract the training video frame sequence from the video segment to be detected, and construct the training RGB stream data and training noise stream data based on the training video frame sequence; Step 2: Extract high-frequency residual features from the training noise stream data, and perform early fusion of the high-frequency residual features with the original pixel features in the training RGB stream data to obtain dual-modal fusion features; Step 3: Input the dual-modal fusion features into a dual-stream 3D Swin-UNet with a spatiotemporal attention mechanism for processing to obtain a dense prediction mask sequence; Step 4: Calculate the loss function based on the set supervision mode and dense prediction mask sequence, and update the network parameters of the dual-stream 3D Swin-UNet with spatiotemporal attention mechanism based on the loss function value to obtain the video restoration area localization model. Step 5: Construct an RGB stream and a noise stream using the video frame sequence extracted from the video segment to be detected. Extract high-frequency residual features from the noise stream and fuse them with the RGB stream. Then input the fused features into the video restoration region localization model for localization to obtain the video restoration region localization result of the video segment to be detected. The number of frames in the video restoration region localization result is consistent with the length of the video frame sequence extracted from the video segment to be detected.

[0006] Furthermore, the training noise stream data is used to extract features through constrained learnable convolutional layers to obtain high-frequency residual features.

[0007] Furthermore, the constraints on learnable convolutional layers include: The weight of the central element is fixed at -1; The sum of the weights of all elements is 0.

[0008] Furthermore, the dual-stream 3D Swin-UNet, which incorporates a spatiotemporal attention mechanism, includes: A spatiotemporal encoder for extracting multi-scale spatiotemporal features, a decoder with attention mechanism, and a dense temporal prediction head; The spatiotemporal encoder and decoder have an integrated spatial channel squeezing-excited attention module for recalibrating feature maps in the skip connection; The input of the dense timing prediction head is connected to the output of the decoder.

[0009] Furthermore, the supervision models include full-time supervision model and sparse-time supervision model.

[0010] Furthermore, when the supervision mode is set to full temporal supervision mode, pixel-level loss calculation is performed on the prediction results of each frame in the dense prediction mask sequence; When the supervision mode is set to sparse temporal supervision mode, the ground truth mask of the prediction result of the center frame in the dense prediction mask sequence is obtained, the pixel-level loss between the prediction result of the center frame and the ground truth mask is calculated, the temporal consistency loss between the prediction results of adjacent frames in the dense prediction mask sequence is calculated, and the L1 distance between the prediction masks of adjacent frames is calculated using the temporal consistency loss.

[0011] Furthermore, pixel-level losses include class balance loss, Dice loss, edge consistency loss, temporal consistency loss, and contrast loss.

[0012] The present invention also provides a video restoration region positioning device, which applies a video restoration region positioning method. The video restoration region positioning device includes: The module is used to extract training video frame sequences from the video segments to be detected, and to construct training RGB stream data and training noise stream data based on the training video frame sequences. The fusion module is used to extract high-frequency residual features from the training noise stream data and perform early fusion of the high-frequency residual features with the original pixel features in the training RGB stream data to obtain dual-modal fusion features; The processing module is used to process the dual-modal fusion features input into a two-stream 3D Swin-UNet with a spatiotemporal attention mechanism to obtain a dense prediction mask sequence. The update module is used to calculate the loss function based on the set supervision mode and dense prediction mask sequence, and update the network parameters of the dual-stream 3D Swin-UNet with spatiotemporal attention mechanism based on the loss function value to obtain the video restoration region localization model. The localization module is used to construct an RGB stream and a noise stream using the video frame sequence extracted from the video segment to be detected. High-frequency residual features are extracted from the noise stream and fused with the RGB stream before being input into the video restoration region localization model for localization. This results in the localization result of the video restoration region of the video segment to be detected. The number of frames in the localization result of the video restoration region is consistent with the length of the video frame sequence extracted from the video segment to be detected.

[0013] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a video restoration area positioning method.

[0014] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a video restoration area localization method.

[0015] The above-described solution of the present invention has the following beneficial effects: Compared with existing technologies, this invention first extracts high-frequency residual features from the training noise stream data, and then performs early fusion of the high-frequency residual features with the original pixel features in the training RGB stream data to capture the correlation features between content and noise. Based on the set supervision mode and dense prediction mask sequence, a loss function is calculated, and the network parameters of the dual-stream 3D Swin-UNet with a spatiotemporal attention mechanism are updated according to the loss function value. This eliminates the dependence on optical flow and effectively solves the problems of existing video restoration and detection methods when facing modern deep learning restoration algorithms, such as failure to detect low-contrast areas, loss of small target forgery traces, low inference efficiency due to excessive reliance on optical flow features, and excessive dependence on massive frame-by-frame labeled data in fully supervised training. This achieves efficient and accurate localization under sparse temporal supervision.

[0016] Other beneficial effects of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the dual-stream 3D Swin-UNet and the spatial channel extrusion excitation attention module in an embodiment of the present invention; Figure 3 The figure shows the experimental results of an embodiment of the present invention; Figure 4 This is a schematic diagram of the video restoration area positioning device in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the terminal device in an embodiment of the present invention. Detailed Implementation

[0018] To make the technical problems, solutions, and advantages of this invention clearer, a detailed description will be provided below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] In the description of this invention, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0020] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a locking connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0021] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0022] This invention addresses existing problems by providing a video restoration area localization method and related equipment.

[0023] like Figure 1 As shown, an embodiment of the present invention provides a method for locating a video restoration area, including: Step 1: Extract the training video frame sequence from the video segment to be detected, and construct the training RGB stream data and training noise stream data based on the training video frame sequence; Step 2: Extract high-frequency residual features from the training noise stream data, and perform early fusion of the high-frequency residual features with the original pixel features in the training RGB stream data to obtain dual-modal fusion features; Step 3: Input the dual-modal fusion features into a dual-stream 3D Swin-UNet with a spatiotemporal attention mechanism for processing to obtain a dense prediction mask sequence; Step 4: Calculate the loss function based on the set supervision mode and dense prediction mask sequence, and update the network parameters of the dual-stream 3D Swin-UNet with spatiotemporal attention mechanism based on the loss function value to obtain the video restoration area localization model. Step 5: Construct an RGB stream and a noise stream using the video frame sequence extracted from the video segment to be detected. Extract high-frequency residual features from the noise stream and fuse them with the RGB stream. Then input the fused features into the video restoration region localization model for localization to obtain the video restoration region localization result of the video segment to be detected. The number of frames in the video restoration region localization result is consistent with the length of the video frame sequence extracted from the video segment to be detected.

[0024] In this embodiment of the invention, step 1 specifically includes: Obtain the video segment to be detected; Extract a video frame sequence of length T from the video segment to be detected as the training video frame sequence for training the model. Adjust the video frames of different resolutions in the training video frame sequence to a fixed size, such as 384×384, and normalize the parameter to the mean. During the training phase, the training video frame sequences are preprocessed with random Gaussian blur, JPEG compression noise injection, and color dithering enhancement to simulate different video qualities and destroy the artifact fingerprints of specific restoration algorithms, thereby improving the robustness of the model. The preprocessed training video frame sequence is divided into RGB stream data and noise stream data.

[0025] In this embodiment of the invention, when performing random Gaussian blurring, the size of the Gaussian blur kernel is set to 5×5, and the sigma range is 0.1~2.0; when performing JPEG compression noise injection, the JPEG compression quality factor is set to 30. 100, Color jitter enhancement, adjusts brightness and contrast.

[0026] In this embodiment of the invention, RGB stream data directly transmits the original video frame, preserving semantic content and photometric information.

[0027] Specifically, the training noise stream data is used to extract features through constrained learnable convolutional layers to obtain high-frequency residual features that are not affected by the image content.

[0028] Specifically, the constraints on learnable convolutional layers include: The weight of the central element is fixed at -1; The sum of the weights of all elements is 0.

[0029] In this embodiment of the invention, the learnable convolutional layer includes one or more constrained learnable 3D convolutional kernels. The kernel size is set to 5×5, and the number of input channels is consistent with the RGB stream data. Before each forward propagation, the kernel weights of the learnable convolutional layer are normalized using the following formula: ; This hard constraint mechanism forces the model to ignore the semantic content of the image (such as color and shape) and only respond to the high-frequency residual relationship between local pixels by constraining the quantitative relationship between the center weight of the convolution kernel and the surrounding weights. This ensures that the filter has perfect high-pass characteristics, suppresses the low-frequency content of the image (such as color and shape), and only responds to the high-frequency residual relationship between local pixels, thereby extracting artifact features in the spatiotemporal dimension.

[0030] In this embodiment of the invention, high-frequency residual features are concatenated with RGB stream data in the channel dimension to form a bimodal fusion feature with dimensions B×6×T×H×W, where B represents the batch number, 6 represents the number of channels, T represents the time dimension, H represents the height, and W represents the width. This early fusion enables the model to establish a correlation between semantic content and noise residuals.

[0031] It should be noted that, in addition to channel concatenation, early fusion of dual-stream features in this embodiment of the invention can also be achieved through element-wise summation, adaptive weighted fusion, etc. This embodiment of the invention does not limit the specific method of early fusion of dual-stream features.

[0032] like Figure 2 As shown, specifically, the dual-stream 3D Swin-UNet that incorporates a spatiotemporal attention mechanism includes: A spatiotemporal encoder for extracting multi-scale spatiotemporal features, a decoder with attention mechanism, and a dense temporal prediction head; The spatiotemporal encoder and decoder have an integrated spatial channel squeezing-excited attention module for recalibrating feature maps in the skip connection; The input of the dense timing prediction head is connected to the output of the decoder.

[0033] Specifically, the spatiotemporal encoder is implemented based on the Video Swin Transformer, which divides video frames into small blocks and then computes self-attention within a 3D moving window through multi-layered stacked sliding window blocks. This captures both intra-frame spatial correlations and inter-frame temporal dependencies. After four levels of downsampling, it extracts multi-scale spatiotemporal features ranging from high-resolution shallow features to low-resolution deep semantic features. The decoder is implemented based on a 3D U-Net structure, including upsampling units composed of 3D transposed convolutions to progressively enlarge the feature map size, gradually restoring the features extracted by the encoder to the original resolution and generating a repaired region mask. The spatial channel squeezed excitation attention module integrated in the skip connections of the spatiotemporal encoder and decoder contains two branches: spatial attention and channel attention. It recalibrates the feature map through parallel spatial and channel attention branches, suppresses background texture noise, and then inputs it into a dense temporal prediction head to generate a dense prediction mask sequence of one of the lengths of the training video frame sequence.

[0034] It should be noted that the Video SwinTransformer and the spatial channel squeezing excitation attention module that make up the dual-stream 3D Swin-UNet in the embodiments of the present invention are modules in the prior art. The improvement of the embodiments of the present invention lies in combining the Video Swin Transformer and the spatial channel squeezing excitation attention module with the early fusion of dual-stream features, the dense temporal prediction head and the composite loss function to form a collaborative localization mechanism, rather than the structural improvement of the module itself.

[0035] To adapt to application scenarios with different data annotation costs, the dual-stream 3D Swin-UNet proposed in this embodiment supports two supervision modes, which can be flexibly switched by configuring the calculation logic of the loss function without changing the network structure.

[0036] Specifically, the supervision models include full-time-series supervision model and sparse-time-series supervision model.

[0037] In this embodiment of the invention, the full-temporal supervision mode is suitable for high-quality datasets with finely labeled frames, which can maximize the model's ability to capture difficult samples (such as fast-moving objects) and achieve the highest detection accuracy; the sparse temporal supervision mode is suitable for situations where data labeling costs are limited and only sparse keyframe labels are available, using only... Even with 20% (for example) labeled data, effective feature learning can still be achieved through a self-supervised mechanism, significantly reducing the data labeling threshold.

[0038] Specifically, when the supervision mode is set to full temporal supervision mode, pixel-level loss calculation is performed on the prediction results of each frame in the dense prediction mask sequence; When the supervision mode is set to sparse temporal supervision mode, the ground truth mask of the prediction result of the center frame in the dense prediction mask sequence is obtained, the pixel-level loss between the prediction result of the center frame and the ground truth mask is calculated, and the adjacent frames in the dense prediction mask sequence (such as...) are calculated. and , and The temporal consistency loss between prediction results is calculated, and the L1 distance between prediction masks of adjacent frames is calculated using the temporal consistency loss.

[0039] In the sparse temporal supervision mode of this invention, the center frame obtains a clear supervision signal through pixel-level loss, while the temporal consistency loss forces the prediction results of adjacent frames to maintain spatiotemporal coherence with the center frame and adjacent frames. If the center frame locates a forged region, the prediction mask of the adjacent frame must be consistent with the position and shape of the forged region in the center frame due to temporal correlation. Otherwise, the temporal loss will increase. Thus, the supervision information of the center frame is indirectly transmitted to the unlabeled adjacent frames through backpropagation, realizing full-sequence supervised training.

[0040] Specifically, pixel-level losses include class balance loss, Dice loss, edge consistency loss, temporal consistency loss, and contrast loss, with the following specific expressions: ; in, Indicates pixel-level loss. This represents the weights of each loss function. , Represents class-balanced loss, This indicates Dice's loss. This represents the edge consistency loss. Indicates the time-series consistency loss. This indicates a comparative loss.

[0041] Specifically, the expression for calculating the temporal consistency loss between prediction results of adjacent frames in a dense prediction mask sequence is as follows: ; in, Indicates the first frame, Indicates the first frame.

[0042] The embodiments of the present invention verify the provided method through specific experiments, as follows: Experimental hardware environment: Nvidia RTX 3090 graphics card with 24GB of VRAM Software framework: PyTorch The hyperparameters include: a learning rate of 1e-4, a sample size of 5, and a length of 5. Datasets: The datasets used are the publicly available video restoration datasets DAVIS2016 and MOSE. Forgery methods: Context Prior Network (CPNet, abbreviated as CP), Video Inpainting (VI) method, Fuse-Former (End-to-End Framework for Flow-Guided Video Inpainting, E2FGVI), and Open Policy Network (OPN, abbreviated as OP). Comparison method: Comparison with existing state-of-the-art methods (such as detection models corresponding to VFL and UVL). Evaluation metrics: Intersection over Union (IoU) and F1 score were used. Intersection over Union (IoU) is a key metric for measuring the degree of overlap between the predicted and ground truth regions. In video inpainting detection, it can be used to evaluate the accuracy of the localization of the inpainted region. In video inpainting tasks, IoU is usually extended to 3D spatiotemporal IoU, and its calculation formula is as follows: ; in, For the predicted repair area cube (containing spatial and temporal dimensions), To realistically repair the region cube, A higher value indicates a better overlap between the predicted and actual regions. In video restoration detection, It is usually used in conjunction with a preset threshold (such as 0.5) to determine whether the predicted region has effectively identified the repair region.

[0043] Pixel-level F1-Score: The pixel-level F1-Score is a commonly used metric in pixel-level classification tasks, used to measure the balance between precision and recall in localization of repaired regions. Its calculation formula is as follows: ; in, , TP represents correctly identified repair pixels, FP represents non-repair pixels that were misidentified as repair pixels, and FN represents missed repair pixels. The Pixel-level F1-Score can comprehensively evaluate the performance of repair area localization, and is especially sensitive when dealing with small-scale repair areas.

[0044] The test results are shown in Table 1 below: Table 1 Experimental Results

[0045] Table 1 above shows the performance metrics (cross-over ratio, F1 score) of the forged region localization for each forgery method (forgery method CP, OP, VI) in the DAVIS2016 dataset according to the embodiments of the present invention. Among them, the values ​​of CP, OP, and VI are shown in Table 1. The labeled methods are the trained fake methods, and the unlabeled methods are the untrained fake methods; for example, in the first group, this invention uses the dataset of fake methods OP and VI for training, therefore OP is... VI is The datasets of the forged methods CP, OP, and VI were used for testing. The test data obtained are shown in Table 1 above. Under full supervision, the various indicators of the present invention embodiment are improved compared with the comparison method ViLocal. At the same time, under sparse supervision, the supervision cost required by the present invention embodiment is reduced to 1 / T of the original cost (T is 5 in this experiment), while the localization performance index only decreases by about 20%.

[0046] Figure 3 This paper illustrates the IoU index and relative decrease value of different model combinations (E2FGVI→CPNet, FuseFormer→CPNet, OPM→CPNet, E2FGVI→RiseFormer, OPN→FuseFormer, E2FGVI→OPN) under Full Supervision and Sparse Supervision conditions in the DAVIS2016 dataset according to embodiments of the present invention.

[0047] from Figure 3 As can be seen, under full supervision, the IoU index of each model combination is at a high level; under sparse supervision, although the IoU decreases, the relative decrease is controlled at about 20%, indicating that the present invention can maintain good performance in spoofing region localization while significantly reducing supervision costs.

[0048] Compared with existing technologies, this invention first extracts high-frequency residual features from the training noise stream data, and then performs early fusion of the high-frequency residual features with the original pixel features in the training RGB stream data to capture the correlation features between content and noise. Based on the set supervision mode and dense prediction mask sequence, a loss function is calculated, and the network parameters of the dual-stream 3D Swin-UNet with a spatiotemporal attention mechanism are updated according to the loss function value. This eliminates the dependence on optical flow and effectively solves the problems of existing video restoration and detection methods when facing modern deep learning restoration algorithms, such as failure to detect low-contrast areas, loss of small target forgery traces, low inference efficiency due to excessive reliance on optical flow features, and excessive dependence on massive frame-by-frame labeled data in fully supervised training. This achieves efficient and accurate localization under sparse temporal supervision.

[0049] like Figure 4 As shown, this embodiment of the invention also provides a video restoration area positioning device 100, which applies a video restoration area positioning method. The video restoration area positioning device 100 includes: The construction module 101 is used to extract training video frame sequences from the video segments to be detected, and to construct training RGB stream data and training noise stream data based on the training video frame sequences; The fusion module 102 is used to extract high-frequency residual features from the training noise stream data and perform early fusion of the high-frequency residual features with the original pixel features in the training RGB stream data to obtain dual-modal fusion features; Processing module 103 is used to process the dual-modal fusion features input into a dual-stream 3DSwin-UNet with a spatiotemporal attention mechanism to obtain a dense prediction mask sequence. The update module 104 is used to calculate the loss function based on the set supervision mode and dense prediction mask sequence, and update the network parameters of the dual-stream 3D Swin-UNet with spatiotemporal attention mechanism based on the loss function value to obtain the video restoration area localization model. The localization module 105 is used to construct an RGB stream and a noise stream using the video frame sequence extracted from the video segment to be detected, and to extract high-frequency residual features from the noise stream and fuse them with the RGB stream before inputting them into the video restoration region localization model for localization, so as to obtain the video restoration region localization result of the video segment to be detected. The number of frames in the video restoration region localization result is consistent with the length of the video frame sequence extracted from the video segment to be detected.

[0050] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0051] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0052] This invention also provides a terminal device, such as... Figure 5 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 5 The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100. When the processor D100 executes the computer program D102, it implements the video restoration region localization method described above.

[0053] The terminal device D10 can be a desktop computer, laptop, handheld computer, server, server cluster, or cloud server, etc. This terminal device may include, but is not limited to, a processor D100 and a memory D101. Those skilled in the art will understand that... Figure 5 This is merely an example of terminal device D10 and does not constitute a limitation on terminal device D10. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0054] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0055] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.

[0056] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0057] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0058] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a video restoration area localization method.

[0059] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a building device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0060] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for locating a video restoration region, characterized in that, include: Step 1: Extract training video frame sequences from the video segment to be detected, and construct training RGB stream data and training noise stream data based on the training video frame sequences; Step 2: Extract high-frequency residual features from the training noise stream data, and perform early fusion of the high-frequency residual features with the original pixel features in the training RGB stream data to obtain dual-modal fusion features; Step 3: Input the dual-modal fusion features into a dual-stream 3D Swin-UNet with a spatiotemporal attention mechanism for processing to obtain a dense prediction mask sequence; Step 4: Calculate the loss function based on the set supervision mode and the dense prediction mask sequence, and update the network parameters of the dual-stream 3D Swin-UNet with spatiotemporal attention mechanism based on the loss function value to obtain the video restoration region localization model. Step 5: Construct an RGB stream and a noise stream using the video frame sequence extracted from the video segment to be detected. Extract high-frequency residual features from the noise stream and fuse them with the RGB stream. Then input the fused features into the video restoration region localization model for localization to obtain the video restoration region localization result of the video segment to be detected. The number of frames in the video restoration region localization result is consistent with the length of the video frame sequence extracted from the video segment to be detected.

2. The video restoration region positioning method according to claim 1, characterized in that, The training noise stream data is used to extract features through constrained learnable convolutional layers to obtain high-frequency residual features.

3. The video restoration region positioning method according to claim 2, characterized in that, The constraints on the learnable convolutional layer include: The weight of the central element is fixed at -1; The sum of the weights of all elements is 0.

4. The video restoration region positioning method according to claim 1, characterized in that, A dual-stream 3D Swin-UNet that incorporates a spatiotemporal attention mechanism includes: A spatiotemporal encoder for extracting multi-scale spatiotemporal features, a decoder with attention mechanism, and a dense temporal prediction head; The skip connection between the spatiotemporal encoder and the decoder integrates a spatial channel squeezing excitation attention module for recalibrating feature maps; The input of the dense temporal prediction head is connected to the output of the decoder.

5. The video restoration region positioning method according to claim 1, characterized in that, The supervision modes include full-time supervision mode and sparse-time supervision mode.

6. The video restoration region positioning method according to claim 5, characterized in that, When the supervision mode is set to full temporal supervision mode, pixel-level loss calculation is performed on the prediction results of each frame in the dense prediction mask sequence. When the supervision mode is set to sparse temporal supervision mode, the ground truth mask of the prediction result of the center frame in the dense prediction mask sequence is obtained, the pixel-level loss between the prediction result of the center frame and the ground truth mask is calculated, the temporal consistency loss between the prediction results of adjacent frames in the dense prediction mask sequence is calculated, and the L1 distance between the prediction masks of adjacent frames is calculated using the temporal consistency loss.

7. The video restoration region positioning method according to claim 6, characterized in that, The pixel-level losses include class balance loss, Dice loss, edge consistency loss, temporal consistency loss, and contrast loss.

8. A video restoration area positioning device, characterized in that, The video restoration region localization device, using the video restoration region localization method as described in any one of claims 1-7, comprises: A construction module is used to extract training video frame sequences from the video segments to be detected, and to construct training RGB stream data and training noise stream data based on the training video frame sequences; The fusion module is used to extract high-frequency residual features from the training noise stream data and perform early fusion of the high-frequency residual features with the original pixel features in the training RGB stream data to obtain dual-modal fusion features. The processing module is used to process the dual-modal fusion features into a dual-stream 3D Swin-UNet with a spatiotemporal attention mechanism to obtain a dense prediction mask sequence. The update module is used to calculate the loss function according to the set supervision mode and the dense prediction mask sequence, and update the network parameters of the dual-stream 3D Swin-UNet with spatiotemporal attention mechanism according to the loss function value to obtain the video restoration area localization model. The localization module is used to construct an RGB stream and a noise stream using the video frame sequence extracted from the video segment to be detected, and to extract high-frequency residual features from the noise stream and fuse them with the RGB stream before inputting them into the video restoration region localization model for localization, thereby obtaining the video restoration region localization result of the video segment to be detected. The number of frames in the video restoration region localization result is consistent with the length of the video frame sequence extracted from the video segment to be detected.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the video restoration region localization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the video restoration region localization method as described in any one of claims 1 to 7.

Images