A method and apparatus for repairing missing video frames
By generating defect masks through self-supervised learning and pre-trained models, and combining defect generation and repair networks, the model is optimized to achieve efficient and highly generalizable defective video frame repair, solving the problems of low efficiency and poor performance in existing technologies.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILE COMM LTD RES INST
- Filing Date
- 2022-10-09
- Publication Date
- 2026-06-30
AI Technical Summary
Existing technologies are inefficient and have poor repair results in repairing missing video frames. They rely on additional data and manual annotation, which limits data diversity and repair effectiveness.
A self-supervised learning method is adopted to generate a defect mask using a pre-trained defect detection model, and generate pseudo-defect regions through a defect generation and repair network model. Finally, the optimal repair result is selected, and the model is optimized by combining self-supervised training to reduce the dependence on additional data.
It improves the efficiency and effectiveness of missing video frame repair, reduces the use of human resources, has stronger generalization ability, and achieves self-supervised missing frame repair.
Smart Images
Figure CN116977192B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for repairing missing video frames. Background Technology
[0002] Self-supervised learning refers to using unlabeled data to extract representational information for supervised learning. Defective video frames refer to video frames where some parts display normally while others do not, such as uneven lighting, shadows, raindrops or water obscuring the lens, watermarks, stains obscuring the lens, missing pixels, or missing pixels.
[0003] In related technologies, the main methods for repairing missing video frames are as follows: The first is the fully supervised learning method: using the missing image and its corresponding unmissing image to learn an image inpainting model in a supervised manner, and then using the inpainting model to repair the missing video frame; the second is the weakly supervised learning method: first, manually annotating the mask of the missing area of the image, and then using the mask and a weakly supervised approach to learn the image inpainting model, and then using the image inpainting model to repair the missing video frame; the third is the method using unpaired data: collecting some unmissing images that do not correspond to the missing image, then using adversarial learning to learn the image inpainting model, and then using the image inpainting model to repair the missing video frame.
[0004] All three methods require additional data to learn the repair model, making them highly dependent on such data. Furthermore, manually labeled data requires significant human resources and is inefficient. The acquisition of non-destructive images requires strict limitations on the acquisition conditions, which restricts data diversity and consequently limits the effectiveness of defect repair. Summary of the Invention
[0005] This invention provides a method and apparatus for repairing missing video frames, thereby solving the technical problems of low efficiency and poor repair effect of missing video frame repair methods in related technologies.
[0006] To solve the above-mentioned technical problems, the present invention is implemented as follows:
[0007] In a first aspect, embodiments of the present invention provide a method for repairing missing video frames, the method comprising:
[0008] The video stream to be detected is input into at least one defect detection model, and defect detection is performed on the video frames of the video stream, and at least one defect mask is output.
[0009] If there is a defective region in the defective mask, the edge processing of the defective mask is performed to obtain a candidate defective mask;
[0010] Based on the candidate defect mask, determine the candidate defect region;
[0011] Outside the candidate defect mask but within the video frame, a random mask is determined, and based on the random mask, a candidate lossless region is determined in the video frame.
[0012] The candidate undamaged regions are input into at least one defect generation network model to generate pseudo-defect regions.
[0013] The pseudo-defect region is input into at least one defect repair network model to obtain at least one repair result for the video frame;
[0014] Among the at least one repair result, the optimal repair result is selected;
[0015] The candidate defect region corresponding to the optimal repair result is input into the defect repair network model to obtain the repair result of the defective video frame.
[0016] Preferably, after inputting the video stream to be detected into at least one defect detection model, performing defect detection on the video frames of the video stream, and outputting at least one defect mask, the method further includes:
[0017] If there is no defective area in the defect mask, the video frame is output, and defect detection continues for the next video frame.
[0018] Preferably, if a defective region exists in the defective mask, edge processing is performed on the defective mask to obtain a candidate defective mask, including:
[0019] Using C pre-set kernels, edge expansion and / or erosion are performed on the defect mask to obtain candidate defect masks;
[0020] The number of candidate defect masks is the product of the number of cores and the number of defect masks.
[0021] Preferably, after inputting the pseudo-defect region into the defect repair network model and obtaining at least one repair result for the video frame, the method further includes:
[0022] The at least one repair result is evaluated, and the evaluation result is obtained;
[0023] Based on the evaluation results, the at least one defect detection model, the at least one defect generation network model, and the at least one defect repair network model are optimized and trained.
[0024] Preferably, evaluating the at least one repair result and obtaining the evaluation result includes:
[0025] Set up E evaluation indicators;
[0026] The at least one repair result is evaluated according to each evaluation index to obtain F initial evaluation results, where each repair result corresponds to E initial evaluation results, and F = D × E;
[0027] Calculate the average score of the E initial evaluation results corresponding to each repair result;
[0028] The average score is determined as the evaluation result;
[0029] Where D represents the number of repair results.
[0030] Preferably, optimizing and training the at least one defect detection model, the at least one defect generation network model, and the at least one defect repair network model based on the evaluation results includes:
[0031] Arrange the average scores in ascending or descending order;
[0032] The endpoints of the interval for the average score are determined based on the highest and lowest average scores.
[0033] Map the interval of the average score to [0, 1];
[0034] The corresponding value of each average score in [0, 1] is used as the weight of the repair result corresponding to the average score;
[0035] The weights are determined as the coefficients of the loss functions of the corresponding defect detection model, defect generation network model, and defect repair network model, thus obtaining the loss function;
[0036] The at least one defect detection model, the at least one defect generation network model, and the at least one defect repair network model are optimized and trained according to the loss function.
[0037] The loss function is inversely proportional to the training performance of the model.
[0038] Preferably, the loss function is:
[0039]
[0040] Where, λ GAN , λ identity , λ rec G represents the weighting coefficients, both ranging from [0, 10]. D It is a defective generative network model, and Dis is a discriminator used only for adversarial training. The optimization objective of GD is to minimize max Dis LGAN (G D Dis) This indicates that the optimization objective of Dis is to maximize L. GAN (G D Dis);
[0041] l rec Let l be the loss function of the defect repair network model. rec =||R n -I n ||1 where R n For the repair result, I n It is the candidate lossless region;
[0042] The loss function of the generative network model includes at least: adversarial loss l GAN and identity mapping loss l identity :
[0043] The resistance loss l GAN This can be expressed by the following formula:
[0044]
[0045] The identity mapping loss l identity This can be expressed by the following formula:
[0046] l identity =||I d -G D (I d )||1;
[0047] Where E represents the mathematical expectation, x ~ p(I) d ) indicates that x follows I d The probability distribution p(I) d ), I d For all candidate defect regions, x is a region from I d A randomly selected sample; z ~ p(I) n ) indicates that z obeys I n The probability distribution p(I) n ), I n For all candidate lossless regions, z is a region from I n A sample randomly selected from the data.
[0048] Preferably, among the at least one repair result, selecting the optimal repair result includes:
[0049] The repair result corresponding to the maximum value in the average score is determined as the optimal repair result.
[0050] Preferably, the candidate defect region corresponding to the optimal repair result is input into the defect repair network model to obtain the repair result of the defective video frame, which is achieved by the following formula:
[0051] R = G R (I·ψ(I))+I·inv(ψ(I));
[0052] Where R represents the repair result of the missing video frame, and G R Let I represent the defect repair network model, I represent the defective video frame, ψ(I) be the candidate defect mask corresponding to the optimal repair result, I·ψ(I) represent the candidate defect region corresponding to the optimal repair result, and inv(ψ(I)) represent the non-defect mask, which represents the inversion of the candidate defect mask corresponding to the optimal repair result, and I·inv(ψ(I)) represent the non-defect region.
[0053] Preferably, the at least one defect detection model detects the same type of defect, and each defect detection model outputs one defect mask.
[0054] Preferably, the defect detection model is a shadow detector, which includes at least one of the following: BDRAR, DSC.
[0055] Secondly, embodiments of the present invention provide a device for repairing missing video frames, the device comprising:
[0056] The defect detection module is used to input the video stream to be detected into at least one defect detection model, perform defect detection on the video frames of the video stream, and output at least one defect mask.
[0057] An edge processing module is used to perform edge processing on the defect mask if there is a defect area in the defect mask, so as to obtain a candidate defect mask;
[0058] The first determining module is used to determine the candidate defect region based on the candidate defect mask;
[0059] The second determining module is used to determine a random mask outside the candidate defect mask and within the video frame, and to determine a candidate lossless region in the video frame based on the random mask;
[0060] The generation module is used to input the candidate undamaged regions into at least one defect generation network model to generate pseudo-defect regions;
[0061] The first acquisition module is used to input the pseudo-defect region into at least one defect repair network model to obtain at least one repair result of the video frame;
[0062] A selection module is used to select the optimal repair result from the at least one repair result;
[0063] The second acquisition module is used to input the candidate defect region corresponding to the optimal repair result into the defect repair network model to obtain the repair result of the defective video frame.
[0064] Preferably, the defect detection module is further configured to input the video stream to be detected into at least one defect detection model, perform defect detection on the video frames of the video stream, and output at least one defect mask. If there is no defect area in the defect mask, the video frame is output, and defect detection is continued on the next video frame of the video frame.
[0065] Preferably, the edge processing module is further configured to use C pre-set kernels to perform edge dilation and / or erosion on the defect mask to obtain candidate defect masks;
[0066] The number of candidate defect masks is the product of the number of cores and the number of defect masks.
[0067] Preferably, the device further includes:
[0068] An evaluation module is used to evaluate the at least one repair result after inputting the pseudo-defect region into the defect repair network model and obtaining at least one repair result of the video frame, and to obtain the evaluation result.
[0069] An optimization training module is used to optimize the training of the at least one defect detection model, the at least one defect generation network model, and the at least one defect repair network model based on the evaluation results.
[0070] Preferably, the evaluation module is further configured to set E evaluation indicators;
[0071] The at least one repair result is evaluated according to each evaluation index to obtain F initial evaluation results, where each repair result corresponds to E initial evaluation results, and F = D × E;
[0072] Calculate the average score of the E initial evaluation results corresponding to each repair result;
[0073] The average score is determined as the evaluation result;
[0074] Where D represents the number of repair results.
[0075] Preferably, the optimized training module is further configured to arrange the average scores in ascending or descending order;
[0076] The endpoints of the interval for the average score are determined based on the highest and lowest average scores.
[0077] Map the interval of the average score to [0, 1];
[0078] The corresponding value of each average score in [0, 1] is used as the weight of the repair result corresponding to the average score;
[0079] The weights are determined as the coefficients of the loss functions of the corresponding defect detection model, defect generation network model, and defect repair network model, thus obtaining the loss function;
[0080] The at least one defect detection model, the at least one defect generation network model, and the at least one defect repair network model are optimized and trained according to the loss function.
[0081] The loss function is inversely proportional to the training performance of the model.
[0082] Preferably, the loss function is:
[0083]
[0084] Where, λ GAN , λ identity , λ rec G represents the weighting coefficients, both ranging from [0, 10]. D It is a defective generative network model, and Dis is a discriminator used only for adversarial training. G represents D The optimization objective is to minimize max Dis L GAN (G D Dis) This indicates that the optimization objective of Dis is to maximize L. GAN (G D Dis);
[0085] l rec Let l be the loss function of the defect repair network model. rec =||R n -I n ||1 where R n For the repair result, I n It is the candidate lossless region;
[0086] The loss function of the generative network model includes at least: adversarial loss l GAN and identity mapping loss l identity ;
[0087] The resistance loss l GAN This can be expressed by the following formula:
[0088]
[0089] The identity mapping loss l identity This can be expressed by the following formula:
[0090] l identity =||I d -G D (I d )||1;
[0091] Where E represents the mathematical expectation, x ~ p(I) d ) indicates that x follows I d The probability distribution p(I) d ), I d For all candidate defect regions, x is a region from I d A randomly selected sample; z ~ p(I) n ) indicates that z obeys I n The probability distribution p(I) n ), I n For all candidate lossless regions, z is a region from I n A sample randomly selected from the data.
[0092] Preferably, the selection module is further configured to determine the repair result corresponding to the maximum value in the average score as the optimal repair result.
[0093] Preferably, the second acquisition module is further configured to input the candidate defect region corresponding to the optimal repair result into the defect repair network model to obtain the repair result of the defective video frame, which is achieved by the following formula:
[0094] R = G R (I·ψ(I))+I·inv(ψ(I));
[0095] Where R represents the repair result of the missing video frame, and G R Let I represent the defect repair network model, I represent the defective video frame, ψ(I) be the candidate defect mask corresponding to the optimal repair result, I·ψ(I) represent the candidate defect region corresponding to the optimal repair result, and inv(ψ(I)) represent the non-defect mask, which represents the inversion of the candidate defect mask corresponding to the optimal repair result, and I·inv(ψ(I)) represent the non-defect region.
[0096] Preferably, the at least one defect detection model detects the same type of defect, and each defect detection model outputs one defect mask.
[0097] Preferably, the defect detection model is a shadow detector, which includes at least one of the following: BDRAR, DSC.
[0098] Thirdly, embodiments of the present invention provide an electronic device, the electronic device comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the method for repairing missing video frames as described in the first aspect.
[0099] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for repairing missing video frames as described in the first aspect.
[0100] In this embodiment, a defect detection model is first used to generate a defect region mask, avoiding the low efficiency caused by manually labeled masks. Furthermore, the defect detection model exhibits high detection performance and generalization ability. Secondly, this application determines the optimal repair result by comparing the repair effects of each repair process on the pseudo-defect region (a repair process can be understood as a repair process composed of a defect detection model, a defect generation network model, and a defect repair network model). The defect repair network model corresponding to the optimal repair result is then used to repair the defective video frame, thereby further ensuring the repair effect. Finally, as can be seen from the overall technical flow of this embodiment, this application does not utilize additional data (such as manually labeled data) for supervised learning of the model. Instead, it uses a pre-trained defect detection model to obtain a defect mask and combines it with a self-supervised training method for the defect repair network model to achieve defect repair. This reduces the use of human resources, improves repair efficiency, and possesses stronger generalization ability. Attached Figure Description
[0101] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0102] Figure 1 A flowchart illustrating a method for repairing missing video frames provided in an embodiment of the present invention;
[0103] Figure 2 A flowchart illustrating a method for repairing missing video frames provided in an embodiment of the present invention;
[0104] Figure 3 A flowchart illustrating a method for repairing missing video frames provided in an embodiment of the present invention;
[0105] Figure 4 This is a schematic diagram of a processing flow for missing video frames provided in an embodiment of the present invention;
[0106] Figure 5 A structural block diagram of a device for repairing missing video frames provided in an embodiment of the present invention;
[0107] Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0108] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0109] like Figure 1 As shown, this embodiment of the invention provides a method for repairing missing video frames, the method comprising:
[0110] Step S101: Input the video stream to be detected into at least one defect detection model, perform defect detection on the video frames of the video stream, and output at least one defect mask;
[0111] Step S102: If there is a defective region in the defective mask, perform edge processing on the defective mask to obtain a candidate defective mask;
[0112] Step S103: Determine the candidate defect region based on the candidate defect mask;
[0113] Step S104: Determine a random mask outside the candidate defect mask and within the video frame; based on the random mask, determine the candidate lossless region in the video frame.
[0114] Step S105: Input the candidate undamaged region into at least one defect generation network model to generate a pseudo defect region;
[0115] Step S106: Input the pseudo-defect region into at least one defect repair network model to obtain at least one repair result for the video frame;
[0116] Step S107: Select the optimal repair result from at least one repair result;
[0117] Step S108: Input the candidate defect region corresponding to the optimal repair result into the defect repair network model to obtain the repair result of the defective video frame.
[0118] In step S101, at least one defect detection model detects the same type of defect, and each defect detection model outputs one defect mask. It should be noted that the defect type can be: rain stains (complex weather or geographical environment), watermarks, shadows (half-lit faces, high-resolution satellite image segmentation), dirt (such as lens smudges), uneven lighting, etc. For example, the defect detection model is a shadow detector, which includes at least one of the following: BDRAR, DSC. The method shown in this embodiment is for defect detection models that detect the same type of defect, that is, at least one defect detection model detects the same type of defect.
[0119] In a preferred implementation, the video stream to be detected is input into at least one defect detection model. After defect detection is performed on the video frames of the video stream and at least one defect mask is output, the method further includes: if no defect region is found in the defect mask, a video frame is output, and defect detection continues on the next video frame. It is understood that the detection unit in this embodiment is each video frame of the video stream to be detected. If no defect region is found in the defect mask, it indicates that the currently detected video frame is lossless, and the current video frame is output. Defect detection can continue on the next video frame until the detection of the video stream to be detected is completed. This allows defect detection to cover every video frame, improving the accuracy of defect detection. Furthermore, only a pre-trained defect detection model-generated defect mask is needed, eliminating the need for manual annotation. The difficulty of existing defect detection is far lower than that of defect repair, while the effectiveness and generalization ability of defect detection are far superior to those of repair.
[0120] In the preferred implementation, in addition to using a pre-trained defect detection model, data priors can also be used for detection.
[0121] In step S102, if there is a defective region in the defective mask, the defective mask is edge-processed to obtain candidate defective masks, including: using C pre-set kernels to perform edge dilation and / or erosion on the defective mask to obtain candidate defective masks; wherein, the number of candidate defective masks is the product of the number of kernels and the number of defective masks.
[0122] C dilation and erosion kernels can be pre-set. The kernel size should not exceed 1% of the image width and should not be less than 1, where C ≥ 1. The C kernels can be used to dilate and / or erode the defect mask to obtain D candidate defect masks, where D = B × C, B is the number of defect masks, and C is the number of kernels.
[0123] Dilation and erosion are opposites. Dilation blends the target point into the background, expanding outwards. Dilation processing can merge broken targets, making it easier to extract them as a whole. Erosion can eliminate connected boundaries, causing image boundaries to shrink inwards. Erosion processing can separate different targets that are stuck together and remove small grain noise.
[0124] In steps S103 and S104, D candidate defect regions can be determined based on D candidate defect masks, and D random masks can be determined outside the D candidate defect masks and within the video frame. Based on the random masks, D candidate lossless regions can be determined in the video frame.
[0125] Understandably, after determining the candidate defect masks, the candidate defect regions can be identified. Then, using the candidate defect masks, a random mask is determined. The random mask is used to determine the candidate non-destructive regions, and the random mask must satisfy the condition that it is not within the region of the D candidate defect masks. After executing steps S103 and S104, the candidate defect regions and candidate non-destructive regions can be determined.
[0126] It should also be noted that the number of candidate defect masks, candidate defect regions, random masks, and candidate non-destructive regions is the same, which is D.
[0127] In steps S105 and S106, candidate lossless regions can be input into at least one defect generation network model to generate pseudo-defect regions, and pseudo-defect regions can be input into at least one defect repair network model to obtain at least one repair result for video frames.
[0128] Understandably, candidate lossless regions can be converted into pseudo-defective regions, and repaired to generate repair results. The repair results can then be compared with their corresponding candidate lossless regions to verify the repair effect and thus understand the merits of the defect repair network model.
[0129] In steps S107 and S108, among at least one repair result, the optimal repair result is selected, and the candidate defect region corresponding to the optimal repair result is input into the defect repair network model to obtain the repair result of the defective video frame. It is understood that among the repair results for pseudo-defective regions, the optimal repair result can be selected. The optimal repair result represents that the detection accuracy and repair effect of the defect detection model, defect generation network model, and defect repair network model corresponding to the output process are better than the models corresponding to other repair results. Therefore, the candidate defect region corresponding to the optimal repair result can be input into the defect repair network model to obtain the final repair result of the defective video frame. Thus, the defect repair of one defective video frame in the video stream to be detected can be completed, and the next defective video frame can be repaired according to the method shown in this embodiment until the repair is complete.
[0130] The optimal repair result can be determined as follows: At least one repair result is evaluated, and the evaluation result is obtained. Specifically, E evaluation indicators are set; at least one repair result is evaluated according to each evaluation indicator, resulting in F initial evaluation results, where each repair result corresponds to E initial evaluation results, and F = D × E; the average score of the E initial evaluation results corresponding to each repair result is calculated; the average score is determined as the evaluation result; where D is the number of repair results, the repair result corresponding to the maximum average score is determined as the optimal repair result.
[0131] Understandably, the number of blind evaluation indicators for image quality can be set to E, where E≥1, and the number of evaluation results is F, where F=D×E. The average score of each defect repair result on the E indicators is calculated, and the repair result corresponding to the maximum value of the average score is determined as the optimal repair result. The candidate defect region corresponding to the optimal repair result is input into the corresponding defect repair network model to infer the repair result of the defective video frame.
[0132] In a preferred implementation, the candidate defect region corresponding to the optimal repair result is input into the defect repair network model to obtain the repair result of the defective video frame, which is achieved by the following formula: R = G R (I·ψ(I))+I·inv(ψ(I));
[0133] Where R represents the restoration result of the missing video frame, and G R Let I represent the defect repair network model, where I represents the defective video frame, ψ(I) is the candidate defect mask corresponding to the optimal repair result, I·ψ(I) represents the candidate defect region corresponding to the optimal repair result, and inv(ψ(I)) represents the non-defect mask, which represents the inversion of the candidate defect mask corresponding to the optimal repair result, and I·inv(ψ(I)) represents the non-defect region.
[0134] It should be noted that after obtaining the repair results, on the one hand, the optimal repair result needs to be selected from the repair results of the pseudo-defect areas (as described above), and on the other hand, the repair results need to be evaluated, and the model needs to be optimized and trained based on the evaluation results to continuously improve the model's repair effect. Correspondingly, in a preferred implementation, after inputting the pseudo-defect areas into the defect repair network model and obtaining at least one repair result for the video frame, the method further includes: evaluating at least one repair result and obtaining the evaluation result, and optimizing and training at least one defect detection model, at least one defect generation network model, and at least one defect repair network model based on the evaluation result.
[0135] The defect generation network model is trained through adversarial training between pseudo-defect regions and real defect regions. Its training data consists of candidate defect regions and candidate undefective regions. The defect repair neural network model is trained using self-supervised training (without utilizing additional data, such as manually labeled data). Its training data consists of candidate undefective regions or candidate undefective regions and video frames from the video stream.
[0136] Among them, such as Figure 2 As shown, the optimization training of at least one defect detection model, at least one defect generation network model, and at least one defect repair network model based on the evaluation results includes:
[0137] Step S201: Arrange the average scores in ascending or descending order;
[0138] Step S202: Determine the endpoints of the average score interval based on the highest and lowest average scores;
[0139] Step S203: Map the interval of the average score to [0,1];
[0140] Step S204: Use the corresponding value of each average score in [0,1] as the weight of the repair result corresponding to the average score;
[0141] Step S205: Determine the weights as the coefficients of the loss functions of the corresponding defect detection model, defect generation network model, and defect repair network model to obtain the loss function;
[0142] Step S206: Optimize and train at least one defect detection model, at least one defect generation network model, and at least one defect repair network model according to the loss function;
[0143] The loss function is inversely proportional to the training performance of the model.
[0144] It should be noted that, firstly, the score range of the evaluation results needs to be uniformly mapped to [0,1]. The closer to 0, the higher the quality; conversely, the further away, the lower the quality. Then, the evaluation results can be used to optimize the corresponding defect detection model, defect generation model, and defect repair network model end-to-end. Specifically, the value corresponding to each average score in [0,1] is used as the weight of the repair result corresponding to the average score. These weights are then used as the coefficients of the loss function for the corresponding defect detection model, defect generation network model, and defect repair network model, resulting in the loss function. This loss function is then used to optimize and train at least one defect detection model, at least one defect generation network model, and at least one defect repair network model, i.e., updating the model parameters end-to-end. This allows for continuous model optimization, reducing the model's dependence on detection accuracy and enhancing its robustness and generalization ability. It should be noted that the loss function is inversely proportional to the model's training effect. The magnitude of the loss function can be used to judge the degree of model training; a larger loss function indicates a poorer training effect, while a smaller loss function indicates a better training effect.
[0145] In one possible implementation, the loss function is:
[0146]
[0147] Where, λ GAN , λ identity , λ rec G represents the weighting coefficients, both ranging from [0, 10]. D It is a defective generative network model, and Dis is a discriminator used only for adversarial training. G represents D The optimization objective is to minimize max Dis L GAN (G D Dis) This indicates that the optimization objective of Dis is to maximize L. GAN (G D Dis);
[0148] l rec For the loss function of the defect repair network model, l rec =||R n -I n ||1, where R n For the repair result, I n It is a candidate lossless region;
[0149] The loss function of a generative network model should at least include: adversarial loss i GAN and identity mapping loss l identity :
[0150] Combat loss GANThis can be expressed by the following formula:
[0151]
[0152] Identity mapping loss l identity This can be expressed by the following formula:
[0153] l identity =||I d -G D (I d )||1;
[0154] Where E represents the mathematical expectation, x ~ p(I) d ) indicates that x follows I d The probability distribution p(I) d ), I d For all candidate defect regions, x is a region from I d A randomly selected sample; z ~ p(I) n ) indicates that z obeys I n The probability distribution p(I) n ), I n For all candidate lossless regions, z is a region from I n A sample randomly selected from the data.
[0155] Therefore, firstly, a defect detection model is used to generate a mask for the defective region, avoiding the low efficiency problem caused by manually labeled masks. Furthermore, the defect detection model exhibits high detection performance and generalization ability. Secondly, this application determines the optimal repair result by comparing the repair effects of each repair process on the pseudo-defective region (the repair process can be understood as: defect detection model, defect generation network model, and defect repair network model). The defect repair network model corresponding to the optimal repair result is then used to repair the defective video frame, thereby further ensuring the repair effect. Finally, as can be seen from the overall technical flow of this application's embodiments, this application does not utilize additional data (such as manually labeled data) for supervised learning of the model. Instead, it uses a pre-trained defect detection model to obtain a defect mask and combines it with a self-supervised training method for the defect repair network model to achieve defect repair. This reduces the use of human resources, improves repair efficiency, and possesses stronger generalization ability.
[0156] like Figure 3 As shown, this embodiment of the invention provides a method for repairing missing video frames, the method comprising:
[0157] Step S301: Obtain a video frame for defect detection;
[0158] Step S302: Determine if there is any loss. If yes, proceed to step S303; otherwise, return to step S301.
[0159] Step S303: Perform edge expansion and erosion on the defect mask detected in step S301 to obtain the defect mask, the defect area, and the undamaged area;
[0160] Step S304: Construct a defect generation network model. This model is trained using adversarial learning and can convert the lossless region obtained in step S302 into a pseudo-defect region.
[0161] Step S305: Construct a defect repair network model. This model uses self-supervised training to process the pseudo-defect areas in step S304 to obtain pseudo-defect repair results.
[0162] Step S306: Perform blind image quality evaluation on the pseudo-defect repair results of step S305, input the defect area result corresponding to the best result into the defect repair network model, and infer the defect frame repair result.
[0163] Step S307: Optimize the model end-to-end using the evaluation results from step S306;
[0164] Step S308: Execute S1 until the video ends.
[0165] In step S301, for the input video stream, A pre-trained defect detection models are obtained to perform defect detection on the video frames in the video stream, where A ≥ 1. For example, if the defect is a color defect, i.e., uneven lighting or shadow coverage, the pre-trained model can be selected from currently available mainstream shadow detectors such as BDRAR and DSC. If a defective region is detected, proceed to step S302; if no defect is detected, directly output the current frame and execute step S1 until the video ends.
[0166] In step S302, the detected B defect masks are subjected to edge dilation and erosion, where B≤A, to obtain D defect masks, defective regions, and undamaged regions.
[0167] Specifically, C dilation and erosion kernels are set, with the kernel size not exceeding 1% of the image width and not less than 1, where C ≥ 1;
[0168] Using the C cores of S21 to expand and erode the B defect masks, D defect masks are obtained, where D = B × C;
[0169] The video frame is divided into D defect regions using D defect masks;
[0170] Obtain D random masks that are not located within the D defective mask regions, and obtain D lossless regions.
[0171] In step S303, a defect generation neural network model is constructed. This model converts the D undamaged regions in step S302 into D pseudo-defective regions. The training method of the model is set to adversarial training, and the training data consists of several defective and undamaged regions obtained in step S302.
[0172] In step S304, a defect repair neural network model is constructed to repair the D pseudo-defect regions obtained in step S303, and the pseudo-defect repair results are obtained; the training method of the model is set to self-supervised training, and the training data are the D lossless regions and video frames obtained in step S302.
[0173] In step S305, a blind image quality assessment is performed on the D pseudo-defect repair results, and the model is optimized and the defect frame repair results are obtained using the assessment results.
[0174] Specifically, the number of blind evaluation indicators for image quality is set to E, where E≥1, and the number of evaluation results is F, where F=D×E. The score range of the evaluation results is uniformly mapped to [0,1]. The closer to 0, the higher the quality, and vice versa.
[0175] Calculate the average score of each defect repair result on E indicators, input the defect area corresponding to the result with the highest average score into the defect repair network in step S304, and infer the defect video frame repair result.
[0176] In step S306, the model is optimized end-to-end using the evaluation results from step S305;
[0177] Specifically, D repair results are assigned repair quality weights, which are the repair results mapped to the [0,1] interval in step S305;
[0178] Using the weights as coefficients of the model's loss function and updating the model parameters end-to-end can reduce the model's dependence on detection accuracy and enhance its robustness and generalization ability.
[0179] In step S307, step S301 is executed until the detection of the video stream to be detected is completed.
[0180] This invention provides a method for repairing missing video frames based on self-supervised learning. Combining a pre-trained model and a neural network model, this method can repair missing frames without using labeled data. It maximizes the use of artificial intelligence technology, reduces the use of human resources, eliminates the dependence of the repair model on additional data, realizes self-supervised repair of missing video frames, and can automatically perform end-to-end optimization, with stronger generalization ability.
[0181] Figure 4A schematic diagram of a processing flow for missing video frames according to an embodiment of the present invention is shown below. Figure 4 Taking an example and combining it with formulas, we will illustrate the process of self-supervised learning.
[0182] Assuming that the defective video frame I consists of a defective region and a non-defective region, the defect mask M can be obtained by pre-training the defect detector ψ. d :
[0183] M d =ψ(I);
[0184] According to M d M can be obtained d Random mask M outside the region n Applying two masks to I can yield the defective region I. d and the undamaged region I n :
[0185]
[0186] The next step is divided into two processes: the training process and the reasoning process, as follows: Figure 4 As shown, the solid line represents the training process, and the dashed line represents the inference process. The training process involves optimizing the model, while the inference process involves determining the final restoration result for the current missing video frame.
[0187] During training, the defective generative network model G D The input is I n The output is the pseudo-defect region P. d P d =G D (I n ), G D The training method is adversarial training, and the loss function should be at least adversarial loss l. GAN and identity mapping loss l identity :
[0188]
[0189] l identity =||I d -G D (I d )||1
[0190] Where E represents the mathematical expectation, Dis is the discriminator used only for adversarial training, and x ~ p(I d ) indicates that x follows I d The probability distribution p(I) d ), I d For all candidate defect regions, x is a region from I dA randomly selected sample; z ~ p(I) n ) indicates that z obeys I n The probability distribution p(I) n ), I n For all candidate lossless regions, z is a region from I n A sample randomly selected from the data.
[0191] The defect repair network model G R The input is P d The output is the pseudo-defect region P. d Repair results R n :R n =G R (P d The defect repair network model is trained using self-supervised training (with I...). n As a supervisory function, the loss function can be the reconstruction loss l rec :l rec =||R n -I n ||1;where R n For the repair result, I n It is the candidate lossless region;
[0192] In summary, the objective function (loss function) for the entire network training is as follows:
[0193]
[0194] Where λ GAN , λ identity , λ rec These represent the weighting coefficients, with values ranging from [0, 10]. G represents D The optimization objective is to minimize max Dis L GAN (G D Dis) This indicates that the optimization objective of Dis is to maximize L. GAN (G D Dis).
[0195] During the reasoning process, the missing region I of the missing video frame d Will input defect repair network G R The output is the repair result of the missing area. Embedding this result into the missing frame yields the repair result R for that frame. The entire process can be represented as:
[0196] R = G R (I·ψ(I))+I·inv(ψ(I));
[0197] Where R represents the repair result of the missing video frame, and G R Let I represent the defect repair network model, I represent the defective video frame, ψ(I) be the candidate defect mask corresponding to the optimal repair result, I·ψ(I) represent the candidate defect region corresponding to the optimal repair result, and inv(ψ(I)) represent the non-defect mask, which represents the inversion of the candidate defect mask corresponding to the optimal repair result, and I·inv(ψ(I)) represent the non-defect region.
[0198] This invention provides a method for repairing missing video frames based on self-supervised learning. Combining a pre-trained model and a neural network model, this method can repair missing frames without using labeled data. It maximizes the use of artificial intelligence technology, reduces the use of human resources, eliminates the dependence of the repair model on additional data, realizes self-supervised repair of missing video frames, and can automatically perform end-to-end optimization, with stronger generalization ability.
[0199] This invention provides a device for repairing missing video frames, such as... Figure 5 As shown, the device 50 includes:
[0200] The defect detection module 501 is used to input the video stream to be detected into at least one defect detection model, perform defect detection on the video frames of the video stream, and output at least one defect mask.
[0201] The edge processing module 502 is used to perform edge processing on the defective mask if there is a defective region in the defective mask, so as to obtain a candidate defective mask.
[0202] The first determining module 503 is used to determine the candidate defect region based on the candidate defect mask;
[0203] The second determining module 504 is used to determine a random mask outside the candidate defect mask and within the video frame, and to determine the candidate lossless region in the video frame based on the random mask;
[0204] The generation module 505 is used to input candidate lossless regions into at least one defect generation network model to generate pseudo defective regions.
[0205] The first acquisition module 506 is used to input the pseudo-defect region into at least one defect repair network model and acquire at least one repair result of the video frame.
[0206] Module 507 is selected to select the optimal repair result from at least one repair result;
[0207] The second acquisition module 508 is used to input the candidate defect region corresponding to the optimal repair result into the defect repair network model to obtain the repair result of the defective video frame.
[0208] In a preferred implementation, the defect detection module 501 is further configured to input the video stream to be detected into at least one defect detection model, perform defect detection on the video frames of the video stream, and output at least one defect mask. If there is no defect area in the defect mask, the video frame is output, and defect detection is continued on the next video frame of the video frame.
[0209] In a preferred implementation, the edge processing module 502 is further configured to use pre-set C kernels to perform edge expansion and / or erosion on the defect mask respectively to obtain candidate defect masks.
[0210] The number of candidate defect masks is the product of the number of kernels and the number of defect masks.
[0211] In a preferred implementation, device 50 further includes:
[0212] The evaluation module is used to evaluate at least one restoration result after inputting the pseudo-defect region into the defect restoration network model and obtaining at least one restoration result of the video frame, and obtain the evaluation result.
[0213] The optimization training module is used to optimize the training of at least one defect detection model, at least one defect generation network model, and at least one defect repair network model based on the evaluation results.
[0214] In a preferred implementation, the evaluation module is also used to set E evaluation metrics;
[0215] Each repair result is evaluated according to each evaluation index to obtain F initial evaluation results, where each repair result corresponds to E initial evaluation results, and F = D × E;
[0216] Calculate the average score of the E initial evaluation results corresponding to each repair result;
[0217] The average score was determined as the evaluation result.
[0218] Where D represents the number of repair results.
[0219] Preferably, the optimized training module is also used to arrange the average scores in ascending or descending order;
[0220] The endpoints of the average score interval are determined based on the highest and lowest average scores.
[0221] Map the interval of the average score to [0, 1];
[0222] The corresponding value of each average score in [0, 1] is used as the weight of the repair result corresponding to the average score;
[0223] The weights are determined as the coefficients of the loss functions of the corresponding defect detection model, defect generation network model, and defect repair network model, thus obtaining the loss function;
[0224] Based on the loss function, at least one defect detection model, at least one defect generation network model, and at least one defect repair network model are optimized and trained.
[0225] The loss function is inversely proportional to the training performance of the model.
[0226] Preferably, the loss function is:
[0227]
[0228] Where, λ GAN , λ identity , λ rec G represents the weighting coefficients, both ranging from [0, 10]. D It is a defective generative network model, and Dis is a discriminator used only for adversarial training. G represents D The optimization objective is to minimize max Dis L GAN (G D Dis) This indicates that the optimization objective of Dis is to maximize L. GAN (G D Dis);
[0229] l rec For the loss function of the defect repair network model, l rec =||R n -I n ||1, where R n For the repair result, I n It is a candidate lossless region;
[0230] The loss function of a generative network model should include at least: adversarial loss. GAN and identity mapping loss l identity :
[0231] Combat loss GAN This can be expressed by the following formula:
[0232]
[0233] Identity mapping loss l identity This can be expressed by the following formula:
[0234] l identity =||I d -G D (I d)||1;
[0235] Where E represents the mathematical expectation, x ~ p(I) d ) indicates that x follows I d The probability distribution p(I) d ), I d For all candidate defect regions, x is a region from I d A randomly selected sample; z ~ p(I) n ) indicates that z obeys I n The probability distribution p(I) n ), I n For all candidate lossless regions, z is a region from I n A sample randomly selected from the data.
[0236] In a preferred implementation, module 507 is further configured to determine the repair result corresponding to the maximum value in the average score as the optimal repair result.
[0237] In a preferred implementation, the second acquisition module 508 is further configured to input the candidate defect region corresponding to the optimal repair result into the defect repair network model to obtain the repair result of the defective video frame, which is achieved through the following formula:
[0238] R = G R (I·ψ(I))+I·inv(ψ(I));
[0239] Where R represents the restoration result of the missing video frame, and G R Let I represent the defect repair network model, where I represents the defective video frame, ψ(I) is the candidate defect mask corresponding to the optimal repair result, I·ψ(I) represents the candidate defect region corresponding to the optimal repair result, and inv(ψ(I)) represents the non-defect mask, which represents the inversion of the candidate defect mask corresponding to the optimal repair result, and I·inv(ψ(I)) represents the non-defect region.
[0240] In a preferred implementation, at least one defect detection model detects the same type of defect, and each defect detection model detects at most one defect mask.
[0241] In a preferred implementation, the defect detection model is a shadow detector, which includes at least one of the following: BDRAR, DSC.
[0242] Therefore, firstly, a defect detection model is used to generate a mask for the defective region, avoiding the low efficiency problem caused by manually labeled masks. Furthermore, the defect detection model exhibits high detection performance and generalization ability. Secondly, this application determines the optimal repair result by comparing the repair effects of each repair process on the pseudo-defective region (the repair process can be understood as: defect detection model, defect generation network model, and defect repair network model). The defect repair network model corresponding to the optimal repair result is then used to repair the defective video frame, thereby further ensuring the repair effect. Finally, as can be seen from the overall technical flow of this application's embodiments, this application does not utilize additional data (such as manually labeled data) for supervised learning of the model. Instead, it uses a pre-trained defect detection model to obtain a defect mask and combines it with a self-supervised training method for the defect repair network model to achieve defect repair. This reduces the use of human resources, improves repair efficiency, and possesses stronger generalization ability.
[0243] This invention provides an electronic device 60, such as... Figure 6 As shown, the electronic device 60 includes a processor 601, a memory 602, and a program stored in the memory 602 and executable on the processor 601. When the program is executed by the processor 601, it implements the steps of the method for repairing missing video frames as described in the above embodiment.
[0244] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the method for repairing missing video frames described in the above embodiments, achieving the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0245] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0246] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0247] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.
Claims
1. A method of repairing a missing video frame, the method comprising: The method includes: The video stream to be detected is input into at least one defect detection model, and defect detection is performed on each video frame of the video stream, and at least one defect mask is output. If there is a defective region in the defective mask, the edge processing of the defective mask is performed to obtain a candidate defective mask; Based on the candidate defect mask, determine the candidate defect region; Outside the candidate defect mask but within the video frame, a random mask is determined, and based on the random mask, a candidate lossless region is determined in the video frame. The candidate undamaged regions are input into at least one defect generation network model to generate pseudo-defect regions. The pseudo-defect region is input into at least one defect repair network model to obtain at least one repair result for the video frame; Among the at least one repair result, the optimal repair result is selected; The candidate defect region corresponding to the optimal repair result is input into the defect repair network model to obtain the repair result of the defective video frame; If there is a defective region in the defective mask, the defective mask is edge-processed to obtain candidate defective masks, including: using C pre-set kernels to perform edge dilation and / or erosion on the defective mask to obtain candidate defective masks; wherein, the number of candidate defective masks is the product of the number of kernels and the number of defective masks.
2. The method of claim 1, wherein, The method further includes inputting the video stream to be detected into at least one defect detection model, performing defect detection on the video frames of the video stream, and outputting at least one defect mask. If there is no defective area in the defect mask, the video frame is output, and defect detection continues for the next video frame.
3. The method of claim 1, wherein, After inputting the pseudo-defect region into the defect repair network model and obtaining at least one repair result for the video frame, the method further includes: The at least one repair result is evaluated, and the evaluation result is obtained; Based on the evaluation results, the at least one defect detection model, the at least one defect generation network model, and the at least one defect repair network model are optimized and trained.
4. The method of claim 3, wherein, Evaluating the at least one repair result and obtaining the evaluation result includes: Set up E evaluation indicators; The at least one repair result is evaluated according to each evaluation index to obtain F initial evaluation results, where each repair result corresponds to E initial evaluation results, and F = D × E; Calculate the average score of the E initial evaluation results corresponding to each repair result; The average score is determined as the evaluation result; Where D represents the number of repair results.
5. The method of claim 4, wherein, The optimization training of the at least one defect detection model, the at least one defect generation network model, and the at least one defect repair network model based on the evaluation results includes: Arrange the average scores in ascending or descending order; The endpoints of the interval for the average score are determined based on the highest and lowest average scores. Map the interval of the average score to [0,1]; The corresponding value of each average score in [0,1] is used as the weight of the repair result corresponding to the average score; The weights are determined as the coefficients of the loss functions of the corresponding defect detection model, defect generation network model, and defect repair network model, thus obtaining the loss function; The at least one defect detection model, the at least one defect generation network model, and the at least one defect repair network model are optimized and trained according to the loss function. The loss function is inversely proportional to the training performance of the model.
6. The method according to claim 5, characterized in that, The loss function is: ； wherein, , , respectively represent weighting coefficients, the value range of which is all in [0, 10], is a defect generation network model, is a discriminator only used for adversarial training, represents The optimization objective of is to minimize represents The optimization objective of is to maximize a loss function for the inpainting network model, wherein is the inpainting result, is the candidate lossless region; The loss function of the generation network model at least comprises: an adversarial loss and an identity mapping loss : the adversarial loss is expressed by the following equation: ； the identity mapping loss is expressed by the following equation: ； wherein, denotes the mathematical expectation, denotes is subject to the probability distribution , is all candidate missing regions, is a sample randomly selected from ; the probability distribution is all candidate intact regions, is a sample randomly selected from .
7. The method of claim 5, wherein, Among the at least one repair result, selecting the optimal repair result includes: The repair result corresponding to the maximum value in the average score is determined as the optimal repair result.
8. The method according to claim 1, characterized in that, The candidate defect region corresponding to the optimal repair result is input into the defect repair network model to obtain the repair result of the defective video frame, which is achieved through the following formula: ； in, This indicates the repair result of the missing video frame. This represents a defect repair network model. This refers to the missing video frame. (I) represents the candidate defect mask corresponding to the optimal repair result. This represents the candidate defect region corresponding to the optimal repair result. This represents a non-defect mask, where the non-defect mask is the inverted version of the candidate defect mask corresponding to the optimal repair result. Indicates the non-defective area.
9. The method according to claim 1, characterized in that, The at least one defect detection model detects the same type of defect, and each defect detection model outputs one defect mask.
10. The method according to any one of claims 1-9, characterized in that, The defect detection model is a shadow detector, which includes at least one of the following: BDRAR, DSC.
11. A device for repairing missing video frames, characterized in that, The device includes: The defect detection module is used to input the video stream to be detected into at least one defect detection model, perform defect detection on each video frame of the video stream, and output at least one defect mask. An edge processing module is used to perform edge processing on the defect mask if there is a defect area in the defect mask, so as to obtain a candidate defect mask; The first determining module is used to determine the candidate defect region based on the candidate defect mask; The second determining module is used to determine a random mask outside the candidate defect mask and within the video frame, and to determine a candidate lossless region in the video frame based on the random mask; The generation module is used to input the candidate undamaged regions into at least one defect generation network model to generate pseudo-defect regions; The first acquisition module is used to input the pseudo-defect region into at least one defect repair network model to obtain at least one repair result of the video frame; A selection module is used to select the optimal repair result from the at least one repair result; The second acquisition module is used to input the candidate defect region corresponding to the optimal repair result into the defect repair network model to obtain the repair result of the defective video frame; If a defective region exists in the defective mask, edge processing is performed on the defective mask to obtain candidate defective masks. This includes: using C pre-set kernels to perform edge dilation and / or erosion on the defective mask to obtain candidate defective masks; wherein the number of candidate defective masks is the product of the number of kernels and the number of defective masks.
12. An electronic device, characterized in that, The electronic device includes: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method for repairing missing video frames as described in any one of claims 1-10.
13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method for repairing missing video frames as described in any one of claims 1-10.