Image restoration method and device, electronic equipment and storage medium

Abstract

The invention provides an image restoration method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence, in particular to the field of deep learning and the field of image processing. According to the specific implementation scheme, the method comprises the steps of determining dynamic information of a to-be-restored image based on the to-be-restored image in a video and a previous frame of image of the to-be-restored image; based on the dynamic information, determining a related frame proportion of the repaired image; determining a target related frame of the to-be-restored image in the video based on the related frame proportion; and repairing the to-be-repaired image based on the target related frame to obtain a target image.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to the field of deep learning and image processing. Background Art

[0002] Video restoration technology aims to repair damaged or missing parts in a video to make the video content coherent and reasonable. With the popularity of mobile devices and augmented reality devices, since the memory and computing power of these devices are limited, it is necessary to consider image restoration under memory constraints. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device and storage medium for image restoration.

[0004] According to one aspect of the present disclosure, there is provided an image restoration method, comprising:

[0005] Determine dynamic information of the image to be restored based on the image to be restored in the video and the previous frame of the image to be restored;

[0006] Based on the dynamic information, determine the relevant frame ratio of the repaired image;

[0007] Based on the relevant frame ratio, determining the target relevant frame of the image to be repaired in the video;

[0008] The image to be repaired is repaired based on the target-related frame to obtain the target image.

[0009] According to another aspect of the present disclosure, there is provided an image restoration device, comprising:

[0010] A dynamic determination module, used to determine the dynamic information of the image to be repaired based on the image to be repaired in the video and the previous frame of the image to be repaired;

[0011] A ratio determination module, used for determining the relevant frame ratio of the repaired image based on the dynamic information;

[0012] A related frame determination module, used for determining a target related frame of an image to be repaired in a video based on a related frame ratio;

[0013] The image restoration module is used to restore the image to be restored based on the target related frame to obtain the target image.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] The technical solution of the disclosed embodiment can determine the dynamic information of the image to be repaired by using the image to be repaired in the video and the previous frame of the image to be repaired, and adjust the ratio of related frames according to the dynamic information, and repair the image to be repaired in combination with the adjusted related frames. Therefore, the technical solution of the disclosed embodiment can adaptively adjust the ratio of related frames instead of fixed input settings, reduce the processing of invalid related frames, and thus improve the quality and efficiency of image repair while reducing the requirements of computing power and memory.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flowchart of an image restoration method provided by an embodiment of the present disclosure;

[0024] Figure 2 is a schematic diagram of an application example of the image restoration method provided by an embodiment of the present disclosure;

[0025] Figure 3 is a schematic block diagram of an image restoration device provided by an embodiment of the present disclosure;

[0026] Figure 4 is a schematic block diagram of an image restoration device provided by another embodiment of the present disclosure;

[0027] Figure 5 It is a block diagram of an electronic device used to implement the image restoration method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0029] Figure 1 The video processing method provided by an embodiment of the present disclosure is shown. The method can be applied to a video processing device, which can be deployed in an electronic device. The electronic device is, for example, a single or multi-machine terminal, server or other processing device. Among them, the terminal can be a user equipment (User Equipment, UE) such as a mobile device, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.; the server can be a stand-alone server or a server cluster. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the method may include the following steps S110 to S140.

[0030] Step S110: determining the dynamic information of the image to be restored based on the image to be restored in the video and the previous frame of the image to be restored.

[0031] In the disclosed embodiment, the image to be restored in the video may be any frame image in the video. For example, the image to be restored may include an image with low definition and missing parts in the video.

[0032] Optionally, an image designated by the user to be repaired may be determined as the image to be repaired, or multiple frames of images in a video may be identified and the image to be repaired may be determined from the multiple frames of images in the video.

[0033] Optionally, the number of images to be repaired may be one or more. In the case where there are multiple images to be repaired, the dynamic information of each image to be repaired is determined based on each image to be repaired in the video and the previous frame image of each image to be repaired, wherein the image to be repaired in the video may also be referred to as the current frame, i.e., the image to be repaired currently being processed.

[0034] Exemplarily, when the image to be restored is an image of the second frame of video in a video, the previous frame of image of the image to be restored may be an image of the first frame of video in the video.

[0035] In the disclosed embodiment, the dynamic information of the image to be restored may represent the dynamic changes between the image to be restored and the previous frame of the image to be restored, that is, the difference between the image to be restored and the previous frame of the image to be restored.

[0036] Optionally, the dynamic information may include text and / or a numerical value (such as a percentage) representing the magnitude of the dynamic change between the image to be restored and the previous frame of the image to be restored. For example, when the numerical value is high, the dynamic change between the image to be restored and the previous frame of the image to be restored is large.

[0037] Optionally, the dynamic information of the image to be repaired can be obtained through one or more indicators, wherein the method of determining the indicator can include perceptual hash algorithm, Euclidean distance, cosine similarity, convolutional neural network, etc.

[0038] Step S120: Determine the relevant frame ratio of the image to be restored based on the dynamic information.

[0039] Exemplarily, the target related frames that need to be determined in image restoration may include adjacent frames and / or reference frames of the image to be restored, then the related frame ratio of the image to be restored may include the ratio of adjacent frames and / or the ratio of reference frames, and by determining the ratio of adjacent frames and / or reference frames, the number of adjacent frames and / or reference frames can be determined.

[0040] Among them, the adjacent frames may include frames that are temporally adjacent to the image to be repaired, and these frames may provide timing information for subsequent video repair, such as motion trajectory and continuity. Exemplarily, assuming that the image to be repaired is the i-th frame in the video, i is an integer greater than or equal to 1, then the adjacent frames may include the ik-th frame to the i-1-th frame in the video, and / or the i+1-th frame to the i+k-th frame, k is an integer greater than or equal to 1.

[0041] The reference frame may include a video frame containing a similar background or scene to the image to be restored, which provides static background details for subsequent video restoration. The temporal distance between the reference frame and the image to be restored may be unlimited, for example, the temporal distance between the reference frame and the image to be restored may be relatively far.

[0042] Optionally, the relevant frame ratio may include the ratio of the target relevant frames in the video. For example, the relevant frame ratio may be 0.001. When the video has 2000 frames, it can be determined based on the ratio that the target relevant frames have 2 frames.

[0043] Optionally, the relevant frame ratio may include the ratio of a single type of target-related frames in all target-related frames. Exemplarily, the target-related frames include adjacent frames and reference frames, and the relevant frame ratio may include an adjacent frame ratio and / or a reference frame ratio. For example, when the reference frame ratio of the restored image is 0.7, the adjacent frame ratio of the restored image is 0.3. Optionally, the reference frame ratio may be determined based on dynamic information, and the adjacent frame ratio may be determined based on the reference frame ratio. Alternatively, the adjacent frame ratio may be determined based on dynamic information, and the reference frame ratio may be determined based on the adjacent frame ratio.

[0044] Step S130: determining target related frames of the image to be repaired in the video based on the related frame ratio.

[0045] In the embodiment of the present disclosure, the number of reference frames and adjacent frames can be determined based on the relevant frame ratio, and then a corresponding number of target relevant frames can be extracted from the video. For example, the adjacent frames include the first k frame images and / or the last k frame images of the reference frame, and the value of k is determined based on the relevant frame ratio.

[0046] Optionally, based on the number of adjacent frames, a corresponding number of adjacent frames may be determined in the video using a timestamp or a frame index of the video.

[0047] Optionally, based on the number of reference frames, a corresponding number of reference frames may be determined in the video by identifying the content in each image frame.

[0048] Optionally, candidate reference frames and adjacent frames may be determined in the video first, and then based on the relevant frame ratio, the actually used reference frames and adjacent frames may be extracted from the candidate reference frames and adjacent frames.

[0049] Step S140: repair the image to be repaired based on the target-related frame to obtain a target image.

[0050] In the disclosed embodiment, the target image is an image that has been restored for the image to be restored. For example, the target image has a higher resolution than the image to be restored, or the target image completes the area to be restored compared to the image to be restored.

[0051] Optionally, the image to be repaired may be repaired based on target-related frames by using texture synthesis, deep learning, and other methods in related technologies to obtain a target image.

[0052] Optionally, the target image can be used to replace the image to be repaired in the video to generate a high-quality repaired video with the same time as the original video.

[0053] The related art mainly relies on a fixed number of related frame inputs. However, with the popularity of mobile devices and augmented reality devices, the restoration of long videos or high-resolution videos has become particularly difficult due to the limited memory and computing power of these devices, and the restoration quality is inconsistent when processing static and dynamic video content, making it difficult to balance memory efficiency and video quality. In addition, the related art also proposes improved solutions based on optical flow, Transformer and fusion methods, but the input configuration is still not fully optimized, so that the best restoration effect cannot be achieved under limited memory conditions. The above method provided by the embodiment of the present disclosure can determine the dynamic information of the image to be restored by using the image to be restored in the video and the previous frame of the image to be restored, and adjust the relevant frame ratio by dynamic information, and repair the image to be restored in combination with the adjusted target related frame. Therefore, the technical solution of the embodiment of the present disclosure can adaptively adjust the relevant frame ratio based on the dynamic information of the image to be restored, rather than a fixed input setting. Since the dynamic information of the image to be restored reflects the amount of change between the image to be restored and other frames, adjusting the relevant frame ratio based on dynamic information can make the relevant frame ratio more accurate and reduce the processing of invalid related frames, thereby reducing the requirements for computing power and memory while improving the image restoration quality and efficiency.

[0054] In some embodiments, the above step S110, determining the dynamic information of the image to be restored based on the image to be restored in the video and the previous frame of the image to be restored, includes:

[0055] Based on the image to be restored and the previous frame image, determine at least one of optical flow information and pixel change amount;

[0056] Based on at least one of the above items, dynamic information is determined.

[0057] Optionally, the optical flow information may be information determined based on the optical flow change between the image to be restored and the previous frame image, for example, it may be an indicator used to describe the direction, speed, size, etc. of pixel movement between the image to be restored and the previous frame image.

[0058] Optionally, the pixel change amount may be information determined based on the pixel difference between the image to be repaired and the previous frame image, for example, the pixel points of the image to be repaired may be compared one by one with the pixel points of the previous frame image to obtain the pixel change amount. For example, the pixel change amount may include the color difference between the pixel values ​​corresponding to each pixel point in the image to be repaired and the previous frame image.

[0059] Optionally, the optical flow information and / or the pixel change amount may be proportional to the dynamic information. For example, the optical flow information and the pixel change amount may be added to obtain the dynamic information. Alternatively, the average value of the optical flow information and the pixel change amount may be calculated to obtain the dynamic information.

[0060] According to the above embodiments, dynamic information can be accurately determined through optical flow information and / or pixel change, so that related frames can be accurately configured subsequently, further improving the quality and efficiency of image restoration.

[0061] In some embodiments, the indicator for determining the dynamic information includes at least optical flow information. Specifically, based on the image to be repaired and the previous frame image, at least one of the optical flow information and the pixel change amount is determined, including:

[0062] Determine the optical flow field between the image to be repaired and the previous frame image;

[0063] Based on the mask corresponding to the area to be repaired and the optical flow field, determine the average optical flow intensity in the area to be repaired;

[0064] Based on the average optical flow intensity, the optical flow information is obtained.

[0065] In the disclosed embodiment, the optical flow field can describe the information of pixel motion between the image to be restored and the previous frame image, and can be in the form of a two-dimensional vector field, which represents the motion direction and speed of each pixel, and can quantify the dynamics of the motion area. Specifically, each pixel corresponds to a two-dimensional vector (u, v), where u and v represent the motion components in the horizontal and vertical directions, respectively. In the optical flow field of an image to be restored and the previous frame image, different moving objects will generate their own sets of motion vectors. Therefore, the optical flow field is not a single two-dimensional vector, but a two-dimensional vector field covering the entire image to be restored and the previous frame image.

[0066] Optionally, an optical flow estimation model may be used to calculate the optical flow field between the image to be restored and the previous frame image. The optical flow estimation model may be, for example, a RAFT (Recurrent All-Pairs Field Transforms) model.

[0067] In the disclosed embodiments, the area to be repaired refers to the area in the image to be repaired that needs to be repaired or filled, which can be marked by a mask. The mask can define the pixel range that needs to be processed, such as an occluded object or lost information. In the video repair task, the area to be repaired may be dynamic, that is, it moves or adjusts over time. For example, in an image to be repaired, if an object is occluded, this occluded area can be defined as the area to be repaired, and this part needs to be repaired.

[0068] Optionally, the optical flow field may include many vectors, and some vectors may be incomplete. Therefore, based on the mask corresponding to the area to be repaired, the optical flow field corresponding to the area to be repaired can be extracted from the image to be repaired and the previous frame image, and the incomplete vectors can be completed. Based on the mask corresponding to the area to be repaired and the completed optical flow field, the average optical flow intensity in the area to be repaired can be determined.

[0069] In the disclosed embodiment, the average optical flow intensity refers to the average intensity of the optical flow in the area to be repaired, rather than a single two-dimensional vector. This is achieved by averaging the optical flow intensity of all pixels in the area to be repaired. The optical flow field of the area to be repaired will be integrated into a scalar value to represent the overall dynamic characteristics. For example, the average optical flow intensity Flow can be calculated by the following formula mask :

[0070]

[0071] Among them, u and v are the components of the total optical flow of the area to be repaired in the horizontal and vertical directions, that is, is the total optical flow intensity in the area to be repaired, and mask_size is the number of pixels marked by the mask corresponding to the area to be repaired.

[0072] According to the above embodiment, the average optical flow intensity in the area to be repaired can be determined through the mask and optical flow field corresponding to the area to be repaired, and the dynamic changes in the area to be repaired can be effectively identified and processed, providing more accurate optical flow information for video restoration, and further improving the accuracy of dynamic information, so that the relevant frames can be accurately configured subsequently, further improving the image restoration quality and efficiency.

[0073] In some embodiments, the indicator for determining the dynamic information includes at least a pixel change amount. Specifically, based on the image to be repaired and the previous frame image, determining at least one of the optical flow information and the pixel change amount includes:

[0074] Based on the mask corresponding to the area to be repaired, respectively determine a first target pixel in the image to be repaired and a second target pixel in the previous frame of image;

[0075] Based on the difference between the first target pixel and the second target pixel, a pixel variation is obtained.

[0076] In the disclosed embodiment, the mask corresponding to the area to be repaired can mark the pixels in the area to be repaired, and the pixels in the area to be repaired in the image to be repaired can be determined as the first target pixels, and the pixels in the area to be repaired in the previous frame image can be determined as the second target pixels.

[0077] In the embodiment of the present disclosure, the pixel change amount may represent the change amount between the pixel value of the first target pixel and the pixel value of the second target pixel.

[0078] For example, the pixel change is calculated by the following formula:

[0079]

[0080] Among them, m*n is the size of the image, M k,t is the mask pixel value at time t (i.e., the image to be repaired), M k,t-1 is the mask pixel value at time t-1 (i.e., the previous frame of the image to be repaired). Furthermore, the formula is only valid for the area to be repaired, that is, in the formula, only the difference between the target pixel in the image to be repaired and the target pixel in the previous frame can have a pixel change. The mask pixel value of the non-to-be-repaired area is always 0, and the pixel change is also 0.

[0081] According to the above embodiment, the difference between the first target pixel in the image to be repaired and the second target pixel in the previous frame image can be used to provide a more accurate pixel change amount for video repair, further improving the accuracy of dynamic information, so that the relevant frames can be accurately configured subsequently, further improving the image repair quality and efficiency.

[0082] In some embodiments, based on at least one of the above, determining the dynamic information includes:

[0083] Based on preset weights, the optical flow information and pixel changes are weighted and summed to obtain dynamic information; wherein the preset weights are obtained by linear regression using the image data set.

[0084] In the embodiment of the present disclosure, the optical flow information and the pixel change amount can be combined to calculate the dynamic information by weighted average. For example, the dynamic information x can be obtained by calculating the optical flow information and the pixel change amount by the following formula: comb :

[0085]

[0086] Among them, x flow and x mask Can be respectively normalized optical flow information and pixel change, m flow and m mask They can be the optical flow information weight and pixel change weight of linear regression respectively.

[0087] Optionally, if the image to be repaired includes fast-moving objects, the preset weight of the optical flow information can be relatively increased to ensure that the fast-changing details can be captured. Correspondingly, if the main content of the image to be repaired is a static background, the preset weight of the pixel change amount can be relatively increased to optimize the quality of background repair.

[0088] Optionally, linear regression can be performed using the image data set to obtain preset weights corresponding to images to be repaired with different contents. Based on the preset weights corresponding to the current image to be repaired, the optical flow information and pixel change amount can be weighted summed to obtain dynamic information.

[0089] According to the above embodiment, by performing linear regression on the image data set, preset weights that match the image to be repaired can be obtained, and by performing weighted summation of the optical flow information and the pixel change amount in combination with the preset weights, more accurate dynamic information can be obtained, thereby improving the adaptability and accuracy of image restoration, and further improving the quality and efficiency of subsequent image restoration.

[0090] In some embodiments, the value interval corresponding to the relevant frame ratio of the image to be restored is the same as the value interval to which the dynamic information belongs.

[0091] Exemplarily, the above step S120, determining the relevant frame ratio of the restored image based on the dynamic information, includes: determining the relevant frame ratio of the restored image from a plurality of relevant frame ratios corresponding to the plurality of value intervals based on the value interval to which the dynamic information belongs. Specifically, the relevant frame ratio corresponding to the value interval to which the dynamic information belongs among the plurality of relevant frame ratios is used as the relevant frame ratio of the image to be restored.

[0092] In the disclosed embodiment, when the value of the dynamic information is within a value range, it can be determined that the value range is to which the dynamic information belongs, and the relevant frame ratio of the repaired image can be determined based on the relevant frame ratio corresponding to the value range. The target relevant frames can include adjacent frames and reference frames of the image to be repaired, and the relevant frame ratio of the repaired image can be the ratio of the adjacent frames to the reference frames.

[0093] In the disclosed embodiment, there may be multiple value intervals, and the relevant frame ratios corresponding to each value interval may be different.

[0094] Optionally, multiple value intervals and the relevant frame ratio corresponding to each value interval can be preconfigured. In one example, only the reference frame ratio corresponding to each value interval can be preconfigured, and based on the reference frame ratio corresponding to each value interval, the adjacent frame ratio corresponding to each value interval can be determined.

[0095] Optionally, the target related frames may include adjacent frames and reference frames of the image to be repaired, and the related frame ratio of the repaired image, that is, the ratio of the adjacent frames to the reference frames, can be obtained by the dynamic information x comb The value of x comb When ≤0.25, the reference frame ratio is high (e.g. 0.875) and the neighboring frame ratio is low (e.g. 0.125). <x comb When ≤0.5, the reference frame ratio is moderate (e.g. 0.625) and the neighboring frame ratio is increased (e.g. 0.375). <x comb When ≤0.85, the dynamic change is large, the adjacent frame ratio is high (e.g., 0.875), and the reference frame ratio is low (e.g., 0.125). These ratios can be optimized through experiments to adapt to different dynamic scenes.

[0096] According to the above embodiment, the relevant frame ratio of the restored image can be accurately determined based on the value interval to which the dynamic information belongs, so that the adjacent frames and reference frames can be accurately configured subsequently, further improving the image restoration quality and efficiency.

[0097] In some embodiments, the target-related frames include adjacent frames and reference frames of the image to be repaired. Accordingly, the above step S140, repairing the image to be repaired based on the target-related frames to obtain the target image, includes:

[0098] The image to be repaired, adjacent frames and reference frames are input into the video repairer to obtain the target image.

[0099] In the embodiment of the present disclosure, the adjacent frames may include frames that are temporally adjacent to the image to be restored, namely, the first and last frames of the image to be restored, which can provide timing information such as motion trajectory and continuity for the video restorer.

[0100] In the embodiment of the present disclosure, the reference frame may include a video frame that is far away from the image to be repaired (i.e., a frame with a large time interval from the image to be repaired) but contains a similar background or scene. These frames provide static background details for the repair model.

[0101] Optionally, the video restorer may adopt a neural network model to perform the task of video restoration based on the image to be restored, adjacent frames, and reference frames to obtain a target image.

[0102] Optionally, an optimized video restorer may be used to process the input image. The optimized video restorer may utilize the result of the optical flow completion module and reuse the completed optical flow information in the optical flow guidance model to reduce computational overhead and improve restoration efficiency.

[0103] Optionally, the repair process may include: the video repairer repairs the pixel values ​​of the to-be-repaired area in the video by configuring the input frame, while achieving temporal consistency between frames, and generating a high-quality and temporally consistent target image.

[0104] According to the above embodiment, the video repairer can be used to repair the image to be repaired in combination with adjacent frames and reference frames, which can further improve the image repair quality and efficiency.

[0105] In order to more clearly understand the technical solution of the embodiment of the present disclosure, a specific application example is provided below. Figure 2 A schematic diagram of the process of image restoration method is shown in FIG. Figure 2 As shown, the image restoration method may include the following steps S210-S250:

[0106] Step S210: obtaining an input frame, that is, obtaining an image to be restored, and adjacent frames and a reference frame of the image to be restored.

[0107] Step S220, visual dynamic analysis, that is, analyzing the video dynamics based on the optical flow information and / or the pixel change amount to obtain dynamic information, may specifically include the following steps S221-S223:

[0108] Step S221, optical flow analysis, may include: using an optical flow estimation model (such as RAFT) to calculate the optical flow field between the image to be repaired and the previous frame image to quantify the dynamics of the motion area. Apply optical flow completion to the area to be repaired, and calculate the average optical flow intensity of the area to be repaired by the formula Where (u, v) is the optical flow component (i.e., optical flow field), and mask_size is the number of pixels marked by the mask corresponding to the area to be repaired. The optical flow information can be obtained based on the average optical flow intensity.

[0109] Step S222, pixel change analysis, may include: calculating the change of the target pixel in the to-be-repaired region between the to-be-repaired image and the previous frame image, and determining the pixel change. For example, the pixel change is calculated by the formula Among them, m·n is the size of the image, M k,t is the mask pixel value at time t (i.e., the image to be repaired), M k,t-1 is the mask pixel value at time t-1 (i.e., the previous frame of the image to be repaired).

[0110] Step S223, calculating dynamic information, may include: inputting the optical flow information and the pixel change amount into the configurator, and obtaining the dynamic information according to the analysis results of the optical flow information and the pixel change amount, wherein the dynamic information x can be obtained by calculating the optical flow information and the pixel change amount using the following formula: comb :

[0111]

[0112] Among them, x flow and x mask can be respectively the normalized optical flow information and mask change, m flow and m mask They can be the optical flow information weight and mask change weight of linear regression respectively.

[0113] Step S230, calculating the relevant frame ratio, may include: based on the dynamic information x comb Adjust the ratio of the adjacent frame to the reference frame. For example: comb When ≤0.25, the reference frame ratio is high (e.g. 0.875) and the neighboring frame ratio is low (e.g. 0.125). <x comb When ≤0.5, the reference frame ratio is moderate (e.g. 0.625) and the neighboring frame ratio is increased (e.g. 0.375). <x comb When ≤0.85, the dynamic change is large, the adjacent frame ratio is high (e.g., 0.875), and the reference frame ratio is low (e.g., 0.125). These ratios can be optimized through experiments to adapt to different dynamic scenes.

[0114] Step S240, repair model processing: that is, sending the configured input frame to the video repairer to complete the repair of the area to be repaired. Specifically, the video repairer can use a neural network model, and optimizing the video repairer can include: using the results of the optical flow completion module, in the optical flow guidance model, repeatedly using the completed optical flow information to reduce the calculation overhead and improve the repair efficiency. The repair process may include: the video repairer fills the pixel values ​​of the area to be repaired in the video by configuring the input frame, and at the same time achieves time consistency between frames.

[0115] Step S250, outputting a repaired frame; the repaired frame is the target image, so that the target image can be used to replace the image to be repaired in the video, generating a high-quality repaired video with the same time as the original video.

[0116] It can be seen that the image restoration method provided by the embodiment of the present disclosure can adaptively adjust the relevant frame ratio according to the dynamic information of the video context, rather than a fixed input setting, so that the processing of invalid relevant frames can be reduced under the condition of limited memory, while reducing the requirements for computing power and memory, and improving the image restoration quality and efficiency. In addition, it can be compatible with multiple video restoration models, improve the consistency of the restoration effect, and has strong universality.

[0117] According to an embodiment of the present disclosure, the present disclosure also provides an image restoration device, Figure 3 A schematic block diagram of an image restoration device provided by an embodiment of the present disclosure is shown. Figure 3 As shown, the device comprises:

[0118] A dynamic determination module 310, configured to determine dynamic information of the image to be restored based on the image to be restored in the video and a previous frame of the image to be restored;

[0119] A ratio determination module 320, for determining a relevant frame ratio of the repaired image based on the dynamic information;

[0120] A related frame determination module 330, for determining a target related frame of an image to be restored in the video based on the related frame ratio;

[0121] The image restoration module 340 is used to restore the image to be restored based on the target related frame to obtain a target image.

[0122] In some embodiments, Figure 4 As shown, the dynamic determination module 310 includes:

[0123] A data determination submodule 410 is used to determine at least one of optical flow information and pixel variation based on the image to be restored and the previous frame image;

[0124] The dynamic determination submodule 420 is used to determine dynamic information based on at least one of the above items.

[0125] In some embodiments, the data determination submodule 410 is specifically configured to:

[0126] Determine the optical flow field between the image to be repaired and the previous frame image;

[0127] Based on the mask corresponding to the area to be repaired and the optical flow field, determine the average optical flow intensity in the area to be repaired;

[0128] Based on the average optical flow intensity, the optical flow information is obtained.

[0129] In some embodiments, the data determination submodule 410 is specifically configured to:

[0130] Based on the mask corresponding to the area to be repaired, respectively determine a first target pixel in the image to be repaired and a second target pixel in the previous frame of image;

[0131] Based on the difference between the first target pixel and the second target pixel, a pixel variation is obtained.

[0132] In some embodiments, the dynamic determination submodule 420 is specifically configured to:

[0133] Based on preset weights, the optical flow information and pixel changes are weighted and summed to obtain dynamic information; wherein the preset weights are obtained by linear regression using the image data set.

[0134] In some embodiments, the value interval corresponding to the relevant frame ratio of the image to be restored is the same as the value interval to which the dynamic information belongs.

[0135] In some embodiments, the target-related frames include adjacent frames and reference frames of the image to be repaired; the image repair module 340 is specifically configured to:

[0136] The image to be repaired, adjacent frames and reference frames are input into the video repairer to obtain the target image.

[0137] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0138] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0139] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0140] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0141] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 to a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0142] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0143] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as the image restoration method. For example, in some embodiments, the image restoration method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the image restoration method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the image restoration method in any other appropriate manner (eg, by means of firmware).

[0144] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0145] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0146] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0148] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0149] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0150] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0151] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image restoration method, comprising: Determine dynamic information of the image to be restored based on the image to be restored in the video and a previous frame of the image to be restored; Based on the dynamic information, determining a relevant frame ratio of the image to be restored; Based on the relevant frame ratio, determining a target relevant frame of the image to be repaired in the video; The image to be repaired is repaired based on the target related frame to obtain a target image.

2. The method according to claim 1, wherein: The determining the dynamic information of the image to be repaired based on the image to be repaired in the video and the previous frame image of the image to be repaired includes: Based on the image to be restored and the previous frame image, determine at least one of optical flow information and pixel change amount; Based on the at least one item, the dynamic information is determined.

3. The method according to claim 2, wherein: The determining at least one of optical flow information and pixel variation based on the image to be restored and the previous frame image includes: Determine the optical flow field between the image to be restored and the previous frame image; Determine the average optical flow intensity in the area to be repaired based on the mask corresponding to the area to be repaired and the optical flow field; The optical flow information is obtained based on the average optical flow intensity.

4. The method according to claim 2 or 3, wherein: The determining at least one of optical flow information and pixel variation based on the image to be restored and the previous frame image includes: Based on the mask corresponding to the to-be-repaired area, respectively determining a first target pixel in the to-be-repaired image and a second target pixel in the previous frame image; The pixel variation is obtained based on a difference between the first target pixel and the second target pixel.

5. The method according to any one of claims 2 to 4, wherein: The determining the dynamic information based on the at least one item includes: Based on preset weights, the optical flow information and the pixel change amount are weighted and summed to obtain the dynamic information; wherein the preset weights are obtained by performing linear regression using an image data set.

6. The method according to any one of claims 1 to 5, wherein: The value interval corresponding to the relevant frame ratio of the image to be restored is the same as the value interval to which the dynamic information belongs.

7. The method according to any one of claims 1 to 6, wherein: The target-related frames include adjacent frames and reference frames of the image to be restored; The step of repairing the image to be repaired based on the target related frame to obtain a target image includes: The image to be restored, the adjacent frame and the reference frame are input into a video restorer to obtain the target image.

8. An image restoration device, comprising: A dynamic determination module, used to determine the dynamic information of the image to be restored based on the image to be restored in the video and the previous frame image of the image to be restored; A ratio determination module, used for determining the relevant frame ratio of the image to be restored based on the dynamic information; A related frame determination module, used for determining a target related frame of the image to be restored in the video based on the related frame ratio; The image restoration module is used to restore the image to be restored based on the target related frame to obtain a target image.

9. The device according to claim 8, wherein: The dynamic determination module comprises: A data determination submodule, used to determine at least one of optical flow information and pixel change amount based on the image to be restored and the previous frame image; The dynamic determination submodule is used to determine the dynamic information based on the at least one item.

10. The device according to claim 9, wherein: The data determination submodule is further used for: Determine the optical flow field between the image to be restored and the previous frame image; Determine the average optical flow intensity in the area to be repaired based on the mask corresponding to the area to be repaired and the optical flow field; The optical flow information is obtained based on the average optical flow intensity.

11. The device according to claim 9 or 10, wherein: The data determination submodule is further used for: Based on the mask corresponding to the to-be-repaired area, respectively determining a first target pixel in the to-be-repaired image and a second target pixel in the previous frame image; The pixel variation is obtained based on a difference between the first target pixel and the second target pixel.

12. The device according to any one of claims 9 to 11, wherein: The dynamic determination submodule is further used for: Based on preset weights, the optical flow information and the pixel change amount are weighted and summed to obtain the dynamic information; wherein the preset weights are obtained by performing linear regression using an image data set.

13. The device according to any one of claims 8 to 12, wherein: The value interval corresponding to the relevant frame ratio of the image to be restored is the same as the value interval to which the dynamic information belongs.

14. The device according to any one of claims 8 to 13, wherein: The target-related frames include adjacent frames and reference frames of the image to be restored; The image restoration module is further used for: The image to be restored, the adjacent frame and the reference frame are input into a video restorer to obtain the target image.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images