Image restoration methods and apparatuses, electronic devices and computer-readable storage media

By acquiring and fusing image feature data with similar timestamps from the video to be repaired, and using feature data with different spatial dimensions for image restoration, the problem of image quality degradation caused by equipment shaking during video shooting is solved, achieving more efficient image clarity improvement.

CN115049553BActive Publication Date: 2026-11-17SHENZHEN SENSETIME TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202210690307.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-17
Publication Date
2026-11-17
Estimated Expiration
2042-06-17

AI Technical Summary

Technical Problem

Existing technologies often result in image quality degradation during video recording due to factors such as equipment shake, defocusing, and low light, and traditional optical flow methods are ineffective in restoring these qualities.

Method used

By acquiring feature data from two images in the video to be repaired whose timestamp difference is less than a threshold, feature data with different spatial dimensions are used for fusion repair, including inter-channel and intra-channel fusion. Image quality is improved through feature extraction, downsampling, upsampling and decoding.

Benefits of technology

It improves image restoration results, enhances image clarity, reduces data processing volume, and improves the efficiency and quality of image restoration.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115049553B_ABST
    Figure CN115049553B_ABST
Patent Text Reader

Abstract

This application discloses an image restoration method, apparatus, electronic device, and computer-readable storage medium. The method includes: acquiring first feature data of a first image and second feature data of a second image; both the first image and the second image belong to a video to be restored, and the difference between the timestamp of the first image and the timestamp of the second image is less than or equal to a time threshold; obtaining at least one third feature data based on the first feature data; the spatial positions of the at least one third feature data are different from the spatial positions of the first feature data; fusing the second feature data with the at least one third feature data to obtain fourth feature data; and restoring the second image based on the fourth feature data to obtain a first restored image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image restoration method and apparatus, electronic device and computer-readable storage medium. Background Technology

[0002] During video recording, factors such as camera shake, focus issues, low ambient light, and high-speed movement of the subject can easily degrade the image quality of the video. These degradation factors include at least one of the following: noise and blur. Traditional methods use optical flow to restore the image quality of video, but the restoration effect is poor. Summary of the Invention

[0003] This application provides an image restoration method and apparatus, an electronic device, and a computer-readable storage medium.

[0004] Firstly, an image restoration method is provided, the method comprising:

[0005] Obtain first feature data of the first image and second feature data of the second image; both the first image and the second image belong to the video to be repaired, and the difference between the timestamp of the first image and the timestamp of the second image is less than or equal to a time threshold.

[0006] Based on the first feature data, at least one third feature data is obtained; the position of the at least one third feature data in the spatial dimension is different from the position of the first feature data in the spatial dimension.

[0007] The second feature data is fused with the at least one third feature data to obtain the fourth feature data;

[0008] The second image is repaired based on the fourth feature data to obtain the first repaired image.

[0009] In the first aspect, when the image restoration device acquires first feature data and second feature data of a first image, and the first image and the second image are two images with different timestamps in the video to be restored, it obtains at least one third feature data that is different from the first feature data in spatial dimension based on the first feature data, thereby increasing the probability that at least one third feature data includes feature data that is in the same spatial dimension as the second feature data.

[0010] In this way, by fusing the second feature data with at least one third feature data to obtain the fourth feature data, the effect of using the feature information carried by the first feature data to repair the feature information carried by the second feature data can be improved, thereby improving the image quality of the second image using the feature information of the first image.

[0011] Therefore, the second image is repaired based on the fourth feature data to obtain the first repaired image. Improving the image quality of the first repaired image can improve the repair effect on the second image, in other words, it can make the second image clearer.

[0012] In any embodiment of this application, the video to be repaired further includes a third image, wherein the timestamp of the second image is located between the timestamp of the first image and the timestamp of the third image;

[0013] Before repairing the second image based on the fourth feature data to obtain the first repaired image, the method further includes:

[0014] Obtain the fifth feature data of the third image;

[0015] The fourth feature data and the fifth feature data are fused to obtain the sixth feature data;

[0016] The step of repairing the second image based on the fourth feature data to obtain the first repaired image includes:

[0017] The second image is repaired based on the sixth feature data to obtain the first repaired image.

[0018] In this embodiment, the image restoration device, after restoring the feature information carried by the second feature data using the feature information carried by the first feature data to obtain the fourth feature data, fuses the fourth feature data and the fifth feature data to obtain the sixth feature data, and obtains the first restored image based on the sixth feature data. This allows the second image to be restored using the feature information of both the first and third images.

[0019] Since the timestamp of the second image is located between the timestamps of the first and third images, this implementation method can improve the restoration effect of the second image.

[0020] Furthermore, in this embodiment, the image restoration device first uses the feature information carried by the first feature data to restore the feature information carried by the second feature data to obtain the fourth feature data, and then uses the feature information carried by the fifth feature data to restore the feature information carried by the fourth feature data to obtain the sixth feature data. This is equivalent to first fusing the feature information of the first image and the feature information of the second image, and then fusing the feature information of the third image and the feature information of the second image, which is equivalent to performing two fusions.

[0021] Since any two of the first, second, and third images are located at different positions in the spatial dimension, if the feature information of the first, second, and third images is directly fused, the positional differences of the three images in the spatial dimension need to be considered simultaneously, which leads to poor image quality in the repaired second image and poor repair effect.

[0022] This implementation method only needs to consider the positional difference between two images in the spatial dimension during each fusion process. Therefore, compared with directly fusing the feature information of three images, it can improve the restoration effect.

[0023] In any embodiment of this application, the number of channels of the fifth feature data is s, where s is a positive number greater than 1;

[0024] The process of fusing the fourth feature data and the fifth feature data to obtain the sixth feature data includes:

[0025] The seventh feature data is obtained by selecting data from m channels of the fifth feature data, where m is less than s;

[0026] The fourth feature data and the seventh feature data are fused to obtain the sixth feature data.

[0027] Since the amount of data for the seventh feature data is smaller than that for the fifth feature data, the image restoration device can reduce the amount of data processing required to obtain the sixth feature data by implementing this method.

[0028] In any embodiment of this application, the step of repairing the second image based on the sixth feature data to obtain the first repaired image includes:

[0029] The sixth feature data is downsampled to obtain the eighth feature data;

[0030] The eighth feature data is subjected to feature extraction processing to obtain the ninth feature data;

[0031] The ninth feature data is upsampled to obtain the tenth feature data; the size of the tenth feature data is the same as the size of the sixth feature data.

[0032] The tenth feature data is decoded to obtain the first repaired image.

[0033] In this embodiment, the image restoration device first downsamples the sixth feature data to obtain the eighth feature data, and then performs feature extraction processing on the eighth feature data to obtain the ninth feature data. This can increase the receptive field of the feature extraction processing and thus improve the effect of the feature extraction processing.

[0034] Therefore, by upsampling the ninth feature data to obtain the tenth feature data with the same size as the sixth feature data, and by decoding the tenth feature data, the first restored image is obtained, which can improve the image quality of the first restored image.

[0035] In any embodiment of this application, fusing the second feature data with the at least one third feature data to obtain the fourth feature data includes:

[0036] The second feature data and the at least one third feature data are fused between channels to obtain the eleventh feature data;

[0037] The eleventh feature data is fused within a channel to obtain the twelfth feature data;

[0038] The fourth feature data is obtained based on the twelfth feature data.

[0039] In this embodiment, the image restoration device obtains eleventh feature data by performing inter-channel fusion of second feature data and at least one third feature data, and then performs intra-channel fusion of the eleventh feature data to obtain twelfth feature data. This is equivalent to achieving three-dimensional information fusion through the fusion of two two-dimensional information. In this way, the fusion effect of the second feature data and at least one third feature data can be improved, and the amount of data processing generated by the fusion can be reduced.

[0040] In any embodiment of this application, the twelfth feature data includes a thirteenth feature data and a fourteenth feature data, wherein the thirteenth feature data belongs to a first channel of the twelfth feature data and the fourteenth feature data belongs to a second channel of the twelfth feature data;

[0041] The process of obtaining the fourth feature data based on the twelfth feature data includes:

[0042] Obtain the first weight of the first channel and the second weight of the second channel;

[0043] The first weight is used as the fusion weight of the thirteenth feature data, and the second weight is used as the fusion weight of the fourteenth feature data. The thirteenth feature data and the fourteenth feature data are weighted and fused to obtain the fourth feature data.

[0044] In this embodiment, since the first weight represents the importance of the information in the first channel, the second weight represents the importance of the information in the second channel, and the thirteenth feature data is the data of the first channel and the fourteenth feature data is the data of the second channel, the image restoration device performs weighted fusion of the thirteenth feature data and the fourteenth feature data according to the first weight of the first channel and the second weight of the second channel to obtain the fourth feature data, which can improve the fusion effect.

[0045] In conjunction with any embodiment of this application, obtaining at least one third feature data based on the first feature data includes:

[0046] Obtain at least one offset;

[0047] The first feature data is moved within the spatial dimension according to the at least one offset to obtain the at least one third feature data.

[0048] In this embodiment, the image restoration device moves the first feature data in the spatial dimension according to at least one offset to obtain at least one third feature data.

[0049] In conjunction with any embodiment of this application, after obtaining the first repaired image, the method further includes:

[0050] Replace the second image in the video to be repaired with the first repaired image.

[0051] In this embodiment, the image restoration device replaces the second image in the video to be restored with a first restoration image to obtain a restored video, which can improve the video quality of the video to be restored.

[0052] In any embodiment of this application, the image restoration method is implemented through an image restoration network, which includes a feature extraction module, a feature fusion module, and a decoding module.

[0053] The step of obtaining at least one third feature data based on the first feature data includes:

[0054] The feature fusion module obtains at least one third feature based on the first feature data;

[0055] The step of fusing the second feature data with the at least one third feature data to obtain the fourth feature data includes:

[0056] The second feature data is fused with the at least one third feature data by the feature fusion module to obtain the fourth feature data;

[0057] The step of repairing the second image based on the fourth feature data to obtain the first repaired image includes:

[0058] The feature fusion module and the decoding module obtain the first repaired image of the second image based on the fourth feature data;

[0059] The acquisition of the first feature data of the first image and the second feature data of the second image includes:

[0060] The feature extraction module extracts feature information from the first image to obtain the first feature data;

[0061] The feature extraction module extracts the feature information of the second image to obtain the second feature data.

[0062] In conjunction with any embodiment of this application, the image restoration method further includes a training process for the image restoration network:

[0063] Acquire a training video, which includes a fourth image and a fifth image, wherein the difference between the timestamp of the fourth image and the timestamp of the fifth image is less than or equal to the time threshold.

[0064] The fourth image is used to repair the fifth image to obtain the second repaired image;

[0065] A first loss is obtained based on the first difference between the second repaired image and the sixth image; the sixth image is the true value of the fifth image; the first loss is positively correlated with the first difference.

[0066] The parameters of the image inpainting network are updated based on the first loss.

[0067] In this embodiment, the image restoration device can train the image restoration network to learn the ability to restore images, thereby realizing the image restoration method described above.

[0068] In any embodiment of this application, the step of repairing the fifth image using the fourth image to obtain the second repaired image includes:

[0069] The feature extraction module extracts the feature information of the fourth image to obtain the fifteenth feature data;

[0070] The feature extraction module extracts the feature information of the fifth image to obtain the sixteenth feature data;

[0071] The feature fusion module processes the fifteenth and sixteenth feature data to obtain the seventeenth feature data, which is the restored feature data of the fifth image.

[0072] The second repaired image is obtained by the decoding module based on the seventeenth feature data.

[0073] In conjunction with any embodiment of this application, before updating the parameters of the image inpainting network according to the first loss, the training process further includes:

[0074] The sixteenth feature data is decoded to obtain a decoded image;

[0075] A second loss is obtained based on the second difference between the brightness of the decoded image and the brightness of the sixth image; the second loss is positively correlated with the second difference.

[0076] The step of updating the parameters of the image inpainting network based on the first loss includes:

[0077] The parameters of the image inpainting network are updated based on the first loss and the second loss.

[0078] In this embodiment, during the training process, the image inpainting device processes the fifteenth and sixteenth feature data through a feature fusion module to obtain the seventeenth feature data. The seventeenth feature data is then processed by a decoding module to obtain a second inpainted image of the fifth image. A first loss is obtained based on a first difference between the second inpainted image and the sixth image.

[0079] The feature extraction module extracts feature information from the fourth and fifth images respectively, obtaining the fifteenth and sixteenth feature data. The sixteenth feature data is then decoded to obtain a decoded image. A second loss is derived based on the second difference in brightness between the decoded image and the sixth image. The parameters of the network to be repaired are updated based on this second loss, reducing the feature information of degradation factors extracted by the feature extraction module. Therefore, updating the parameters of the network to be repaired based on the first and second losses improves the repair effect.

[0080] Secondly, an image restoration apparatus is provided, the apparatus comprising:

[0081] The acquisition unit is used to acquire first feature data of a first image and second feature data of a second image; both the first image and the second image belong to the video to be repaired, and the difference between the timestamp of the first image and the timestamp of the second image is less than or equal to a time threshold.

[0082] A first processing unit is configured to obtain at least one third feature data based on the first feature data; the position of each of the at least one third feature data in the spatial dimension is different from the position of the first feature data in the spatial dimension.

[0083] The second processing unit is used to fuse the second feature data with the at least one third feature data to obtain the fourth feature data;

[0084] The third processing unit is used to repair the second image based on the fourth feature data to obtain the first repaired image.

[0085] In any embodiment of this application, the video to be repaired further includes a third image, wherein the timestamp of the second image is located between the timestamp of the first image and the timestamp of the third image;

[0086] The acquisition unit is further configured to acquire the fifth feature data of the third image;

[0087] The second processing unit is further configured to fuse the fourth feature data and the fifth feature data to obtain the sixth feature data;

[0088] The third processing unit is used to repair the second image based on the sixth feature data to obtain the first repaired image.

[0089] In any embodiment of this application, the number of channels of the fifth feature data is s, where s is a positive number greater than 1;

[0090] The second processing unit is used for:

[0091] The seventh feature data is obtained by selecting data from m channels of the fifth feature data, where m is less than s;

[0092] The fourth feature data and the seventh feature data are fused to obtain the sixth feature data.

[0093] In conjunction with any embodiment of this application, the third processing unit is configured to:

[0094] The sixth feature data is downsampled to obtain the eighth feature data;

[0095] The eighth feature data is subjected to feature extraction processing to obtain the ninth feature data;

[0096] The ninth feature data is upsampled to obtain the tenth feature data; the size of the tenth feature data is the same as the size of the sixth feature data.

[0097] The tenth feature data is decoded to obtain the first repaired image.

[0098] In conjunction with any embodiment of this application, the second processing unit is configured to:

[0099] The second feature data and the at least one third feature data are fused between channels to obtain the eleventh feature data;

[0100] The eleventh feature data is fused within a channel to obtain the twelfth feature data;

[0101] The fourth feature data is obtained based on the twelfth feature data.

[0102] In any embodiment of this application, the twelfth feature data includes a thirteenth feature data and a fourteenth feature data, wherein the thirteenth feature data belongs to a first channel of the twelfth feature data and the fourteenth feature data belongs to a second channel of the twelfth feature data;

[0103] The second processing unit is used for:

[0104] Obtain the first weight of the first channel and the second weight of the second channel;

[0105] The first weight is used as the fusion weight of the thirteenth feature data, and the second weight is used as the fusion weight of the fourteenth feature data. The thirteenth feature data and the fourteenth feature data are weighted and fused to obtain the fourth feature data.

[0106] In conjunction with any embodiment of this application, the first processing unit is configured to:

[0107] Obtain at least one offset;

[0108] The first feature data is moved within the spatial dimension according to the at least one offset to obtain the at least one third feature data.

[0109] In any embodiment of this application, the third processing unit is further configured to replace the second image in the video to be repaired with the first repaired image.

[0110] In any embodiment of this application, the image restoration method is implemented through an image restoration network, which includes a feature extraction module, a feature fusion module, and a decoding module.

[0111] The first processing unit is configured to obtain at least one third feature data based on the first feature data through the feature fusion module;

[0112] The second processing unit is used to fuse the second feature data with the at least one third feature data through the feature fusion module to obtain the fourth feature data;

[0113] The third processing unit is used to obtain a first repaired image of the second image based on the fourth feature data through the feature fusion module and the decoding module;

[0114] The acquisition unit is used to extract feature information of the first image through the feature extraction module to obtain the first feature data.

[0115] The acquisition unit is further configured to extract feature information of the second image through the feature extraction module to obtain the second feature data.

[0116] In conjunction with any embodiment of this application, the image inpainting method performed by the image processing apparatus further includes a training process for the image inpainting network, and the image processing apparatus further includes a training unit, the training unit being used for:

[0117] Acquire a training video, which includes a fourth image and a fifth image, wherein the difference between the timestamp of the fourth image and the timestamp of the fifth image is less than or equal to the time threshold.

[0118] The fourth image is used to repair the fifth image to obtain the second repaired image;

[0119] A first loss is obtained based on the first difference between the second repaired image and the sixth image; the sixth image is the true value of the fifth image; the first loss is positively correlated with the first difference.

[0120] The parameters of the image inpainting network are updated based on the first loss.

[0121] In any embodiment of this application, the training unit is used for:

[0122] The feature extraction module extracts the feature information of the fourth image to obtain the fifteenth feature data;

[0123] The feature extraction module extracts the feature information of the fifth image to obtain the sixteenth feature data;

[0124] The feature fusion module processes the fifteenth and sixteenth feature data to obtain the seventeenth feature data, which is the restored feature data of the fifth image.

[0125] The second repaired image is obtained by the decoding module based on the seventeenth feature data.

[0126] In conjunction with any embodiment of this application, the training unit is further configured to:

[0127] The sixteenth feature data is decoded to obtain a decoded image;

[0128] A second loss is obtained based on the second difference between the brightness of the decoded image and the brightness of the sixth image; the second loss is positively correlated with the second difference.

[0129] The parameters of the image inpainting network are updated based on the first loss and the second loss.

[0130] Thirdly, an electronic device is provided, characterized in that it comprises: a processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect above and any possible implementation thereof.

[0131] Fourthly, another electronic device is provided, comprising: a processor, a transmitting device, an input device, an output device, and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs the method as described in the first aspect above and any possible implementation thereof.

[0132] Fifthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program including program instructions that, when executed by a processor, cause the processor to perform a method as described in the first aspect above and any possible implementation thereof.

[0133] In a sixth aspect, a computer program product is provided, the computer program product comprising a computer program or instructions, wherein, when the computer program or instructions are executed on a computer, the computer performs the method described in the first aspect and any possible implementation thereof.

[0134] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0135] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0136] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0137] Figure 1 A schematic flowchart illustrating an image restoration method provided in an embodiment of this application;

[0138] Figure 2 A schematic diagram of an image coordinate system provided in an embodiment of this application;

[0139] Figure 3 This is a schematic diagram of the network structure of an image restoration network provided in an embodiment of this application;

[0140] Figure 4 This is a schematic diagram of the structure of a feature extraction unit provided in an embodiment of this application;

[0141] Figure 5 A schematic diagram of the structure of a grouping timing and spatial shifting module provided in an embodiment of this application;

[0142] Figure 6 A schematic diagram of a training framework for an image inpainting network provided in an embodiment of this application;

[0143] Figure 7 This is a schematic diagram of the structure of an image restoration device provided in an embodiment of this application;

[0144] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0145] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0146] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0147] It should be understood that in this application, "at least one (item)" refers to one or more, "more than one" refers to two or more, "at least two (items)" refers to two or three or more, and "and / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" can indicate three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " can indicate that the related objects before and after are in an "or" relationship, referring to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple. The character " / " can also represent the division sign in mathematical operations, for example, a / b = a divided by b; 6 / 3 = 2. "At least one of the following" or similar expressions.

[0148] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0149] The execution subject of this application embodiment is an image restoration device, which can be any electronic device capable of executing the technical solutions disclosed in the method embodiments of this application. Optionally, the image restoration device can be one of the following: a mobile phone, a computer, a tablet computer, or a wearable smart device.

[0150] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The embodiments of this application are described below with reference to the accompanying drawings. Please refer to... Figure 1 , Figure 1 This is a schematic flowchart of an image restoration method provided in an embodiment of this application.

[0151] 101. Obtain the first feature data of the first image and the second feature data of the second image.

[0152] In this embodiment of the application, the first image and the second image are any two images in the video to be repaired, that is, both the first image and the second image belong to the video to be repaired, and the difference between the timestamp of the first image and the timestamp of the second image is less than or equal to the time threshold.

[0153] In this embodiment, the video to be repaired can be either an offline video or an online video. Offline videos can be videos captured by a camera or a mobile smart device. Online videos can be videos captured in real-time by a camera.

[0154] The smaller the difference in timestamps between two images in a video, the higher the correlation between the content of the two images. In this embodiment, the timestamp of an image in a video represents the playback time of the image during video playback. For example, if the video to be repaired includes image a, and the timestamp of image a is t1, then image a is played at time t1 of the video to be repaired.

[0155] In this embodiment, a time threshold is used to distinguish whether the difference between the timestamps of two images is large or small. Specifically, if the difference between the timestamps of two images is greater than the time threshold, it indicates that the difference between the timestamps of two images is large; if the difference between the timestamps of two images is less than or equal to the time threshold, it indicates that the difference between the timestamps of two images is small.

[0156] Optionally, the first image and the second image are two adjacent frames in the video to be repaired.

[0157] In this embodiment of the application, the image feature information (including the feature information of the first image and the feature information of the second image mentioned above, as well as the feature information of the (i-1)th frame image, the i-th frame image, the (i+1)th frame image, the fourth image, and the fifth image, which will be mentioned below) includes at least one of the following: image texture information, image color information, shape information of objects within the image, image brightness information, and spatial relationship information of different objects within the image. The image feature data carries the image feature information.

[0158] Specifically, the first feature data carries feature information of the first image, wherein the feature information of the first image includes at least one of the following: texture information of the first image, color information of the first image, shape information of objects within the first image, brightness information of the first image, and spatial relationship information of different objects within the first image. The second feature data carries feature information of the second image, wherein the feature information of the second image includes at least one of the following: texture information of the second image, color information of the second image, shape information of objects within the second image, brightness information of the second image, and spatial relationship information of different objects within the second image.

[0159] For example, if the objects in the first image include a table and a stool, then the feature information of the first image includes: the texture information of the table, the color information of the table, the shape information of the table, and the brightness information of the table. The feature data of the first image (i.e., the first feature data) carries the following information: the texture information of the table, the color information of the table, the shape information of the table, and the brightness information of the table.

[0160] For example, if the objects in the second image include a table and a stool, then the feature information of the second image includes: the texture information of the table, the color information of the table, the shape information of the table, the brightness information of the table, the texture information of the stool, the color information of the stool, the shape information of the stool, the brightness information of the stool, and the spatial relationship information between the table and the stool. The feature data of the second image (i.e., the second feature data) carries the following information: the texture information of the table, the color information of the table, the shape information of the table, the brightness information of the table, the texture information of the stool, the color information of the stool, the shape information of the stool, the brightness information of the stool, and the spatial relationship information between the table and the stool.

[0161] In one implementation of acquiring first feature data of a first image and second feature data of a second image, the image restoration device receives the first feature data and second feature data input by a user through an input component. The input component includes at least one of the following: a keyboard, a mouse, a touchscreen, a touchpad, or an audio input device.

[0162] In another implementation of acquiring first feature data of the first image and second feature data of the second image, the image restoration device receives the first feature data and second feature data sent by the terminal. The terminal can be any of the following: a mobile phone, a computer, a tablet computer, or a server.

[0163] In another implementation of obtaining first feature data of a first image and second feature data of a second image, the image restoration device obtains first feature data by performing feature extraction processing on the first image and second feature data by performing feature extraction processing on the second image.

[0164] 102. Based on the first feature data mentioned above, at least one third feature data is obtained.

[0165] Before proceeding with the following description, the spatial dimensions in the embodiments of this application are defined. Using Height (hereinafter referred to as H) to represent the height of the image, Width (hereinafter referred to as W) to represent the width of the image, and Channel (hereinafter referred to as C) to represent the channel of the image (i.e., the depth of the image), the size of the image can be expressed as H*W*C.

[0166] At this point, the image coordinate system can be represented as OHWC. Specifically, the image coordinate system is as follows: Figure 2As shown, the horizontal axis OW represents width, the vertical axis OC represents channel, and the vertical axis OH represents height. The spatial dimension is the plane HOW.

[0167] In this embodiment, the spatial position of at least one third feature data is different from the spatial position of the first feature data; that is, the position of any third feature data in the plane HOW is different from the position of the first feature data in the plane HOW. Optionally, the information carried by at least one third feature data is the same as the information carried by the first feature data.

[0168] In one possible implementation, the image inpainting apparatus acquires a first position of first feature data in a spatial dimension, adds the first position to at least one offset to obtain at least one second position, wherein, if the number of offsets is greater than 1, any two offsets are different. The first feature data is then moved to each of the at least one second position to obtain at least one third feature data. Optionally, if the number of offsets is greater than 1, at least one offset follows an arithmetic sequence.

[0169] In another possible implementation, the image restoration apparatus acquires at least one preset position, wherein, if the number of preset positions is greater than one, any two preset positions are different. First feature data is then moved to each of the at least one preset position to obtain at least one third feature data.

[0170] 103. The second feature data is fused with at least one third feature data to obtain the fourth feature data.

[0171] As mentioned above, the difference between the timestamps of the first image and the second image is small, therefore the image content of the first image and the image content of the second image are highly correlated. Thus, by fusing the first image and the second image, the image quality of the second image can be improved using the first image.

[0172] At least one third feature data carries feature information from the first image, and the second feature data carries feature information from the second image. Therefore, by fusing the second feature data with at least one third feature data, the feature information of the first image can be used to repair the feature information of the second image, thereby achieving the effect of restoring the image quality of the second image.

[0173] For example, by fusing the second feature data with at least one third feature data, the texture information carried by the at least one third feature data can be used to enrich the texture information carried by the second feature data, thereby achieving the effect of repairing the texture of the second image.

[0174] For example, by fusing the second feature data with at least one third feature data, the color information carried by the at least one third feature data can be used to correct the color information carried by the fourth feature data, thereby achieving the effect of correcting the color of the second image.

[0175] Since any two images with different timestamps in the video to be repaired are spatially different, directly fusing the feature data of two images with different timestamps results in poor fusion quality. Therefore, to improve the fusion effect, the feature data of two images with different timestamps can be spatially aligned before fusing, reducing the spatial difference in the position of the feature data of the two images with different timestamps, thereby improving the fusion effect.

[0176] In this embodiment of the application, the image restoration device obtains at least one third feature data whose position in the spatial dimension is different from that of the first feature data based on the first feature data. Then, the at least one third feature data can be used as feature data including the first feature data aligned with the second feature data.

[0177] It should be understood that since the spatial position difference between the first and second feature data is uncertain, the spatial position difference between the third feature data and the first feature data is an estimate of the spatial position difference between the first and second feature data. Therefore, the more third feature data there are, the greater the probability that at least one third feature data includes feature data in the same spatial position as the first feature data.

[0178] The image restoration apparatus can improve the fusion effect of feature data of the first image and feature data of the second image by fusing second feature data with at least one third feature data, which includes feature data that aligns the first feature data to the second feature data.

[0179] 104. Based on the fourth feature data mentioned above, the second image is repaired to obtain the first repaired image.

[0180] In this embodiment of the application, the first repaired image is the image obtained after repairing the image quality of the second image, that is, the image quality of the first repaired image is higher than the image quality of the second image.

[0181] In one possible implementation, the image restoration device restores the second image by decoding the fourth feature data, thereby obtaining the first restored image.

[0182] In this embodiment of the application, when the image restoration device acquires first feature data and second feature data of a first image, and the first image and the second image are two images with different timestamps in the video to be restored, it obtains at least one third feature data that is different from the first feature data in spatial dimension based on the first feature data, thereby increasing the probability that at least one third feature data includes feature data that is in the same spatial dimension as the second feature data.

[0183] In this way, by fusing the second feature data with at least one third feature data to obtain the fourth feature data, the effect of using the feature information carried by the first feature data to repair the feature information carried by the second feature data can be improved, thereby improving the image quality of the second image using the feature information of the first image.

[0184] Therefore, the second image is repaired based on the fourth feature data to obtain the first repaired image. Improving the image quality of the first repaired image can improve the repair effect on the second image, in other words, it can make the second image clearer.

[0185] It should be understood that the first image and the second image in the embodiments of this application are descriptive objects determined for the purpose of concisely describing the implementation process of the technical solution, and should not be construed as the video to be repaired consisting of only the first image and the second image. In practical applications, the video to be repaired may include x images, where x is a positive integer greater than 1. Moreover, when x is greater than 2, the image repair device can repair any image. The repair method is to use images whose timestamps differ from the timestamps of the image to be repaired by less than or equal to a time threshold to repair the image to be repaired. For the specific implementation process, please refer to the implementation method of using the first image to repair the second image described in steps 101 to 104.

[0186] As an optional implementation, the video to be repaired also includes a third image, with the timestamp of the second image falling between the timestamps of the first and third images. For example, the timestamp of the first image is t1, the timestamp of the second image is t2, and the timestamp of the third image is t3. Then, t1 < t2 < t3, or t3 < t2 < t1.

[0187] Optionally, the first image, the second image, and the third image are three adjacent frames in the video to be repaired.

[0188] The image restoration device also performs the following steps:

[0189] 201. Obtain the fifth feature data of the third image mentioned above.

[0190] In this embodiment of the application, the fifth feature data carries feature information of the third image, wherein the feature information of the third image includes at least one of the following: texture information of the third image, color information of the third image, shape information of objects in the third image, brightness information of the third image, and spatial relationship information of different objects in the third image.

[0191] In one implementation of acquiring fifth feature data of a third image, the image restoration device receives the fifth feature data input by a user through an input component.

[0192] In another implementation of acquiring the fifth feature data of the third image, the image restoration device receives the fifth feature data sent by the terminal.

[0193] In another implementation of obtaining the fifth feature data of the third image, the image restoration device obtains the fifth feature data by performing feature extraction processing on the third image.

[0194] 202. The fourth feature data and the fifth feature data are fused together to obtain the sixth feature data.

[0195] The image restoration device obtains a sixth feature data by fusing the fourth and fifth feature data. It can use the feature information carried by the fifth feature data to restore the feature information carried by the fourth feature data, thereby achieving the effect of restoring the image quality of the second image.

[0196] For example, by fusing the fourth feature data with the fifth feature data, the texture information carried by the fifth feature data can be used to enrich the texture information carried by the fourth feature data, thereby achieving the effect of repairing the texture of the second image.

[0197] For example, by fusing the fourth feature data with the fifth feature data, the color information carried by the fifth feature data can be used to correct the color information carried by the fourth feature data, thereby achieving the effect of correcting the color of the second image.

[0198] Having obtained the sixth feature data, the image restoration device performs the following steps during step 104:

[0199] 203. The second image is repaired based on the sixth feature data to obtain the first repaired image.

[0200] In one possible implementation, the image restoration device restores the second image by decoding the sixth feature data, thereby obtaining the first restored image.

[0201] In this embodiment, the image restoration device, after restoring the feature information carried by the second feature data using the feature information carried by the first feature data to obtain the fourth feature data, fuses the fourth feature data and the fifth feature data to obtain the sixth feature data, and obtains the first restored image based on the sixth feature data. This allows the second image to be restored using the feature information of both the first and third images.

[0202] Since the timestamp of the second image is located between the timestamps of the first and third images, this implementation method can improve the restoration effect of the second image.

[0203] Furthermore, in this embodiment, the image restoration device first uses the feature information carried by the first feature data to restore the feature information carried by the second feature data to obtain the fourth feature data, and then uses the feature information carried by the fifth feature data to restore the feature information carried by the fourth feature data to obtain the sixth feature data. This is equivalent to first fusing the feature information of the first image and the feature information of the second image, and then fusing the feature information of the third image and the feature information of the second image, which is equivalent to performing two fusions.

[0204] Since any two of the first, second, and third images are located at different positions in the spatial dimension, if the feature information of the first, second, and third images is directly fused, the positional differences of the three images in the spatial dimension need to be considered simultaneously, which leads to poor image quality in the repaired second image and poor repair effect.

[0205] This implementation method only needs to consider the positional difference between two images in the spatial dimension during each fusion process. Therefore, compared with directly fusing the feature information of three images, it can improve the restoration effect.

[0206] As an optional implementation, the number of channels for the fifth feature data is s, where s is an integer greater than 1.

[0207] The image restoration device performs the following steps during step 202:

[0208] 301. Select m channels of data from the fifth feature data above to obtain the seventh feature data.

[0209] In this embodiment, m is a positive integer less than s. For example, s is 3, and the three channels of the fifth feature data are channel a, channel b, and channel c, respectively. In the fifth feature data, data belonging to channel a is data d, data belonging to channel b is data e, and data belonging to channel c is data f. If m = 1, then the seventh feature data can be any one of data d, data e, or data f.

[0210] 302. The fourth feature data and the seventh feature data are fused together to obtain the sixth feature data.

[0211] Since the amount of data for the seventh feature data is smaller than that for the fifth feature data, the image restoration device can reduce the amount of data processing required to obtain the sixth feature data by executing steps 301 and 302.

[0212] As an optional implementation, the image restoration device performs the following steps during step 203:

[0213] 401. The sixth feature data above is downsampled to obtain the eighth feature data.

[0214] The image restoration device can extract the feature information of the sixth feature data by downsampling the sixth feature data to obtain the eighth feature data.

[0215] 402. Perform feature extraction processing on the above eighth feature data to obtain the ninth feature data.

[0216] 403. Upsample the above ninth feature data to obtain the tenth feature data.

[0217] In this embodiment of the application, the image restoration device can increase the size of the feature information of the ninth feature data by upsampling the ninth feature data to obtain the tenth feature data, wherein the size of the tenth feature data is the same as the size of the sixth feature data.

[0218] 404. Decode the tenth feature data to obtain the first repaired image.

[0219] In this embodiment, the image restoration device first downsamples the sixth feature data to obtain the eighth feature data, and then performs feature extraction processing on the eighth feature data to obtain the ninth feature data. This can increase the receptive field of the feature extraction processing and thus improve the effect of the feature extraction processing.

[0220] Therefore, by upsampling the ninth feature data to obtain the tenth feature data with the same size as the sixth feature data, and by decoding the tenth feature data, the first restored image is obtained, which can improve the image quality of the first restored image.

[0221] As an optional implementation, the image restoration apparatus, upon obtaining the tenth feature data, further performs the following steps:

[0222] 501. Merge the sixth feature data with the tenth feature data to obtain the eighteenth feature data.

[0223] Since the eighth feature data is obtained by extracting feature information from the sixth feature data, the sixth feature data carries richer low-level feature information than the eighth feature data. This low-level feature information includes at least one of the following: texture information, color information, spatial relationship information, and shape information. Furthermore, the eighth feature data carries richer semantic information than the sixth feature data.

[0224] Therefore, when the image restoration device obtains tenth feature data with the same size as the sixth feature data based on the eighth feature data, it can enrich the low-level feature information in the tenth feature data and obtain the eighteenth feature data by fusing the sixth feature data and the tenth feature data.

[0225] Having obtained the eighteenth feature data, the image restoration device performs the following steps during step 404:

[0226] 502. Decode the eighteenth feature data to obtain the first restored image.

[0227] In this embodiment, the image restoration device, having obtained tenth feature data with the same size as the sixth feature data based on the eighth feature data, can enrich the low-level feature information in the tenth feature data by fusing the sixth and tenth feature data, thus obtaining the eighteenth feature data. Therefore, by decoding the eighteenth feature data to obtain the first restored image, the image quality of the first restored image can be improved, thereby enhancing the restoration effect of the second image.

[0228] As an optional implementation, the image restoration device performs the following steps during step 103:

[0229] 601. The second feature data and at least one third feature data are fused between channels to obtain the eleventh feature data.

[0230] In this embodiment, channel fusion refers to fusing the feature information carried by data from different channels. In one possible implementation, the image restoration apparatus obtains the eleventh feature data by performing pointwise convolution on the second feature data and at least one third feature data.

[0231] Optionally, the image restoration device uses a 1*1 convolution kernel to perform inter-channel convolution on the second feature data and at least one third feature data to obtain the eleventh feature data.

[0232] 602. Perform intra-channel fusion on the above eleventh feature data to obtain the twelfth feature data.

[0233] In this embodiment, in-channel fusion refers to fusing the data from each channel separately, that is, fusing feature information in the aforementioned spatial dimension. In one possible implementation, the image restoration device obtains the twelfth feature data by performing in-channel depthwise convolution on the eleventh feature data.

[0234] Optionally, the image inpainting device uses a k*k convolution kernel to perform intra-channel convolution on the eleventh feature data to obtain the twelfth feature data. k is an odd number greater than 5.

[0235] 603. Based on the twelfth feature data mentioned above, the fourth feature data mentioned above is obtained.

[0236] In one possible implementation, the image restoration device uses the twelfth feature data as the fourth feature data.

[0237] In another possible implementation, the image restoration device obtains the fourth feature data by fusing the twelfth feature data across channels.

[0238] In this embodiment, the image restoration device obtains eleventh feature data by performing inter-channel fusion of second feature data and at least one third feature data, and then performs intra-channel fusion of the eleventh feature data to obtain twelfth feature data. This is equivalent to achieving three-dimensional information fusion through the fusion of two two-dimensional information. In this way, the fusion effect of the second feature data and at least one third feature data can be improved, and the amount of data processing generated by the fusion can be reduced.

[0239] As an optional implementation, the twelfth feature data includes the thirteenth feature data and the fourteenth feature data, where the thirteenth feature data belongs to the first channel of the twelfth feature data and the fourteenth feature data belongs to the second channel of the twelfth feature data.

[0240] The image restoration device performs the following steps during step 603:

[0241] 701. Obtain the first weight of the first channel and the second weight of the second channel.

[0242] In this embodiment, the first weight represents the importance of the information in the first channel, and the second weight represents the importance of the feature information in the second channel. Specifically, the higher the importance of the information in the first channel, the greater the first weight; and the higher the importance of the information in the second channel, the greater the second weight.

[0243] Optionally, if the information in the first channel is more important than the information in the second channel, the first weight is greater than the second weight; if the information in the first channel is less important than the information in the second channel, the first weight is less than the second weight; if the information in the first channel is the same as the information in the second channel, the first weight is equal to the second weight.

[0244] In this embodiment, the higher the importance of the channel information, the better the restoration effect achieved by using the information of that channel to restore image quality.

[0245] In one implementation of obtaining a first weight of a first channel and a second weight of a second channel, the image restoration device receives a first weight and a second weight input by a user through an input component.

[0246] In another implementation of obtaining the first weight of the first channel and the second weight of the second channel, the image restoration device receives the first weight and the second weight sent by the terminal.

[0247] In another implementation of obtaining the second weight of the second channel from the first weight of the first channel, the first weight and the second weight are obtained by training a deep learning model.

[0248] 702. The first weight is used as the fusion weight of the thirteenth feature data, and the second weight is used as the fusion weight of the fourteenth feature data. The thirteenth feature data and the fourteenth feature data are weighted and fused to obtain the fourth feature data.

[0249] In this embodiment, since the first weight represents the importance of the information in the first channel, the second weight represents the importance of the information in the second channel, and the thirteenth feature data is the data of the first channel and the fourteenth feature data is the data of the second channel, the image restoration device performs weighted fusion of the thirteenth feature data and the fourteenth feature data according to the first weight of the first channel and the second weight of the second channel to obtain the fourth feature data, which can improve the fusion effect.

[0250] It should be understood that the first and second channels in the embodiments of this application are descriptive objects determined for the purpose of concisely describing the implementation process of the technical solution, and should not be construed as the twelfth feature data including only the first and second channels. In practical applications, the twelfth feature data may include r channels, where r is a positive integer greater than 1. Moreover, when r is greater than 2, the image restoration device can perform a weighted summation of the data from all channels.

[0251] As an optional implementation, the image restoration device performs the following steps during step 102:

[0252] 801. Obtain at least one offset.

[0253] In this embodiment, the number of offsets is the same as the number of third feature data, and there is a one-to-one correspondence between the offsets and the third feature data. When the number of offsets is greater than 1, any two offsets are different.

[0254] In one implementation of obtaining at least one offset, the image restoration device receives at least one offset input by a user through an input component.

[0255] In another implementation of obtaining at least one offset, the image restoration device receives at least one offset sent by the terminal.

[0256] 802. Move the first feature data within the spatial dimension according to at least one offset to obtain at least one third feature data.

[0257] In this embodiment, the positional difference between the third feature data and the first feature data in the spatial dimension is the aforementioned offset. For example, at least one displacement includes an offset a, and at least one third feature data includes third feature data b. Then, the positional difference between the third feature data b and the first feature data in the spatial dimension is offset a.

[0258] For example, at least one displacement includes offset a and offset b, and at least one third feature data includes third feature data c and third feature data d. Third feature data c is obtained by moving the first feature data within a spatial dimension according to offset a, and third feature data d is obtained by moving the first feature data within a spatial dimension according to offset b. Then, the positional difference between the third feature data c and the first feature data in the spatial dimension is offset a, and the positional difference between the third feature data c and the first feature data in the spatial dimension is offset b.

[0259] In this embodiment, the image restoration device moves the first feature data in the spatial dimension according to at least one offset to obtain at least one third feature data.

[0260] As an optional implementation, the image restoration apparatus also performs the following steps:

[0261] 901. Replace the second image in the video to be repaired with the first repaired image described above.

[0262] The image restoration device replaces the second image in the video to be restored with a first restoration image to obtain a restored video, which can improve the video quality of the video to be restored.

[0263] Based on the technical solutions provided in the embodiments of this application, the embodiments of this application also provide an image restoration network, which can realize the technical solutions described above.

[0264] The image restoration network includes a feature extraction module, a feature fusion module, and a decoding module. The feature extraction module extracts feature information from the image to obtain feature data. Specifically, the feature extraction module extracts feature information from a first image to obtain first feature data, and then extracts feature information from a second image to obtain second feature data.

[0265] The first and second feature data are processed by the feature fusion module to obtain the fourth feature data. Optionally, steps 102 and 103 described above can both be implemented by the feature fusion module, that is, the feature fusion module obtains at least one third feature data based on the first feature data, and the feature fusion module fuses the second feature data with at least one third feature data to obtain the fourth feature data.

[0266] Step 104 can be implemented through a feature fusion module and a decoding module. Specifically, when the implementation process of step 104 includes decoding processing, the decoding processing is implemented through the decoding module. For example, in the implementation process of step 104, the image restoration device obtains a first restored image by decoding the fourth feature data. In this case, the decoding processing of the fourth feature data can be performed through the decoding module. As another example, in the implementation process of step 104, the image restoration device obtains a first restored image by decoding the sixth feature data. In this case, the decoding processing of the sixth feature data can be performed through the decoding module.

[0267] When the implementation process of step 104 includes processing other than decoding, the processing other than decoding is implemented by the feature fusion module. For example, when the image restoration device executes steps 401 to 404 during the execution of step 203, steps 401 to 403 are all processing other than decoding, and step 404 is decoding. That is, steps 401 to 403 are all implemented by the feature fusion module, and step 404 is implemented by the decoding module.

[0268] For example, when the image restoration device obtains the first restored image by executing steps 501 and 502, step 501 is a process other than decoding, and step 502 is a decoding process. That is, step 501 is implemented by the feature fusion module, and step 502 is implemented by the decoding module.

[0269] In one possible implementation, the network structure of the image inpainting network is as follows: Figure 3 As shown. Figure 3As shown, the (i-1)th frame, the i-th frame, and the (i+1)th frame are three adjacent frames in the video to be repaired. The feature extraction module can extract the feature information of these three frames respectively, and obtain the feature data of the (i-1)th frame, the i-th frame, and the (i+1)th frame.

[0270] The feature fusion module processes the feature data of the (i-1)th frame, the i-th frame, and the (i+1)th frame to obtain the repaired feature data of the (i-1)th frame, the i-th frame, and the (i+1)th frame.

[0271] The (i-1)th frame image is concatenated with its repaired feature data to obtain the feature data to be decoded for the (i-1)th frame image. The (i-1)th frame image is then concatenated with its repaired feature data to obtain the feature data to be decoded for the i-th frame image. Finally, the (i+1)th frame image is concatenated with its repaired feature data to obtain the feature data to be decoded for the i+1th frame image.

[0272] The decoding module decodes the feature data of the (i-1)th frame to obtain the repaired (i-1)th frame image. The decoding module then decodes the feature data of the ith frame to obtain the repaired ith frame image. Finally, the decoding module decodes the feature data of the (i+1)th frame to obtain the repaired (i+1)th frame image.

[0273] As an optional implementation, the feature extraction module includes N feature extraction units, where N is a positive integer. The structure of the feature extraction unit is as follows: Figure 4 As shown, Figure 4 As shown, the feature extraction unit includes a residual module (ResBlock), a downsampling module (downsampling), an upsampling module (upsampling), and a skip connection (skip).

[0274] As an optional implementation, grouped spatial-temporal shift blocks (GSTSB) are used as feature fusion modules, wherein the structure of the GSTSB is as follows: Figure 5 As shown. Figure 5 As shown, fi -1 f is the feature data of the (i-1)th frame image. i f is the feature data of the i-th frame image. i+1 Let f be the feature data of the (i+1)th frame image. During the forward temporal fusion process, f... i-1 f i fi+1 Divided into two parts, we get This is one of the two parts of the feature data of the (i-2)th frame image.

[0275] Based on at least one offset Moving in spatial dimensions yields at least one Will and Concatenate the (i-1)th frame to obtain the forward concatenation feature. Combine the forward concatenation feature of the (i-1)th frame with at least one... The data are input together into a large convolutional kernel, and after processing by the large convolutional kernel, the forward fused feature data of the (i-1)th frame is obtained (i.e., ).

[0276] In this embodiment, the large convolutional kernel sequentially performs inter-channel fusion, intra-channel fusion, and weighted fusion on the input data. For details, please refer to steps 601 to 603 and steps 701 to 702.

[0277] The forward fusion feature data of the i-th frame can be obtained in the same way (i.e. ), the forward fusion feature data of the (i+1)th frame (i.e. (This will not be elaborated upon here.)

[0278] In the process of reverse timing fusion, Divided into two parts, we get This is one of the two parts obtained by dividing the forward fusion feature data of the (i+2)th frame into two parts.

[0279] Based on at least one offset Moving in spatial dimensions yields at least one Will and The reverse splicing feature of the (i-1)th frame is obtained by splicing. The reverse splicing feature of the (i-1)th frame is then combined with at least one... The images are input together into a large convolutional kernel, and the repaired feature data of the (i-1)th frame is obtained after processing by the large convolutional kernel. The repaired feature data of the i-th frame and the i+1th frame can be obtained in the same way, which will not be described in detail here.

[0280] In this implementation, by fusing the feature information of the three frames through forward and reverse temporal fusion, the amount of data processing required to fuse the feature information of the three frames can be reduced. Therefore, based on this implementation, at least two frames can be repaired in parallel, thereby improving the repair efficiency.

[0281] As an optional implementation, the training process of the image inpainting network includes:

[0282] 1001. Obtain training videos.

[0283] In this embodiment of the application, the training video includes a fourth image and a fifth image, wherein the difference between the timestamp of the fourth image and the timestamp of the fifth image is less than or equal to a time threshold.

[0284] In one implementation of acquiring training videos, the image inpainting device receives training videos input by a user through an input component.

[0285] In another implementation of acquiring training videos, the image inpainting device receives training videos sent by the terminal.

[0286] 1002. Use the fourth image to repair the fifth image to obtain the second repaired image.

[0287] In one possible implementation, feature information from the fourth image is extracted by a feature extraction module to obtain the fifteenth feature data. Feature information from the fifth image is then extracted by the same module to obtain the sixteenth feature data. The fifteenth and sixteenth feature data are processed by a feature fusion module to obtain the seventeenth feature data. The decoding module then uses the seventeenth feature data to obtain the second restored image of the fifth image.

[0288] In this embodiment, the seventeenth feature data is the repaired feature data of the fifth image. That is, the fifteenth and sixteenth feature data are processed by the feature fusion module, and the feature information carried by the fifteenth feature data is used to repair the sixteenth feature data to obtain the seventeenth feature data.

[0289] 1003. Based on the first difference between the second repaired image and the sixth image, the first loss is obtained.

[0290] In this embodiment, the sixth image is the ground truth of the fifth image. Specifically, the image content of the sixth image is the same as that of the fifth image, and the image quality of the sixth image is higher than that of the fifth image.

[0291] In this embodiment of the application, the first loss is positively correlated with the first difference.

[0292] 1004. Based on the first loss mentioned above, update the parameters of the image restoration network.

[0293] In one possible implementation, the image inpainting device updates the parameters of the image inpainting network based on the first loss until the total loss converges, thus completing the inpainting of the image inpainting network.

[0294] As an optional implementation, the image restoration apparatus also performs the following steps:

[0295] 1101. Decode the sixteenth feature data mentioned above to obtain a decoded image.

[0296] 1102. Based on the second difference between the brightness of the decoded image and the brightness of the sixth image, the second loss is obtained.

[0297] The second difference is obtained by calculating the difference between the brightness of the decoded image and the brightness of the sixth image, and then the second loss can be obtained based on the second difference, wherein the second loss is positively correlated with the second difference.

[0298] The presence of degradation factors introduces errors into the structural information of an image, which in turn reduces the accuracy of image alignment. Therefore, the presence of degradation factors in an image leads to decreased image alignment accuracy, resulting in a poorer restoration effect when image quality is restored by fusing image information.

[0299] In the process of obtaining the sixteenth feature data by extracting feature information from the fifth image, the extracted feature information may include feature information of degradation factors, which may lead to the sixteenth feature data carrying feature information of degradation factors. As a result, the second restored image of the fifth image obtained based on the sixteenth feature data contains degradation factors, thereby reducing the restoration effect.

[0300] Since the sixth image is the true value of the fifth image, it has fewer degradation factors compared to the fifth image. Therefore, the sixth image can be used as a basis to determine whether the sixteenth feature data carries degradation factors.

[0301] Considering that degradation factors can alter image brightness, after decoding the sixteenth feature data to obtain a decoded image, and using the second loss derived from the second difference between the brightness of the decoded image and the sixth image, the feature extraction module can be supervised to determine whether to extract degradation factor features when extracting feature information from the fifth image. Therefore, updating the parameters of the network to be repaired based on the second loss reduces the amount of degradation factor features extracted by the feature extraction module.

[0302] In the event of a second loss, the image restoration device performs the following steps during step 1004:

[0303] 1103. Update the parameters of the image restoration network based on the first loss and the second loss described above.

[0304] In one possible implementation, the image inpainting device calculates the sum of a first loss and a second loss to obtain the total loss of the image inpainting network. The parameters of the image inpainting network are updated based on the total loss until the total loss converges, thus completing the inpainting of the image inpainting network.

[0305] In another possible implementation, the image inpainting device performs a weighted sum of the first and second losses to obtain the total loss of the image inpainting network. The parameters of the image inpainting network are then updated based on the total loss until the total loss converges, completing the inpainting of the image inpainting network.

[0306] In another possible implementation, the image inpainting device calculates the sum of the first loss and the second loss to obtain the total loss of the image inpainting network. The parameters of the image inpainting network are updated based on the total loss until both the total loss and the first loss converge, thus completing the inpainting of the image inpainting network.

[0307] In this embodiment, during the training process, the image inpainting device processes the fifteenth and sixteenth feature data through a feature fusion module to obtain the seventeenth feature data. The seventeenth feature data is then processed by a decoding module to obtain the second inpainted image of the fifth image. A first loss is obtained based on the first difference between the second inpainted image and the sixth image.

[0308] The feature extraction module extracts feature information from the fourth and fifth images respectively, obtaining the fifteenth and sixteenth feature data. The sixteenth feature data is then decoded to obtain a decoded image. A second loss is derived based on the second difference in brightness between the decoded image and the sixth image. The parameters of the network to be repaired are updated based on this second loss, reducing the feature information of degradation factors extracted by the feature extraction module. Therefore, updating the parameters of the network to be repaired based on the first and second losses improves the repair effect.

[0309] It should be understood that the fourth and fifth images in the training video are descriptive objects determined for the concise description of the training process of the network to be repaired. It should not be interpreted as meaning that during the training of the network to be repaired, the first and second losses are obtained solely by processing the fourth and fifth images in the training video, and then the parameters of the network to be repaired are updated based on the first and second losses. In actual training, the same processing as that performed on the fourth and fifth images can be performed on any two images in the training video.

[0310] As an optional implementation, this application also provides a training framework for an image inpainting network. Please refer to... Figure 6 , Figure 6 The diagram shown is a schematic of a training framework for an image inpainting network provided in an embodiment of this application.

[0311] like Figure 6 As shown, during the training process of the image inpainting network, the feature extraction module extracts the feature information of each frame of the training video through N feature extraction units to obtain the feature data of each frame. The structure of the feature extraction unit can be found in [reference needed]. Figure 4 .

[0312] The feature data of each frame image is input into the feature fusion module. The feature fusion module processes the feature data of each frame image through N feature fusion units to obtain the repaired feature data of each frame image.

[0313] The layout of the feature fusion unit is the same as that of the feature extraction unit. The difference lies in the fact that the residual unit of the feature fusion unit is the aforementioned GSTSB, that is, by... Figure 4 Replacing the residual module in the feature extraction unit structure shown with GSTSB yields the following result: Figure 6 The feature fusion unit in. It should be understood that, in Figure 6 In this context, the parameters of the feature extraction unit are different from those of the feature fusion unit.

[0314] The second loss for each frame is obtained by applying additional supervision to the feature data of each frame. Specifically, the feature data of each frame is decoded to obtain the decoded image of each frame. The second loss for each frame is obtained based on the second difference between the brightness of the decoded image and the brightness of the ground truth image.

[0315] The restored feature data of each frame is decoded by the decoding module to obtain the restored image of each frame. The first loss of each frame is obtained based on the first difference in brightness between the restored image and the true value of each frame.

[0316] Finally, based on the first loss and the second loss of each frame, the parameters of the network to be repaired are updated, and the training of the network to be repaired is completed.

[0317] Based on the technical solutions provided in the embodiments of this application, the embodiments of this application also provide several possible application scenarios.

[0318] Scenario A: With the increasing prevalence of mobile phone video recording, various factors contribute to degradation in these videos, resulting in low quality. Current image restoration technologies require significant hardware resources, which, coupled with the limited resources of mobile phones, lead to slow processing speeds for restoring video quality using current image restoration techniques.

[0319] Thanks to the small amount of data processed and the fast processing speed, the technical solution provided by this application embodiment can not only improve the image restoration effect, but also improve the restoration speed.

[0320] In one possible scenario, a user uses their phone to film a basketball game at night. Due to the low light levels, the video contains a lot of noise. Furthermore, because both the basketball players and the ball are in motion during a game, the video is severely blurry.

[0321] The mobile phone can repair the captured video based on the technical solution provided in the embodiments of this application, which can remove noise and blur in the video and improve the video quality.

[0322] Scenario B: As dashcams become increasingly popular, car owners are using them to record videos more frequently. Since dashcams need to record video in all weather conditions, rain blurring occurs in rainy weather recordings. Rain blurring includes at least one of the following: raindrops and rain streaks. Therefore, dashcam videos recorded in rainy weather will exhibit rain blurring.

[0323] Clearly, rain blurring is a type of degradation factor mentioned above; its presence causes video to become blurry, thus reducing video quality. Based on the technical solution provided in this application, the dashcam can remove rain blurring from the captured video, improving video clarity.

[0324] Scenario C: With the popularity of extreme sports, the use of action cameras (GoPro) is also increasing. Since the subjects shot by action cameras are usually in motion, motion blur is prone to appear in the videos shot by action cameras, resulting in blurry videos.

[0325] Based on the technical solutions provided in the embodiments of this application, the blur caused by motion in the video can be removed, thereby improving the clarity of the video captured by the action camera.

[0326] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0327] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, while using clear signs / information to inform users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0328] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0329] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an image restoration device 1 provided in an embodiment of this application. The image restoration device 1 includes: an acquisition unit 11, a first processing unit 12, a second processing unit 13, and a third processing unit 14. Optionally, the image restoration device 1 further includes a training unit 15. Specifically:

[0330] The acquisition unit 11 is used to acquire first feature data of the first image and second feature data of the second image; both the first image and the second image belong to the video to be repaired, and the difference between the timestamp of the first image and the timestamp of the second image is less than or equal to a time threshold.

[0331] The first processing unit 12 is configured to obtain at least one third feature data based on the first feature data; the position of the at least one third feature data in the spatial dimension is different from the position of the first feature data in the spatial dimension.

[0332] The second processing unit 13 is used to fuse the second feature data with the at least one third feature data to obtain the fourth feature data;

[0333] The third processing unit 14 is used to repair the second image according to the fourth feature data to obtain the first repaired image.

[0334] In any embodiment of this application, the video to be repaired further includes a third image, wherein the timestamp of the second image is located between the timestamp of the first image and the timestamp of the third image;

[0335] The acquisition unit 11 is also used to acquire the fifth feature data of the third image;

[0336] The second processing unit 13 is further configured to fuse the fourth feature data and the fifth feature data to obtain the sixth feature data;

[0337] The third processing unit 14 is used to repair the second image based on the sixth feature data to obtain the first repaired image.

[0338] In any embodiment of this application, the number of channels of the fifth feature data is s, where s is a positive number greater than 1;

[0339] The second processing unit 13 is used for:

[0340] The seventh feature data is obtained by selecting data from m channels of the fifth feature data, where m is less than s;

[0341] The fourth feature data and the seventh feature data are fused to obtain the sixth feature data.

[0342] In conjunction with any embodiment of this application, the third processing unit 14 is configured to:

[0343] The sixth feature data is downsampled to obtain the eighth feature data;

[0344] The eighth feature data is subjected to feature extraction processing to obtain the ninth feature data;

[0345] The ninth feature data is upsampled to obtain the tenth feature data; the size of the tenth feature data is the same as the size of the sixth feature data.

[0346] The tenth feature data is decoded to obtain the first repaired image.

[0347] In any embodiment of this application, the second processing unit 13 is configured to:

[0348] The second feature data and the at least one third feature data are fused between channels to obtain the eleventh feature data;

[0349] The eleventh feature data is fused within a channel to obtain the twelfth feature data;

[0350] The fourth feature data is obtained based on the twelfth feature data.

[0351] In any embodiment of this application, the twelfth feature data includes a thirteenth feature data and a fourteenth feature data, wherein the thirteenth feature data belongs to a first channel of the twelfth feature data and the fourteenth feature data belongs to a second channel of the twelfth feature data;

[0352] The second processing unit 13 is used for:

[0353] Obtain the first weight of the first channel and the second weight of the second channel;

[0354] The first weight is used as the fusion weight of the thirteenth feature data, and the second weight is used as the fusion weight of the fourteenth feature data. The thirteenth feature data and the fourteenth feature data are weighted and fused to obtain the fourth feature data.

[0355] In any embodiment of this application, the first processing unit 12 is configured to:

[0356] Obtain at least one offset;

[0357] The first feature data is moved within the spatial dimension according to the at least one offset to obtain the at least one third feature data.

[0358] In any embodiment of this application, the third processing unit 14 is further configured to replace the second image in the video to be repaired with the first repaired image.

[0359] In any embodiment of this application, the image restoration method is implemented through an image restoration network, which includes a feature extraction module, a feature fusion module, and a decoding module.

[0360] The first processing unit 12 is used to obtain at least one third feature data based on the first feature data through the feature fusion module;

[0361] The second processing unit 13 is used to fuse the second feature data with the at least one third feature data through the feature fusion module to obtain the fourth feature data;

[0362] The third processing unit 14 is used to obtain the first repaired image of the second image based on the fourth feature data through the feature fusion module and the decoding module;

[0363] The acquisition unit 11 is used to extract feature information of the first image through the feature extraction module to obtain the first feature data;

[0364] The acquisition unit 11 is further configured to extract feature information of the second image through the feature extraction module to obtain the second feature data.

[0365] In conjunction with any embodiment of this application, the image restoration method performed by the image processing apparatus further includes a training process for the image restoration network, and the image processing apparatus further includes a training unit 15, the training unit 15 being used for:

[0366] Acquire a training video, which includes a fourth image and a fifth image, wherein the difference between the timestamp of the fourth image and the timestamp of the fifth image is less than or equal to the time threshold.

[0367] The fourth image is used to repair the fifth image to obtain the second repaired image;

[0368] A first loss is obtained based on the first difference between the second repaired image and the sixth image; the sixth image is the true value of the fifth image; the first loss is positively correlated with the first difference.

[0369] The parameters of the image inpainting network are updated based on the first loss.

[0370] In any embodiment of this application, the training unit 15 is used for:

[0371] The feature extraction module extracts the feature information of the fourth image to obtain the fifteenth feature data;

[0372] The feature extraction module extracts the feature information of the fifth image to obtain the sixteenth feature data;

[0373] The feature fusion module processes the fifteenth and sixteenth feature data to obtain the seventeenth feature data, which is the restored feature data of the fifth image.

[0374] The second repaired image is obtained by the decoding module based on the seventeenth feature data.

[0375] In conjunction with any embodiment of this application, the training unit 15 is further configured to:

[0376] The sixteenth feature data is decoded to obtain a decoded image;

[0377] A second loss is obtained based on the second difference between the brightness of the decoded image and the brightness of the sixth image; the second loss is positively correlated with the second difference.

[0378] The parameters of the image inpainting network are updated based on the first loss and the second loss.

[0379] In this embodiment of the application, when the image restoration device acquires first feature data and second feature data of a first image, and the first image and the second image are two images with different timestamps in the video to be restored, it obtains at least one third feature data that is different from the first feature data in spatial dimension based on the first feature data, thereby increasing the probability that at least one third feature data includes feature data that is in the same spatial dimension as the second feature data.

[0380] In this way, by fusing the second feature data with at least one third feature data to obtain the fourth feature data, the effect of using the feature information carried by the first feature data to repair the feature information carried by the second feature data can be improved, thereby improving the image quality of the second image using the feature information of the first image.

[0381] Therefore, the second image is repaired based on the fourth feature data to obtain the first repaired image. Improving the image quality of the first repaired image can improve the repair effect on the second image, in other words, it can make the second image clearer.

[0382] In this embodiment, the acquisition unit 11 can be a data interface, and the first processing unit 12, the second processing unit 13, the third processing unit 14 and the training unit 15 can all be graphics processors.

[0383] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0384] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are coupled together via connectors, which include various interfaces, transmission lines, or buses, etc., and are not limited in this embodiment. It should be understood that in the various embodiments of this application, coupling refers to mutual connection in a specific way, including direct connection or indirect connection through other devices, such as through various interfaces, transmission lines, buses, etc.

[0385] Processor 21 can be one or more graphics processing units (GPUs). If processor 21 is a GPU, the GPU can be a single-core GPU or a multi-core GPU. Optionally, processor 21 can be a processor group composed of multiple GPUs, with the multiple processors coupled to each other via one or more buses. Optionally, the processor can also be other types of processors, etc., which are not limited in this embodiment.

[0386] The memory 22 can be used to store computer program instructions, as well as various types of computer program code, including program code for executing the scheme of this application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0387] Input device 23 is used to input data and / or signals, and output device 24 is used to output data and / or signals. Input device 23 and output device 24 can be independent devices or an integrated device.

[0388] It is understood that in this embodiment of the application, the memory 22 can be used not only to store related instructions, but also to store related data. For example, the memory 22 can be used to store first feature data and second feature data obtained through the input device 23, or the memory 22 can also be used to store data obtained through the processor 21, etc. This embodiment of the application does not limit the specific data stored in the memory.

[0389] Understandable, Figure 8 This is merely a simplified design of an electronic device. In practical applications, the image restoration apparatus may also include other necessary components, including, but not limited to, any number of input / output devices, processors, memories, etc., and all image restoration apparatuses that can implement the embodiments of this application are within the protection scope of this application.

[0390] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0391] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will also readily understand that the various embodiments of this application have different focuses, and for the sake of convenience and brevity, the same or similar parts may not be repeated in different embodiments. Therefore, parts not described or not described in detail in one embodiment can be referred to the descriptions in other embodiments.

[0392] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0393] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0394] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0395] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0396] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An image inpainting method characterized by, The method includes: Obtain first feature data of the first image and second feature data of the second image; both the first image and the second image belong to the video to be repaired, and the difference between the timestamp of the first image and the timestamp of the second image is less than or equal to a time threshold. Based on the first feature data, multiple third feature data are obtained; the positions of the multiple third feature data in the spatial dimension are all different from the positions of the first feature data in the spatial dimension; obtaining multiple third feature data based on the first feature data includes: acquiring multiple offsets, each of the offsets being different from the others; moving the first feature data in the spatial dimension based on the multiple offsets to obtain the multiple third feature data; The second feature data is fused with the plurality of third feature data to obtain the fourth feature data; The second image is repaired based on the fourth feature data to obtain the first repaired image.

2. The method according to claim 1, characterized in that, The video to be repaired also includes a third image, and the timestamp of the second image is between the timestamp of the first image and the timestamp of the third image; Before repairing the second image based on the fourth feature data to obtain the first repaired image, the method further includes: Obtain the fifth feature data of the third image; The fourth feature data and the fifth feature data are fused to obtain the sixth feature data; The step of repairing the second image based on the fourth feature data to obtain the first repaired image includes: The second image is repaired based on the sixth feature data to obtain the first repaired image.

3. The method according to claim 2, characterized in that, The number of channels for the fifth feature data is s, where s is a positive number greater than 1; The process of fusing the fourth feature data and the fifth feature data to obtain the sixth feature data includes: The seventh feature data is obtained by selecting data from m channels of the fifth feature data, where m is less than s; The fourth feature data and the seventh feature data are fused to obtain the sixth feature data.

4. The method according to claim 2 or 3, characterized in that, The step of repairing the second image based on the sixth feature data to obtain the first repaired image includes: The sixth feature data is downsampled to obtain the eighth feature data; The eighth feature data is subjected to feature extraction processing to obtain the ninth feature data; The ninth feature data is upsampled to obtain the tenth feature data; the size of the tenth feature data is the same as the size of the sixth feature data. The tenth feature data is decoded to obtain the first repaired image.

5. The method according to any one of claims 1 to 3, characterized in that, The step of fusing the second feature data with the plurality of third feature data to obtain the fourth feature data includes: The second feature data and the plurality of third feature data are fused across channels to obtain the eleventh feature data; The eleventh feature data is fused within a channel to obtain the twelfth feature data; The fourth feature data is obtained based on the twelfth feature data.

6. The method according to claim 5, characterized in that, The twelfth feature data includes the thirteenth feature data and the fourteenth feature data. The thirteenth feature data belongs to the first channel of the twelfth feature data, and the fourteenth feature data belongs to the second channel of the twelfth feature data. The process of obtaining the fourth feature data based on the twelfth feature data includes: Obtain the first weight of the first channel and the second weight of the second channel; The first weight is used as the fusion weight of the thirteenth feature data, and the second weight is used as the fusion weight of the fourteenth feature data. The thirteenth feature data and the fourteenth feature data are weighted and fused to obtain the fourth feature data.

7. The method according to any one of claims 1 to 3, characterized in that, After obtaining the first repaired image, the method further includes: Replace the second image in the video to be repaired with the first repaired image.

8. The method according to any one of claims 1 to 3, characterized in that, The image restoration method is implemented through an image restoration network, which includes a feature extraction module, a feature fusion module, and a decoding module. The process of obtaining multiple third feature data based on the first feature data includes: The feature fusion module obtains at least one third feature based on the first feature data; The step of fusing the second feature data with the plurality of third feature data to obtain the fourth feature data includes: The second feature data is fused with the plurality of third feature data by the feature fusion module to obtain the fourth feature data; The step of repairing the second image based on the fourth feature data to obtain the first repaired image includes: The feature fusion module and the decoding module obtain the first repaired image of the second image based on the fourth feature data; The acquisition of the first feature data of the first image and the second feature data of the second image includes: The feature extraction module extracts feature information from the first image to obtain the first feature data; The feature extraction module extracts the feature information of the second image to obtain the second feature data.

9. The method according to claim 8, characterized in that, The image restoration method also includes the training process of the image restoration network: Acquire a training video, which includes a fourth image and a fifth image, wherein the difference between the timestamp of the fourth image and the timestamp of the fifth image is less than or equal to the time threshold. The fourth image is used to repair the fifth image to obtain the second repaired image; The first loss is obtained based on the first difference between the second repaired image and the sixth image; The sixth image is the true value of the fifth image; the first loss is positively correlated with the first difference; The parameters of the image inpainting network are updated based on the first loss.

10. The method according to claim 9, characterized in that, The step of repairing the fifth image using the fourth image to obtain the second repaired image includes: The feature extraction module extracts the feature information of the fourth image to obtain the fifteenth feature data; The feature extraction module extracts the feature information of the fifth image to obtain the sixteenth feature data; The feature fusion module processes the fifteenth and sixteenth feature data to obtain the seventeenth feature data, which is the restored feature data of the fifth image. The second repaired image is obtained by the decoding module based on the seventeenth feature data.

11. The method according to claim 10, characterized in that, Before updating the parameters of the image inpainting network based on the first loss, the training process further includes: The sixteenth feature data is decoded to obtain a decoded image; A second loss is obtained based on the second difference between the brightness of the decoded image and the brightness of the sixth image; the second loss is positively correlated with the second difference. The step of updating the parameters of the image inpainting network based on the first loss includes: The parameters of the image inpainting network are updated based on the first loss and the second loss.

12. An image restoration device, characterized in that, The device includes: The acquisition unit is used to acquire first feature data of a first image and second feature data of a second image; both the first image and the second image belong to the video to be repaired, and the difference between the timestamp of the first image and the timestamp of the second image is less than or equal to a time threshold. A first processing unit is configured to obtain multiple third feature data based on the first feature data; the positions of the multiple third feature data in the spatial dimension are all different from the positions of the first feature data in the spatial dimension; obtaining multiple third feature data based on the first feature data includes: acquiring multiple offsets, each of the offsets being different from the others; moving the first feature data within the spatial dimension based on the multiple offsets to obtain the multiple third feature data. The second processing unit is used to fuse the second feature data with the plurality of third feature data to obtain the fourth feature data; The third processing unit is used to repair the second image based on the fourth feature data to obtain the first repaired image.

13. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 11.

Citation Information

Patent Citations

  • Variable-length input super-resolution video reconstruction method based on deep learning

    CN111524068A

  • Video super-resolution processing method and device, and storage medium

    CN112700392A