Image processing method, device, electronic device and storage medium

Abstract

The present disclosure provides an image processing method, device, electronic device and storage medium, which belongs to the field of multimedia technology. The method includes: obtaining multiple reference image frames of a target image frame from a target video, wherein the number of frames between the reference image frame and the target image frame is not greater than the number of target frames; based on a three-dimensional convolution kernel, sliding symmetrically frame by frame on the multiple reference image frames with the target image frame as the center, to obtain image fusion features; based on the image fusion features, determining a high-resolution image corresponding to the target image frame. The above scheme can expand the receptive field of the three-dimensional convolution kernel by sliding the three-dimensional convolution kernel symmetrically frame by frame on multiple reference image frames with the target image frame as the center, and can process the target image frame and the reference image frame in each convolution, making full use of the influence of the reference image frame on the target image frame, so that the high-resolution image obtained based on the image fusion feature is clearer.

Description

Technical Field

[0001] The present disclosure relates to the field of multimedia technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Video super-resolution technology is an important technology in video enhancement. The core of this technology is to use the information contained in the image frame related to the target image frame in the video to fill the target image frame, and obtain a high-resolution image corresponding to the target image frame. How to obtain clear high-resolution images has become the focus of research in the industry.

[0003] In the related art, a target image frame and related image frames are usually processed in sequence by a three-dimensional convolution kernel, and then a high-resolution image of the target image frame is restored based on multiple convolution results.

[0004] However, in the above technical solution, since the receptive field of the three-dimensional convolution kernel is small, when processing an image frame that is far away from the target image frame, the convolution result does not include the characteristics of the target image frame, that is, the influence of the related image frame on the target image frame is not utilized, resulting in the high-resolution image having low clarity. Summary of the invention

[0005] The present disclosure provides an image processing method, device, electronic device and storage medium, which can fully utilize the influence of reference image frames on target image frames, so that the high-resolution image obtained based on the image fusion feature is clearer. The technical solution of the present disclosure is as follows:

[0006] According to one aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0007] Acquire multiple reference image frames of the target image frame from the target video, wherein the number of frames between the reference image frames and the target image frame is not greater than the number of target frames;

[0008] Based on the three-dimensional convolution kernel, the target image frame is centered and symmetrically slides on the multiple reference image frames frame by frame to obtain image fusion features;

[0009] Based on the image fusion feature, a high-resolution image corresponding to the target image frame is determined.

[0010] According to another aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0011] An acquisition unit is configured to acquire a plurality of reference image frames of a target image frame from a target video, wherein the number of frames between the reference image frames and the target image frame is not greater than the number of target frames;

[0012] A fusion unit is configured to perform symmetrical sliding on the multiple reference image frames frame by frame based on a three-dimensional convolution kernel with the target image frame as the center to obtain an image fusion feature;

[0013] The determination unit is configured to determine the high-resolution image corresponding to the target image frame based on the image fusion feature.

[0014] In some embodiments, the three-dimensional convolution kernel includes a first convolution kernel, a fixed convolution kernel, and a second convolution kernel, and the first convolution kernel and the second convolution kernel are used to slide symmetrically frame by frame on the multiple reference image frames with the fixed convolution kernel as the center, and the number of sliding times is equal to the target frame number;

[0015] The fusion unit comprises:

[0016] A first determining subunit is configured to execute a sliding behavior in response to the three-dimensional convolution kernel, and determine a first reference image frame and a second reference image frame corresponding to the sliding behavior, wherein the first reference image frame and the second reference image frame are symmetrical with the target image frame as the center;

[0017] a processing subunit, configured to process the target image frame based on the fixed convolution kernel, process the first reference image frame of the sliding behavior based on the first convolution kernel, and process the second reference image frame of the sliding behavior based on the second convolution kernel to obtain a first intermediate fusion feature;

[0018] The second determining subunit is configured to execute a first intermediate fusion feature and a second intermediate fusion feature corresponding to multiple sliding behaviors to determine the image fusion feature, wherein the second intermediate fusion feature is obtained based on the three-dimensional convolution kernel when the three-dimensional convolution kernel does not start to slide.

[0019] In some embodiments, the processing subunit is configured to process the target image frame based on the fixed convolution kernel to obtain a first intermediate feature; process the first reference image frame based on the first convolution kernel to obtain a second intermediate feature; process the second reference image frame based on the second convolution kernel to obtain a third intermediate feature; and perform weighted summation of the first intermediate feature, the second intermediate feature and the third intermediate feature to obtain the first intermediate fusion feature.

[0020] In some embodiments, the image processing apparatus further comprises:

[0021] a processing unit configured to process the target image frame based on the fixed convolution kernel to obtain a fourth intermediate feature when the three-dimensional convolution kernel does not start to slide;

[0022] The processing unit is further configured to process a third reference image frame based on the first convolution kernel to obtain a fifth intermediate feature, wherein the third reference image frame is a reference image frame corresponding to when the first convolution kernel does not start to slide;

[0023] The processing unit is further configured to execute the processing unit, and is further configured to process the fourth reference image frame based on the second convolution kernel to obtain a sixth intermediate feature, wherein the fourth reference image frame is a reference image frame corresponding to when the second convolution kernel does not start to slide, and the third reference image frame and the fourth reference image frame are symmetrical with the target image frame as the center;

[0024] The fusion unit is further configured to perform weighted summation on the fourth intermediate feature, the fifth intermediate feature and the sixth intermediate feature to obtain the second intermediate fusion feature.

[0025] In some embodiments, the first reference image frame is at least one reference image frame, the second reference image frame is at least one reference image frame, the third reference image frame is at least one reference image frame, the fourth reference image frame is at least one reference image frame, and the number of the first reference image frames is equal to the number of the third reference image frames.

[0026] In some embodiments, the acquisition unit is configured to acquire reference image frames frame by frame from both sides of the target image frame with the target image frame as the center; in response to the number of reference image frames acquired on the first side being greater than the number of reference image frames acquired on the second side, the reference image frame located at a first position on the first side is copied to obtain a reference image frame located at a second position on the second side, and the first position and the second position are symmetrical with the position of the target image frame as the center.

[0027] In some embodiments, the acquisition unit is configured to acquire reference image frames frame by frame from both sides of the target image frame with the target image frame as the center; in response to the number of acquired reference image frames being less than the target number, the target image frame is copied to obtain the missing reference image frames.

[0028] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0029] one or more processors;

[0030] A memory for storing program codes executable by the processor;

[0031] The processor is configured to execute the program code to implement the above-mentioned image processing method.

[0032] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When a program code in the computer-readable storage medium is executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned image processing method.

[0033] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction, which implements the above-mentioned image processing method when executed by a processor.

[0034] The image processing solution provided by the disclosed embodiment can expand the receptive field of the three-dimensional convolution kernel by symmetrically sliding the three-dimensional convolution kernel frame by frame on multiple reference image frames with the target image frame as the center, and can process the target image frame and the reference image frame in each convolution, thereby making full use of the influence of the reference image frame on the target image frame, so that the high-resolution image obtained based on the image fusion feature is clearer.

[0035] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0037] Figure 1 It is a schematic diagram of an implementation environment of an image processing method according to an exemplary embodiment.

[0038] Figure 2 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0039] Figure 3 is a flowchart of another image processing method according to an exemplary embodiment.

[0040] Figure 4 The diagram is a schematic diagram showing a method of acquiring a reference image frame according to an exemplary embodiment.

[0041] Figure 5 It is a schematic diagram showing a method of obtaining a second intermediate fusion feature according to an exemplary embodiment.

[0042] Figure 6 It is a schematic diagram of obtaining a first intermediate fusion feature after sliding a three-dimensional convolution kernel for the first time according to an exemplary embodiment.

[0043] Figure 7 It is a schematic diagram showing a sliding three-dimensional convolution kernel for acquiring image fusion features according to an exemplary embodiment.

[0044] Figure 8 The figure is a rendering of a wheel reconstruction according to an exemplary embodiment.

[0045] Fig. 9 The figure is a rendering of a face reconstruction according to an exemplary embodiment.

[0046] Fig.10 The figure is a schematic diagram showing the structure of an image processing device according to an exemplary embodiment.

[0047] Fig.11 It is a schematic structural diagram of another image processing device according to an exemplary embodiment.

[0048] Fig.12 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0049] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0051] It should be noted that the information (including but not limited to the target object's device information, target object's personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the target object or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the target video involved in this application was obtained with full authorization.

[0052] In order to more clearly understand the present solution, the terms involved in the present disclosure are explained below.

[0053] Three-dimensional convolution kernel: a convolution kernel that can slide in three directions. In the disclosed embodiment, the three-dimensional convolution kernel can slide in two spatial directions and a temporal direction to process image frames in the target video.

[0054] Receptive Field: It is the size of the area where the pixels on the feature map output by each layer of the convolutional neural network are mapped back to the input image.

[0055] Sub-pixel rearrangement: It is a processing method that reorders the pixels on multiple feature maps output by multiple convolutions to obtain a new feature map.

[0056] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of an image processing method according to an exemplary embodiment. Taking the execution by a server as an example, see Figure 1 , the implementation environment includes: a terminal 101 and a server 102.

[0057] The terminal 101 is at least one of a smart phone, a smart watch, a desktop computer, a laptop, an MP3 player, an MP4 player, and a laptop computer. An application that supports video acquisition is installed and run on the terminal 101. The application can be a multimedia application, a social application, or a monitoring application, which is not limited in the embodiment of the present disclosure. Taking the application as a multimedia application as an example, the target video can be obtained by shooting based on the multimedia application, or can be selected from the gallery of the terminal 101 based on the multimedia application, which is not limited in the embodiment of the present disclosure. The terminal 101 can be connected to the server 102 via a wireless network or a wired network, so that the target video can be sent to the server 102. The terminal 101 generally refers to one of a plurality of terminals, and the present embodiment is illustrated by the terminal 101. Those skilled in the art can know that the number of the above terminals can be more or less. For example, the above terminals can be several, or the above terminals can be dozens or hundreds, or more, and the embodiment of the present disclosure does not limit the number of terminals and device types.

[0058] The server 102 is at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 102 can be connected to the terminal 101 and other terminals via a wireless network or a wired network. The server 102 can receive the target video sent by the terminal 101, and obtain the target image frame and the reference image frame of the target image frame from the target video. Then, the server 102 processes the target image frame and the reference image frame to obtain a high-resolution image of the target video frame. Then, the server 102 can return the high-resolution image to the terminal 101, and the terminal 101 displays the high-resolution image. In some embodiments, the number of the above-mentioned servers can be more or less, and the embodiments of the present disclosure are not limited to this. Of course, the server 102 also includes other functional servers to provide more comprehensive and diversified services.

[0059] Figure 2 is a flowchart of an image processing method according to an exemplary embodiment. Figure 2 , taking the execution by the server as an example, the following steps are included:

[0060] In step 201, the server obtains a plurality of reference image frames of a target image frame from a target video, and the number of frames between the reference image frames and the target image frame is not greater than the number of target frames.

[0061] In an embodiment of the present disclosure, the server can perform frame processing on the target video to obtain multiple image frames. The multiple image frames are arranged in chronological order. The server can obtain multiple reference image frames of the target image frame from the multiple image frames based on the target image frame. The target image frame and the multiple reference image frames are also arranged in chronological order. For any reference image frame in the multiple reference image frames, the number of frames between the reference image frame and the target image frame is not greater than the target number of frames. Among them, the target number of frames can be 2 frames, 3 frames or 4 frames, etc., which is not limited in the embodiment of the present disclosure. For any reference image frame, if the number of frames between the reference image frame and the target image frame is zero, it means that the reference image frame is adjacent to the target image frame. If the number of frames between the reference image frame and the target image frame is not zero, it means that there are other reference image frames between the reference image frame and the target image frame.

[0062] In step 202, based on the three-dimensional convolution kernel, the server slides symmetrically frame by frame on multiple reference image frames with the target image frame as the center to obtain image fusion features.

[0063] In the disclosed embodiment, the server can slide the three-dimensional convolution kernel symmetrically frame by frame in the time sequence direction with the target image frame as the center. The three-dimensional convolution kernel can be divided into a first convolution kernel, a fixed convolution kernel and a second convolution kernel. The first convolution kernel and the second convolution kernel slide symmetrically with the fixed convolution kernel as the center. The fixed convolution kernel corresponds to the target image frame. The server can slide the first convolution kernel and the second convolution kernel symmetrically to both sides of the target image frame with the target image frame as the center. During the sliding process, the server can process the target image frame and the reference image frame corresponding to the three-dimensional convolution kernel for multiple times to obtain image fusion features. The reference image frame corresponding to the three-dimensional convolution kernel changes based on the sliding of the three-dimensional convolution kernel. For any sliding, the server can process the reference image frame corresponding to the first convolution kernel based on the first convolution kernel; process the reference image frame corresponding to the second convolution kernel based on the second convolution kernel; and process the target image frame based on the fixed convolution kernel.

[0064] In step 203, the server determines a high-resolution image corresponding to the target image frame based on the image fusion feature.

[0065] In the disclosed embodiment, the server can upsample the image fusion features and determine the high-resolution image corresponding to the target image frame based on the upsampled features. The high-resolution image has a higher resolution than the target image frame and can reflect more image information.

[0066] The solution provided by the embodiment of the present disclosure can expand the receptive field of the three-dimensional convolution kernel by symmetrically sliding the three-dimensional convolution kernel on multiple reference image frames frame by frame with the target image frame as the center, and can process the target image frame and the reference image frame in each convolution, thereby making full use of the influence of the reference image frame on the target image frame, so that the high-resolution image obtained based on the image fusion feature is clearer.

[0067] In some embodiments, the three-dimensional convolution kernel includes a first convolution kernel, a fixed convolution kernel, and a second convolution kernel, and the first convolution kernel and the second convolution kernel are used to slide symmetrically frame by frame on multiple reference image frames with the fixed convolution kernel as the center, and the number of sliding times is equal to the target frame number;

[0068] Based on the three-dimensional convolution kernel, the target image frame is centered and symmetrically slid on multiple reference image frames frame by frame to obtain image fusion features, including:

[0069] In response to a sliding behavior of the three-dimensional convolution kernel, determining a first reference image frame and a second reference image frame corresponding to the sliding behavior, wherein the first reference image frame and the second reference image frame are symmetrical with respect to the target image frame;

[0070] The target image frame is processed based on the fixed convolution kernel, the first reference image frame of the sliding behavior is processed based on the first convolution kernel, and the second reference image frame of the sliding behavior is processed based on the second convolution kernel to obtain a first intermediate fusion feature;

[0071] An image fusion feature is determined based on a first intermediate fusion feature and a second intermediate fusion feature corresponding to the multiple sliding behaviors, where the second intermediate fusion feature is obtained based on the three-dimensional convolution kernel when the three-dimensional convolution kernel does not start to slide.

[0072] The solution provided by the embodiment of the present disclosure can expand the receptive field of the three-dimensional convolution kernel by symmetrically sliding the first convolution kernel and the second convolution kernel in the three-dimensional convolution kernel frame by frame on multiple reference image frames with a fixed convolution kernel as the center. In each convolution, the target image frame and the first reference image frame and the second reference image frame on both sides of the target image frame can be processed, thereby making full use of the influence of the reference image frame on the target image frame. The image fusion features obtained can better reflect the influence of the reference image frame on the target image frame, which is conducive to obtaining clearer high-resolution images subsequently.

[0073] In some embodiments, the target image frame is processed based on a fixed convolution kernel, the first reference image frame of the sliding behavior is processed based on a first convolution kernel, and the second reference image frame of the sliding behavior is processed based on a second convolution kernel to obtain a first intermediate fusion feature, including:

[0074] Processing the target image frame based on a fixed convolution kernel to obtain a first intermediate feature;

[0075] Processing the first reference image frame based on the first convolution kernel to obtain a second intermediate feature;

[0076] Processing the second reference image frame based on the second convolution kernel to obtain a third intermediate feature;

[0077] The first intermediate feature, the second intermediate feature, and the third intermediate feature are weightedly summed to obtain a first intermediate fusion feature.

[0078] The solution provided by the disclosed embodiment performs weighted summation on the features obtained by processing the target image frame with a fixed convolution kernel, the features obtained by processing the first reference image frame with a first convolution kernel, and the features obtained by processing the second reference image frame with a second convolution kernel, so that the first intermediate fusion feature can fully reflect the influence of the first reference image frame and the second reference image frame on the target image frame, which is beneficial to obtaining a clearer high-resolution image subsequently.

[0079] In some embodiments, the image processing method further includes:

[0080] When the three-dimensional convolution kernel does not start to slide, the target image frame is processed based on the fixed convolution kernel to obtain the fourth intermediate feature;

[0081] Processing a third reference image frame based on the first convolution kernel to obtain a fifth intermediate feature, where the third reference image frame is a reference image frame corresponding to when the first convolution kernel does not start to slide;

[0082] Processing a fourth reference image frame based on the second convolution kernel to obtain a sixth intermediate feature, where the fourth reference image frame is a reference image frame corresponding to when the second convolution kernel does not start to slide, and the third reference image frame and the fourth reference image frame are symmetrical with respect to the target image frame;

[0083] The fourth intermediate feature, the fifth intermediate feature and the sixth intermediate feature are weightedly summed to obtain the second intermediate fusion feature.

[0084] The solution provided by the disclosed embodiment, when the three-dimensional convolution kernel has not started to slide, performs weighted summation on the features obtained by processing the target image frame with the fixed convolution kernel, the features obtained by processing the third reference image frame with the first convolution kernel, and the features obtained by processing the fourth reference image frame with the second convolution kernel, so that the second intermediate fusion feature can fully reflect the influence of the third reference image frame and the fourth reference image frame on the target image frame, which is beneficial to obtaining a clearer high-resolution image subsequently.

[0085] In some embodiments, the first reference image frame is at least one reference image frame, the second reference image frame is at least one reference image frame, the third reference image frame is at least one reference image frame, and the fourth reference image frame is at least one reference image frame, and the number of the first reference image frames is equal to the number of the third reference image frames.

[0086] According to the solution provided by the embodiment of the present disclosure, the first convolution kernel and the second convolution kernel can process at least one reference image frame in one convolution process, thereby making full use of the influence of the reference image frame on the target image frame, which is conducive to the subsequent generation of clear high-resolution images.

[0087] In some embodiments, obtaining a plurality of reference image frames corresponding to a target image frame from a target video includes:

[0088] Taking the target image frame as the center, obtaining reference image frames frame by frame from both sides of the target image frame;

[0089] In response to the number of reference image frames acquired by the first side being greater than the number of reference image frames acquired by the second side, the reference image frame located at a first position on the first side is copied to obtain a reference image frame located at a second position on the second side, and the first position and the second position are symmetrical with the position of the target image frame as the center.

[0090] The solution provided by the embodiment of the present disclosure can, when the numbers of reference image frames acquired from both sides of a target image frame are unequal, copy the reference image frames on the side with a larger number of reference image frames and use them as the missing reference image frames on the side with a smaller number of reference image frames. This allows the influence of the reference image frames on the target image frame to be fully utilized when processing the target image frame and the reference image frame, thereby facilitating subsequent acquisition of a clearer high-resolution image.

[0091] In some embodiments, obtaining a plurality of reference image frames corresponding to a target image frame from a target video includes:

[0092] Taking the target image frame as the center, obtaining reference image frames frame by frame from both sides of the target image frame;

[0093] In response to the number of acquired reference image frames being less than the target number, the target image frames are copied to obtain the missing reference image frames.

[0094] The solution provided by the embodiment of the present disclosure can copy the target image frame as the missing reference image frame when the number of reference image frames obtained is less than the target number, so that when the target image frame and the reference image frame are processed, the influence of the reference image frame on the target image frame can be fully utilized, which is conducive to obtaining clearer high-resolution images subsequently.

[0095] Above Figure 2 The basic process of the present disclosure is shown below. The solution provided by the present disclosure is further described based on an implementation method. Figure 3 is a flowchart of another image processing method according to an exemplary embodiment. Figure 3 , the method comprising:

[0096] In step 301, the server obtains a plurality of reference image frames of a target image frame from a target video, and the number of frames between the reference image frames and the target image frame is not greater than the number of target frames.

[0097] In an embodiment of the present disclosure, the server can perform frame processing on the target video to obtain multiple image frames corresponding to the target video. The multiple image frames are arranged in chronological order. The server can obtain multiple reference image frames of the target image frame based on the multiple image frames and the target frame number. The multiple reference image frames and the target image frame are arranged continuously in chronological order, and the number of frames between each reference image frame and the target image frame is not greater than the target frame number. The multiple reference image frames are located on both sides of the target image frame, and the positions of the multiple reference image frames are symmetrical with the target image frame as the center. The number of the multiple reference image frames can be 4, 6 or 8, etc., which is not limited by the embodiment of the present disclosure.

[0098] In the case where there are a target number of image frames in the multiple image frames whose interval with the target image frame is not greater than the target number of frames, the server can directly obtain the target number of image frames from the multiple image frames as multiple reference image frames of the target image frame. The target number is the number of reference image frames to be obtained, and the target number of frames is the number of frames between the reference image frame with the largest interval with the target image frame in the time sequence direction and the target image. Half of the target number is equal to the target number of frames plus one. In the case where there are no target number of image frames in the multiple image frames whose interval with the target image frame is not greater than the target number of frames, the number of reference image frames obtained by the server is less than the target number. In this case, the server can supplement the missing reference image frames of the target image frame by copying the reference image frames of the target image frame that have been obtained; or, the server can supplement the missing reference image frames of the target image frame by copying the target image frame, and the embodiments of the present disclosure are not limited to this.

[0099] In some embodiments, the server can supplement the missing reference image frame of the target image frame by copying the reference image frame of the target image frame that has been obtained. Accordingly, the process of the server obtaining the reference image frame of the target image frame is: the server takes the target image frame as the center and obtains the reference image frame frame by frame from both sides of the target image frame. Then, in response to the number of reference image frames obtained by the first side being greater than the number of reference image frames obtained by the second side, the server copies the reference image frame located at the first position in the first side to obtain the reference image frame located at the second position in the second side. Among them, the first position and the second position are symmetrical with the position of the target image frame as the center. The first side is the side before the timing of the target image frame, and the second side is the side after the timing of the target image frame; or, the first side is the side after the timing of the target image frame, and the second side is the side before the timing of the target image frame, which is not limited in the embodiments of the present disclosure. The solution provided by the embodiment of the present disclosure can, when the numbers of reference image frames acquired from both sides of a target image frame are unequal, copy the reference image frames on the side with a larger number of reference image frames and use them as the missing reference image frames on the side with a smaller number of reference image frames. This allows the influence of the reference image frames on the target image frame to be fully utilized when processing the target image frame and the reference image frame, thereby facilitating subsequent acquisition of a clearer high-resolution image.

[0100] The first position and the second position can be represented by the number of the reference image frame in the target video. Figure 4 FIG. 1 is a schematic diagram showing a method of obtaining a reference image frame according to an exemplary embodiment. Figure 4, the server can number multiple image frames obtained by frame processing based on the time corresponding to the image frames, and the multiple image frames include the target image frame and the reference image frame of the target image frame. The target image frame is the 3rd frame, and there are two reference image frames, the 1st frame and the 2nd frame, before the timing of the target image frame, and there are three reference image frames, the 4th frame, the 5th frame and the 6th frame, after the timing of the target image frame. The number of reference image frames after the timing of the target image frame is greater than the number of reference image frames before the timing of the target image frame. The 2nd frame and the 4th frame are symmetrical with the target image frame as the center position, and the 1st frame and the 5th frame are symmetrical with the target image frame as the center position. There is no reference image frame symmetrical with the position of the 6th frame before the timing of the target image frame. The server can copy the reference image frame of the 6th frame. Then, the server places the copied reference image frame at a position symmetrical to the 6th frame, that is, at a position adjacent to the 1st frame and before the timing of the 1st frame. The multiple reference image frames of the target image frame acquired by the server are frame 6, frame 1, frame 2, frame 4, frame 5 and frame 6 in order of position. The server can also renumber the target image frame and the multiple reference image frames acquired, so that the position of the reference image frame is indicated by the number, which is not repeated here.

[0101] In some embodiments, the server can copy the target image frame to supplement the reference image frame that is missing from the target image frame. Accordingly, the process for the server to obtain the reference image frame of the target image frame is as follows: the server takes the target image frame as the center and obtains the reference image frame frame by frame from both sides of the target image frame. Then, in response to the number of reference image frames obtained being less than the target number, the server copies the target image frame to obtain the missing reference image frame. The solution provided by the embodiment of the present disclosure can copy the target image frame as the missing reference image frame when the number of reference image frames obtained is less than the target number, so that when the target image frame and the reference image frame are processed, the influence of the reference image frame on the target image frame can be fully utilized, which is conducive to obtaining a clearer high-resolution image later.

[0102] For example, the target number is 6. Figure 4 , the target image frame is the 3rd frame. The server obtains 5 reference image frames of the target image frame, and the number of reference image frames obtained is less than the target number. There is a missing reference image frame before the timing of the 3rd frame. The server can copy the 3rd frame to obtain the missing reference image frame. The server can place the copied reference image frame at a position adjacent to the 1st frame and before the timing of the 1st frame. Finally, the multiple reference image frames of the target image frame obtained by the server are the 3rd frame, the 1st frame, the 2nd frame, the 4th frame, the 5th frame and the 6th frame in order of position.

[0103] In step 302, when the three-dimensional convolution kernel has not started to slide, the server processes the target image frame and the reference image frame corresponding to the three-dimensional convolution kernel when the three-dimensional convolution kernel has not started to slide based on the three-dimensional convolution kernel to obtain a second intermediate fusion feature. The three-dimensional convolution kernel includes a first convolution kernel, a fixed convolution kernel and a second convolution kernel. The first convolution kernel and the second convolution kernel are used to slide symmetrically frame by frame on multiple reference image frames with the fixed convolution kernel as the center.

[0104] In an embodiment of the present disclosure, the server can process a target image frame and multiple reference image frames based on a three-dimensional convolution kernel. In the three-dimensional convolution kernel, the first convolution kernel is before the timing of the fixed convolution kernel, and the second convolution kernel is after the timing of the fixed convolution kernel; or, the first convolution kernel is after the timing of the fixed convolution kernel, and the second convolution kernel is before the timing of the fixed convolution kernel, and the embodiment of the present disclosure is not limited to this. The fixed convolution kernel in the three-dimensional convolution kernel is fixed at the position of the target image frame for processing the target image frame. The first convolution kernel and the second convolution kernel in the three-dimensional convolution kernel can slide symmetrically frame by frame on multiple reference image frames with the fixed convolution kernel as the center in the timing direction for processing the multiple reference image frames.

[0105] In the case where the three-dimensional convolution kernel does not start to slide, the server can process the target image frame and the reference image frames corresponding to the first convolution kernel and the second convolution kernel respectively based on the three-dimensional convolution kernel. Correspondingly, the process for the server to obtain the second intermediate fusion feature is as follows: in the case where the three-dimensional convolution kernel does not start to slide, the server processes the target image frame based on the fixed convolution kernel to obtain the fourth intermediate feature. The server processes the third reference image frame based on the first convolution kernel to obtain the fifth intermediate feature. The server processes the fourth reference image frame based on the second convolution kernel to obtain the sixth intermediate feature. Then, the server performs weighted summation on the fourth intermediate feature, the fifth intermediate feature and the sixth intermediate feature to obtain the second intermediate fusion feature. The solution provided by the embodiment of the present disclosure, in the case where the three-dimensional convolution kernel does not start to slide, by weighted summation of the features obtained by processing the target image frame with the fixed convolution kernel, the features obtained by processing the third reference image frame with the first convolution kernel and the features obtained by processing the fourth reference image frame with the second convolution kernel, so that the second intermediate fusion feature can fully reflect the influence of the third reference image frame and the fourth reference image frame on the target image frame, which is conducive to obtaining a clearer high-resolution image in the future.

[0106] The third reference image frame is a reference image frame corresponding to when the first convolution kernel does not start to slide; the fourth reference image frame is a reference image frame corresponding to when the second convolution kernel does not start to slide. The third reference image frame and the fourth reference image frame are symmetrical with respect to the target image frame.

[0107] In some embodiments, the third reference image frame is at least one reference image frame, and the fourth reference image frame is at least one reference image frame. The number of the third reference image frames is equal to the number of the fourth reference image frames. The number of the third reference image frames can be 1, 2, or 3, etc., which is not limited in the embodiments of the present disclosure. When the third reference image frame and the fourth reference image frame are both reference image frames, the third reference image frame and the fourth reference image frame are both reference image frames adjacent to the target image frame.

[0108] For example, the number of the third reference image frame is 1, and in the three-dimensional convolution kernel, the first convolution kernel is before the timing of the fixed convolution kernel, and the second convolution kernel is after the timing of the fixed convolution kernel. Figure 5 FIG. 1 is a schematic diagram showing a method of obtaining a second intermediate fusion feature according to an exemplary embodiment. Figure 5 , the target image frame is the 4th frame, and the multiple reference image frames of the target image frame are the 1st frame, the 2nd frame, the 3rd frame, the 5th frame, the 6th frame and the 7th frame. When the three-dimensional convolution kernel does not start to slide, the fixed convolution kernel corresponds to the 4th frame, the first convolution kernel corresponds to the 3rd frame, and the second convolution kernel corresponds to the 5th frame. The server can process the 4th frame based on the fixed convolution kernel to obtain the fourth intermediate feature; process the 3rd frame based on the first convolution kernel to obtain the fifth intermediate feature; and process the 5th frame based on the second convolution kernel to obtain the sixth intermediate feature. Then, the server performs weighted summation on the above three features to obtain the second intermediate fusion feature.

[0109] In step 303, in response to a sliding behavior of the three-dimensional convolution kernel, the server determines a first reference image frame and a second reference image frame corresponding to the sliding behavior, and the first reference image frame and the second reference image frame are symmetrical with the target image frame as the center.

[0110] In an embodiment of the present disclosure, the server fixes the fixed convolution kernel in the three-dimensional convolution kernel at the position of the target image frame, and slides the first convolution kernel and the second convolution kernel in the three-dimensional convolution kernel symmetrically frame by frame on multiple reference image frames along the time direction with the fixed convolution kernel as the center. The server slides the three-dimensional convolution kernel once, and the reference image frame corresponding to the first convolution kernel and the reference image frame corresponding to the second convolution kernel will both change. Based on the sliding behavior of the three-dimensional convolution kernel, the server can determine the first reference image frame and the second reference image frame corresponding to the sliding of the three-dimensional convolution kernel. Among them, the first reference image frame is the reference image frame corresponding to the sliding of the first convolution kernel, and the second reference image frame is the reference image frame corresponding to the sliding of the second convolution kernel. The solution provided by the embodiment of the present disclosure can expand the receptive field of the three-dimensional convolution kernel by symmetrically sliding the first convolution kernel and the second convolution kernel in the three-dimensional convolution kernel frame by frame on multiple reference image frames with a fixed convolution kernel as the center. In each convolution, the target image frame and the first reference image frame and the second reference image frame on both sides of the target image frame can be processed, thereby making full use of the influence of the reference image frame on the target image frame. The image fusion features obtained can better reflect the influence of the reference image frame on the target image frame, which is conducive to obtaining clearer high-resolution images subsequently.

[0111] In some embodiments, the first reference image frame is at least one reference image frame, and the second reference image frame is at least one reference image frame. The number of the first reference image frames is equal to the number of the second reference image frames. The number of the first reference image frames may be 1, 2, or 3, etc., which is not limited in the embodiments of the present disclosure. The number of the first reference image frames is equal to the number of the third reference image frames. The number of the first reference image frames is related to the size of the three-dimensional convolution kernel. If the size of the three-dimensional convolution kernel is 3*3*3, the number of the first reference image frames is 1; if the size of the three-dimensional convolution kernel is 3*3*5, the number of the first reference image frames is 2; if the size of the three-dimensional convolution kernel is 3*3*7, the number of the first reference image frames is 3, which is not limited in the embodiments of the present disclosure. The number of layers of the convolution kernel of the three-dimensional convolution kernel is equal to twice the number of the first reference image frames plus one.

[0112] In step 304, the server processes the target image frame based on the fixed convolution kernel, processes the first reference image frame of the sliding behavior based on the first convolution kernel, and processes the second reference image frame of the sliding behavior based on the second convolution kernel to obtain a first intermediate fusion feature.

[0113] In the disclosed embodiment, the server can process the target image frame, the first reference image frame, and the second reference image frame based on the three-dimensional convolution kernel, respectively obtain features corresponding to the image frames, and then fuse the features corresponding to the image frames to obtain the first intermediate fusion feature. Accordingly, the process for the server to obtain the first intermediate fusion feature is as follows: the server processes the target image frame based on the fixed convolution kernel to obtain the first intermediate feature. The server processes the first reference image frame based on the first convolution kernel to obtain the second intermediate feature. The server processes the second reference image frame based on the second convolution kernel to obtain the third intermediate feature. Then, the server performs weighted summation on the first intermediate feature, the second intermediate feature, and the third intermediate feature to obtain the first intermediate fusion feature.

[0114] For example, the server starts to slide the three-dimensional convolution kernel once along the timing direction. In the three-dimensional convolution kernel, the first convolution kernel is before the timing of the fixed convolution kernel, and the second convolution kernel is after the timing of the fixed convolution kernel. Figure 6 FIG. 1 is a schematic diagram showing a method of obtaining a first intermediate fusion feature after sliding a three-dimensional convolution kernel for the first time according to an exemplary embodiment. Figure 6 , the target image frame is the 4th frame, and the multiple reference image frames of the target image frame are the 1st frame, the 2nd frame, the 3rd frame, the 5th frame, the 6th frame and the 7th frame. When the 3D convolution kernel slides for the first time, the first reference image frame is the 2nd frame, and the second reference image frame is the 6th frame. The server processes the 4th frame based on the fixed convolution kernel to obtain the first intermediate feature; processes the 2nd frame based on the first convolution kernel to obtain the second intermediate feature; and processes the 6th frame based on the second convolution kernel to obtain the third intermediate feature. Then, the server performs weighted summation on the above three features to obtain the first intermediate fusion feature.

[0115] In step 305, the server determines an image fusion feature based on the first intermediate fusion feature and the second intermediate fusion feature corresponding to the multiple sliding behaviors.

[0116] In the disclosed embodiment, for any sliding behavior, the server can obtain a first intermediate fusion feature. The server slides the three-dimensional convolution kernel multiple times to obtain multiple first intermediate fusion features. The number of slides is equal to the target frame number. After the three-dimensional convolution kernel slides to a reference image frame whose number of frames is equal to the target frame number, the sliding of the three-dimensional convolution kernel ends. The server fuses the first intermediate fusion features and the second intermediate fusion features corresponding to the multiple sliding behaviors to obtain an image fusion feature.

[0117] For example, Figure 7 FIG. 1 is a schematic diagram showing a method of obtaining image fusion features by sliding a three-dimensional convolution kernel according to an exemplary embodiment. Figure 7, the target image frame is the 4th frame, and the multiple reference image frames of the target image frame are the 1st frame, the 2nd frame, the 3rd frame, the 5th frame, the 6th frame and the 7th frame. When the 3D convolution kernel does not slide, the server processes the 3rd frame, the 4th frame and the 5th frame based on the 3D convolution kernel to obtain the second intermediate fusion feature. Then, the server slides the 3D convolution kernel for the first time, processes the 2nd frame, the 4th frame and the 6th frame, and obtains a first intermediate fusion feature. Then, the server slides the 3D convolution kernel for the second time, processes the 1st frame, the 4th frame and the 7th frame, and obtains a first intermediate fusion feature. At this time, the 1st frame and the 7th frame are both the reference image frames farthest from the 4th frame in the time sequence direction, that is, the number of frames between the 1st frame and the 7th frame and the 4th frame is equal to the number of target frames. The sliding of the 3D convolution kernel ends. The server can perform sub-pixel rearrangement on the two first intermediate fusion features and the second intermediate fusion features to obtain the image fusion feature.

[0118] In step 306, the server determines a high-resolution image corresponding to the target image frame based on the image fusion feature.

[0119] In the disclosed embodiment, the server can upsample the image fusion feature to determine the super-resolution feature of the target image frame. Then, the server can deconvolve the super-resolution feature to obtain a high-resolution image of the target image frame.

[0120] In order to clearly understand this scheme, the following will apply this scheme to the reconstruction of wheels in an image and compare it with previous methods and other methods. Figure 8 The figure is a rendering of a wheel reconstruction according to an exemplary embodiment.

[0121] Figure 8 (a) shows the target image where the wheel is located; Figure 8 (b) shows the original shape of the wheel in the target image; Figure 8 (c) shows a wheel obtained by processing the target image using a bicubic interpolation algorithm; Figure 8 (d) in the figure shows a wheel obtained by processing the target image using a previous method, where the previous method processes the target image and images related to the target image in sequence in time using a three-dimensional convolution kernel. Figure 8 (e) shows the wheel obtained by processing the target image using the image processing method provided by this solution. Figure 8 In (b), Figure 8By comparing (c), (d) and (e) in FIG. 1 , it can be seen that the wheel obtained by the bicubic interpolation algorithm and the previous method is blurred and the detailed information of the wheel is missing. Compared with the other two methods, the wheel obtained by the image processing method provided by this solution is clearer.

[0122] The server can also apply this solution to the reconstruction of faces in images and compare it with previous methods and other methods. Fig. 9 The figure is a rendering of a face reconstruction according to an exemplary embodiment. Fig. 9 (a) shows the target image where the face is located; Fig. 9 (b) shows the original appearance of the face in the target image; Fig. 9 (c) shows the face obtained by processing the target image using the Task-Oriented Flow (TOFlow) method; Fig. 9 (d) in the figure shows a face obtained by processing the target image using a previous method, where the previous method processes the target image and images related to the target image in sequence in time using a three-dimensional convolution kernel. Fig. 9 (e) shows the face obtained by processing the target image using the image processing method provided by this solution. Fig. 9 In (b), Fig. 9 By comparing (c), (d) and (e) in Figure 2, it can be seen that the faces obtained by the optical flow method and the previous method are blurred, and the detailed information of the face is missing. Fig. 9 The face shown in (d) is distorted. Compared with the other two methods, the face obtained by the image processing method provided by the present solution is clearer.

[0123] The server can also apply this solution to the reconstruction process of buildings in the image to obtain clearer buildings, which will not be elaborated here.

[0124] The solution provided by the embodiment of the present disclosure can expand the receptive field of the three-dimensional convolution kernel by symmetrically sliding the three-dimensional convolution kernel on multiple reference image frames frame by frame with the target image frame as the center, and can process the target image frame and the reference image frame in each convolution, thereby making full use of the influence of the reference image frame on the target image frame, so that the high-resolution image obtained based on the image fusion feature is clearer.

[0125] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0126] Fig.10 FIG. 1 is a schematic diagram showing the structure of an image processing device according to an exemplary embodiment. Fig.10 The image processing device includes: an acquisition unit 1001, a fusion unit 1002 and a determination unit 1003.

[0127] The acquisition unit 1001 is configured to acquire a plurality of reference image frames of a target image frame from a target video, wherein the number of frames between the reference image frames and the target image frame is not greater than the number of target frames;

[0128] The fusion unit 1002 is configured to perform symmetrical sliding on multiple reference image frames frame by frame based on a three-dimensional convolution kernel with the target image frame as the center to obtain image fusion features;

[0129] The determination unit 1003 is configured to determine the high-resolution image corresponding to the target image frame based on the image fusion feature.

[0130] In some embodiments, Fig.11 FIG. 1 is a schematic diagram showing the structure of another image processing device according to an exemplary embodiment. Fig.11 , the three-dimensional convolution kernel includes a first convolution kernel, a fixed convolution kernel and a second convolution kernel, the first convolution kernel and the second convolution kernel are used to slide symmetrically frame by frame on multiple reference image frames with the fixed convolution kernel as the center, and the number of sliding times is equal to the target frame number;

[0131] The fusion unit 1002 includes:

[0132] The first determining subunit 1101 is configured to execute a sliding behavior in response to the three-dimensional convolution kernel, and determine a first reference image frame and a second reference image frame corresponding to the sliding behavior, wherein the first reference image frame and the second reference image frame are symmetrical with respect to the target image frame;

[0133] The processing subunit 1102 is configured to process the target image frame based on the fixed convolution kernel, process the first reference image frame of the sliding behavior based on the first convolution kernel, and process the second reference image frame of the sliding behavior based on the second convolution kernel to obtain a first intermediate fusion feature;

[0134] The second determining subunit 1103 is configured to execute a first intermediate fusion feature and a second intermediate fusion feature corresponding to multiple sliding behaviors to determine an image fusion feature, where the second intermediate fusion feature is obtained based on the three-dimensional convolution kernel when the three-dimensional convolution kernel does not start to slide.

[0135] In some embodiments, see Fig.11The processing sub-unit 1102 is configured to process the target image frame based on a fixed convolution kernel to obtain a first intermediate feature; process the first reference image frame based on the first convolution kernel to obtain a second intermediate feature; process the second reference image frame based on the second convolution kernel to obtain a third intermediate feature; and perform weighted summation on the first intermediate feature, the second intermediate feature, and the third intermediate feature to obtain a first intermediate fusion feature.

[0136] In some embodiments, see Fig.11 , the image processing device also includes:

[0137] The processing unit 1004 is configured to process the target image frame based on the fixed convolution kernel when the three-dimensional convolution kernel does not start to slide, so as to obtain a fourth intermediate feature;

[0138] The processing unit 1004 is further configured to process a third reference image frame based on the first convolution kernel to obtain a fifth intermediate feature, where the third reference image frame is a reference image frame corresponding to when the first convolution kernel does not start to slide;

[0139] The processing unit 1004 is further configured as an execution processing unit, and is further configured to process a fourth reference image frame based on the second convolution kernel to obtain a sixth intermediate feature, where the fourth reference image frame is a reference image frame corresponding to when the second convolution kernel does not start to slide, and the third reference image frame and the fourth reference image frame are symmetrical with respect to the target image frame;

[0140] The fusion unit 1002 is further configured to perform weighted summation on the fourth intermediate feature, the fifth intermediate feature and the sixth intermediate feature to obtain a second intermediate fusion feature.

[0141] In some embodiments, see Fig.11 , the first reference image frame is at least one reference image frame, the second reference image frame is at least one reference image frame, the third reference image frame is at least one reference image frame, the fourth reference image frame is at least one reference image frame, and the number of the first reference image frames is equal to the number of the third reference image frames.

[0142] In some embodiments, see Fig.11 The acquisition unit 1001 is configured to acquire reference image frames frame by frame from both sides of the target image frame with the target image frame as the center; in response to the number of reference image frames acquired on the first side being greater than the number of reference image frames acquired on the second side, the reference image frame located at a first position on the first side is copied to obtain a reference image frame located at a second position on the second side, and the first position and the second position are symmetrical with the position of the target image frame as the center.

[0143] In some embodiments, see Fig.11 The acquisition unit is configured to acquire reference image frames frame by frame from both sides of the target image frame with the target image frame as the center; in response to the number of acquired reference image frames being less than the target number, the target image frame is copied to obtain the missing reference image frames.

[0144] The device provided by the embodiment of the present disclosure can expand the receptive field of the three-dimensional convolution kernel by symmetrically sliding the three-dimensional convolution kernel frame by frame on multiple reference image frames with the target image frame as the center, and can process the target image frame and the reference image frame in each convolution, thereby making full use of the influence of the reference image frame on the target image frame, so that the high-resolution image obtained based on the image fusion feature is clearer.

[0145] It should be noted that the device provided in the above embodiment uses the division of the above functional units as an example when processing an image. In actual applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the image processing device provided in the above embodiment and the image processing method embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.

[0146] Regarding the device in the above embodiment, the manner in which each module performs operations has been described in the embodiment of the method, and will not be elaborated here.

[0147] When the electronic device is provided as a server, Fig.12 This is a block diagram of a server 1200 according to an exemplary embodiment. The server 1200 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1201 and one or more memories 1202, wherein the memory 1202 stores at least one program code, and the at least one program code is loaded and executed by the processor 1201 to implement the image processing methods provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 1200 may also include other components for implementing device functions, which will not be described in detail here.

[0148] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1202 including instructions, and the instructions can be executed by a processor 1201 of a server 1200 to complete the above-mentioned image processing method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0149] In an exemplary embodiment, a computer program product is also provided, comprising a computer program / instruction, which implements the above-mentioned image processing method when executed by a processor.

[0150] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be regarded as exemplary, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0151] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is defined by the appended claims.

Claims

1. An image processing method, characterized in that: The image processing method comprises: Taking a target image frame in a target video as the center, obtaining reference image frames frame by frame from both sides of the target image frame, wherein the number of frames between the reference image frame and the target image frame is not greater than the number of target frames; In response to the number of reference image frames acquired by the first side being greater than the number of reference image frames acquired by the second side, a reference image frame located at a first position in the first side is copied to obtain a reference image frame located at a second position in the second side, wherein the first position and the second position are symmetrical with respect to the position of the target image frame as the center; or, in response to the number of reference image frames acquired being less than the target number, the target image frame is copied to obtain the missing reference image frame; In response to a sliding behavior of the three-dimensional convolution kernel, a first reference image frame and a second reference image frame corresponding to the sliding behavior are determined, the first reference image frame and the second reference image frame are symmetrical with the target image frame as the center, the three-dimensional convolution kernel includes a first convolution kernel, a fixed convolution kernel and a second convolution kernel, the fixed convolution kernel is fixed at the position of the target image frame, the first convolution kernel and the second convolution kernel are used to slide symmetrically frame by frame on multiple reference image frames with the fixed convolution kernel as the center, and the number of sliding times is equal to the number of target frames; The target image frame is processed based on the fixed convolution kernel, the first reference image frame of the sliding behavior is processed based on the first convolution kernel, and the second reference image frame of the sliding behavior is processed based on the second convolution kernel to obtain a first intermediate fusion feature; determining an image fusion feature based on a first intermediate fusion feature and a second intermediate fusion feature corresponding to the multiple sliding behaviors, wherein the second intermediate fusion feature is obtained based on the three-dimensional convolution kernel when the three-dimensional convolution kernel does not start to slide; Based on the image fusion feature, a high-resolution image corresponding to the target image frame is determined.

2. The image processing method according to claim 1, characterized in that: The processing of the target image frame based on the fixed convolution kernel, the processing of the first reference image frame of the sliding behavior based on the first convolution kernel, and the processing of the second reference image frame of the sliding behavior based on the second convolution kernel to obtain a first intermediate fusion feature include: Processing the target image frame based on the fixed convolution kernel to obtain a first intermediate feature; Processing the first reference image frame based on the first convolution kernel to obtain a second intermediate feature; Processing the second reference image frame based on the second convolution kernel to obtain a third intermediate feature; A weighted sum is performed on the first intermediate feature, the second intermediate feature, and the third intermediate feature to obtain the first intermediate fusion feature.

3. The image processing method according to claim 1, characterized in that: The image processing method further comprises: When the three-dimensional convolution kernel does not start to slide, the target image frame is processed based on the fixed convolution kernel to obtain a fourth intermediate feature; Processing a third reference image frame based on the first convolution kernel to obtain a fifth intermediate feature, where the third reference image frame is a reference image frame corresponding to when the first convolution kernel does not start to slide; Processing a fourth reference image frame based on the second convolution kernel to obtain a sixth intermediate feature, wherein the fourth reference image frame is a reference image frame corresponding to when the second convolution kernel does not start to slide, and the third reference image frame and the fourth reference image frame are symmetrical with respect to the target image frame; The fourth intermediate feature, the fifth intermediate feature and the sixth intermediate feature are weightedly summed to obtain the second intermediate fusion feature.

4. The image processing method according to claim 3, characterized in that: The first reference image frame is at least one reference image frame, the second reference image frame is at least one reference image frame, the third reference image frame is at least one reference image frame, and the fourth reference image frame is at least one reference image frame, and the number of the first reference image frames is equal to the number of the third reference image frames.

5. An image processing device, characterized in that: The image processing device comprises: The acquisition unit is configured to execute the acquisition of reference image frames frame by frame from both sides of the target image frame in the target video as the center, wherein the number of frames between the reference image frame and the target image frame is not greater than the target number of frames; in response to the number of reference image frames acquired on the first side being greater than the number of reference image frames acquired on the second side, the reference image frame located at a first position on the first side is copied to obtain a reference image frame located at a second position on the second side, wherein the first position and the second position are symmetrical with the position of the target image frame as the center; or, in response to the number of reference image frames acquired being less than the target number, the target image frame is copied to obtain the missing reference image frame; A fusion unit is configured to execute a sliding behavior in response to a three-dimensional convolution kernel, determine a first reference image frame and a second reference image frame corresponding to the sliding behavior, the first reference image frame and the second reference image frame are symmetrical with the target image frame as the center, the three-dimensional convolution kernel includes a first convolution kernel, a fixed convolution kernel and a second convolution kernel, the fixed convolution kernel is fixed at the position of the target image frame, the first convolution kernel and the second convolution kernel are used to slide symmetrically frame by frame on multiple reference image frames with the fixed convolution kernel as the center, and the number of sliding times is equal to the target frame number; based on the fixed convolution kernel, the target image frame is processed, based on the first convolution kernel, the first reference image frame of the sliding behavior is processed, based on the second convolution kernel, the second reference image frame of the sliding behavior is processed, and a first intermediate fusion feature is obtained; based on the first intermediate fusion features and the second intermediate fusion features corresponding to multiple sliding behaviors, an image fusion feature is determined, and the second intermediate fusion feature is obtained based on the three-dimensional convolution kernel when the three-dimensional convolution kernel does not start to slide; The determination unit is configured to determine the high-resolution image corresponding to the target image frame based on the image fusion feature.

6. The image processing device according to claim 5, characterized in that: The fusion unit is configured to process the target image frame based on the fixed convolution kernel to obtain a first intermediate feature; process the first reference image frame based on the first convolution kernel to obtain a second intermediate feature; and process the second reference image frame based on the second convolution kernel to obtain a third intermediate feature; A weighted sum is performed on the first intermediate feature, the second intermediate feature, and the third intermediate feature to obtain the first intermediate fusion feature.

7. The image processing device according to claim 5, characterized in that: The device also includes: a processing unit configured to process the target image frame based on the fixed convolution kernel to obtain a fourth intermediate feature when the three-dimensional convolution kernel does not start to slide; The processing unit is further configured to process a third reference image frame based on the first convolution kernel to obtain a fifth intermediate feature, wherein the third reference image frame is a reference image frame corresponding to when the first convolution kernel does not start to slide; The processing unit is further configured to execute the processing unit, and is further configured to process a fourth reference image frame based on the second convolution kernel to obtain a sixth intermediate feature, wherein the fourth reference image frame is a reference image frame corresponding to when the second convolution kernel does not start to slide, and the third reference image frame and the fourth reference image frame are symmetrical with the target image frame as the center; The fusion unit is further configured to perform weighted summation on the fourth intermediate feature, the fifth intermediate feature and the sixth intermediate feature to obtain the second intermediate fusion feature.

8. The image processing device according to claim 7, characterized in that: The first reference image frame is at least one reference image frame, the second reference image frame is at least one reference image frame, the third reference image frame is at least one reference image frame, and the fourth reference image frame is at least one reference image frame, and the number of the first reference image frames is equal to the number of the third reference image frames.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the image processing method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method as described in any one of claims 1 to 4.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 4 is implemented.

Images