Image Processing Method and Apparatus

Abstract

The embodiments of this specification provide an image processing method and apparatus. The image processing method includes: obtaining an image at a first moment and an image at a second moment, and determining a sub-image at the first moment and a sub-image at the second moment based on the image at the first moment and the image at the second moment; determining a first result at a target moment and a first optical flow map at the target moment based on the image at the first moment and the image at the second moment, and determining a second result at the target moment and a second optical flow map at the target moment based on the sub-image at the first moment and the sub-image at the second moment; obtaining an image weight value at the target moment according to the first result, the first optical flow map, the second result, and the second optical flow map; and determining a target image at the target moment based on the image weight value, the first result, and the second result, effectively enhancing the image processing effect.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of image processing technology, and particularly to an image processing method. One or more embodiments of this specification also relate to an image processing apparatus, a computing device, and a computer-readable storage medium. Background Art

[0002] With the development of high-definition digital TVs and high-end multimedia information systems, people have higher and higher requirements for the visual effects of video sources. Therefore, it is necessary to increase the frame rate of existing video program sources to achieve better visual effects. Existing methods for increasing the frame rate usually use simple video frame replication or motion estimation and synthesis based on traditional motion estimation algorithms, and cannot effectively handle situations such as replicated motion scenes and shot transitions to obtain smoother frame interpolation results and improve the visual effects of video images.

[0003] Therefore, it is necessary to provide an image processing method that can solve the above problems. Summary of the Invention

[0004] In view of this, the embodiments of this specification provide an image processing method. One or more embodiments of this specification also relate to an image processing apparatus, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0005] According to the first aspect of the embodiments of this specification, an image processing method is provided, including:

[0006] Obtain an image at a first moment and an image at a second moment, and determine a sub-image at the first moment and a sub-image at the second moment based on the image at the first moment and the image at the second moment;

[0007] Determine a first result at a target moment and a first optical flow map at the target moment based on the image at the first moment and the image at the second moment, and determine a second result at the target moment and a second optical flow map at the target moment based on the sub-image at the first moment and the sub-image at the second moment;

[0008] Obtain an image weight value at the target moment according to the first result, the first optical flow map, the second result, and the second optical flow map;

[0009] Determine a target image at the target moment based on the image weight value, the first result, and the second result.

[0010] According to the second aspect of the embodiments of this specification, an image processing method is provided, including:

[0011] Display an image input interface for the user based on the user's image processing request;

[0012] Obtain the first - moment image and the second - moment image input by the user based on the image input interface, and determine the first - moment sub - image and the second - moment sub - image based on the first - moment image and the second - moment image;

[0013] Determine the first result and the first optical flow map at the target moment based on the first - moment image and the second - moment image, and determine the second result and the second optical flow map at the target moment based on the first - moment sub - image and the second - moment sub - image;

[0014] Obtain the image weight value at the target moment according to the first result, the first optical flow map, the second result and the second optical flow map;

[0015] Determine the target image at the target moment based on the image weight value, the first result and the second result, and return the target image at the target moment to the user.

[0016] According to the third aspect of the embodiments of the present specification, there is provided an image processing method, including:

[0017] Receive an image processing request sent by the user carrying the first - moment image and the second - moment image;

[0018] Obtain the first - moment image and the second - moment image, and determine the first - moment sub - image and the second - moment sub - image based on the first - moment image and the second - moment image;

[0019] Determine the first result and the first optical flow map at the target moment based on the first - moment image and the second - moment image, and determine the second result and the second optical flow map at the target moment based on the first - moment sub - image and the second - moment sub - image;

[0020] Obtain the image weight value at the target moment according to the first result, the first optical flow map, the second result and the second optical flow map;

[0021] Determine the target image at the target moment based on the image weight value, the first result and the second result, and return the target image at the target moment to the user.

[0022] According to the fourth aspect of the embodiments of the present specification, there is provided an image processing apparatus, including:

[0023] An acquisition module, configured to acquire the first - moment image and the second - moment image, and determine the first - moment sub - image and the second - moment sub - image based on the first - moment image and the second - moment image;

[0024] A first determination module, configured to determine a first result at a target time and a first optical flow map at the target time based on the image at the first time and the image at the second time, and determine a second result at the target time and a second optical flow map at the target time based on the sub-image at the first time and the sub-image at the second time;

[0025] An obtaining module, configured to obtain an image weight value at the target time according to the first result, the first optical flow map, the second result, and the second optical flow map;

[0026] A second determination module, configured to determine a target image at the target time based on the image weight value, the first result, and the second result.

[0027] According to a fifth aspect of the embodiments of the present specification, there is provided an image processing apparatus, including:

[0028] A display module, configured to display an image input interface for the user based on the user's image processing request;

[0029] An acquisition module, configured to acquire the image at the first time and the image at the second time input by the user based on the image input interface, and determine a sub-image at the first time and a sub-image at the second time based on the image at the first time and the image at the second time;

[0030] A first determination module, configured to determine a first result at a target time and a first optical flow map at the target time based on the image at the first time and the image at the second time, and determine a second result at the target time and a second optical flow map at the target time based on the sub-image at the first time and the sub-image at the second time;

[0031] An obtaining module, configured to obtain an image weight value at the target time according to the first result, the first optical flow map, the second result, and the second optical flow map;

[0032] A second determination module, configured to determine a target image at the target time based on the image weight value, the first result, and the second result, and return the target image at the target time to the user.

[0033] According to a sixth aspect of the embodiments of the present specification, there is provided an image processing apparatus, including:

[0034] A receiving module, configured to receive an image processing request sent by the user and carrying the image at the first time and the image at the second time;

[0035] An acquisition module, configured to acquire the first-moment image and the second-moment image, and determine a first-moment sub-image and a second-moment sub-image based on the first-moment image and the second-moment image;

[0036] A first determination module, configured to determine a first result at a target moment and a first optical flow map at the target moment based on the first-moment image and the second-moment image, and determine a second result at the target moment and a second optical flow map at the target moment based on the first-moment sub-image and the second-moment sub-image;

[0037] An obtaining module, configured to obtain an image weight value at the target moment according to the first result, the first optical flow map, the second result, and the second optical flow map;

[0038] A second determination module, configured to determine a target image at the target moment based on the image weight value, the first result, and the second result, and return the target image at the target moment to the user.

[0039] According to a seventh aspect of the embodiments of the present specification, a computing device is provided, including:

[0040] A memory and a processor;

[0041] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the processor executes the computer-executable instructions, the steps of the image processing method are implemented.

[0042] According to an eighth aspect of the embodiments of the present specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of any one of the image processing methods are implemented.

[0043] An embodiment of the present specification implements an image processing method and apparatus. The image processing method includes acquiring a first-moment image and a second-moment image, and determining a first-moment sub-image and a second-moment sub-image based on the first-moment image and the second-moment image; determining a first result at a target moment and a first optical flow map at the target moment based on the first-moment image and the second-moment image, and determining a second result at the target moment and a second optical flow map at the target moment based on the first-moment sub-image and the second-moment sub-image; obtaining an image weight value at the target moment according to the first result, the first optical flow map, the second result, and the second optical flow map; determining a target image at the target moment based on the image weight value, the first result, and the second result, which greatly improves the image processing effect.

[0044] The described image processing method determines the image weight value at the target moment by obtaining the images at the first moment, the images at the second moment, the sub-images at the first moment, and the sub-images at the second moment, determines the target image at the target moment based on the image weight value, and inserts the target image between the images at the first moment and the images at the second moment to increase the video frame rate density of the image, thereby improving the resolution of the video image, so as to achieve a smoother visual effect of the video image in complex motion scenes and during shot transitions. Description of the Drawings

[0045] Figure 1 It is a specific example diagram of an image processing method applied to the video frame interpolation scenario provided by an embodiment of this specification;

[0046] Figure 2 It is a flowchart of the first image processing method provided by an embodiment of this specification;

[0047] Figure 3A It is a flowchart of determining the first result and the first optical flow map at the target moment in the video frame interpolation scenario for an image processing method provided by an embodiment of this specification;

[0048] Figure 3B It is a flowchart of determining the second result and the second optical flow map at the target moment in the video frame interpolation scenario for an image processing method provided by an embodiment of this specification;

[0049] Figure 3C It is a flowchart of determining the image weight value in the video frame interpolation scenario for an image processing method provided by an embodiment of this specification;

[0050] Figure 4A It is a flowchart of performing defect detection in the video frame interpolation scenario for an image processing method provided by an embodiment of this specification;

[0051] Figure 4B It is a flowchart of the image processing method in the video frame interpolation scenario for an image processing method provided by an embodiment of this specification;

[0052] Figure 5 It is a flowchart of the second image processing method provided by an embodiment of this specification;

[0053] Figure 6 It is a flowchart of the third image processing method provided by an embodiment of this specification;

[0054] Figure 7 It is a schematic structural diagram of the first image processing device provided by an embodiment of this specification;

[0055] Figure 8It is a schematic structural diagram of a second image processing device provided by an embodiment of this specification;

[0056] Figure 9 It is a schematic structural diagram of a third image processing device provided by an embodiment of this specification;

[0057] Figure 10 It is a structural block diagram of a computing device provided by an embodiment of this specification. Detailed implementation manners

[0058] Many specific details are set forth in the following description in order to provide a thorough understanding of this specification. However, this specification can be implemented in many other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of this specification. Therefore, this specification is not limited by the specific implementations disclosed below.

[0059] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a", "the", and "said" used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more of the associated listed items.

[0060] It should be understood that although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0061] First, the noun terms related to one or more embodiments of this specification are explained.

[0062] Video frame interpolation: Generating new video frames using the information of the previous and next frames, thereby increasing the video frame rate and improving the smoothness of the video playback.

[0063] Defect detection: Detecting the interpolation result and identifying the video frames with obvious interpolation defects.

[0064] Optical flow: The amount of movement of the pixel points representing the same object or object in the current frame of the video to the next frame.

[0065] Attention model: It can be implemented through deep network learning or other means. As long as it predicts a weight map, it can be considered an attention model.

[0066] In this specification, an image processing method is provided. This specification also relates to an image processing apparatus, a computing device, and a computer-readable storage medium, which will be described in detail one by one in the following embodiments.

[0067] In the video frame interpolation scenario, in some complex motion scenarios and transitions of video images in a video acquisition terminal, the video image resolution is low, resulting in a poor visual experience for users. Based on this, one or more embodiments of this specification provide an image processing method to solve the above technical problems. However, in actual applications, one or more embodiments of this specification provide an image processing method that can be applied not only to the video frame interpolation scenario but also to any scenario where the quality of images and animated images is improved, such as the scenario of improving the resolution of animated images, etc. This application makes no limitations in this regard.

[0068] For ease of understanding, the following details the image processing in a video in the case of an image processing method applied to the video frame interpolation scenario.

[0069] See Figure 1 , Figure 1 which shows a specific example diagram of an image processing method applied to the video frame interpolation scenario according to an embodiment of this specification.

[0070] Figure 1 The application scenario of includes an image acquisition terminal 102 and a server 104. Specifically, the image acquisition terminal 102 acquires the first-moment image and the second-moment image of any video, where the first-moment image is the i-th frame image of the video, and the second-moment image is the (i + 1)-th frame image of the video, and sends the first-moment image and the second-moment image to the server 104. The server 104 receives the first-moment image and the second-moment image, and performs downsampling processing on the first-moment image and the second-moment image to obtain a first-moment sub-image and a second-moment sub-image. Among them, the first-moment sub-image is a small-scale image obtained by downsampling the first-moment image, and the second-moment sub-image is a small-scale image obtained by downsampling the second-moment image; then the first-moment image and the second-moment image are input into an image frame interpolation model to obtain a first result at the target moment and a first optical flow map. Among them, the first result is a large-scale frame interpolation result at the target moment, and the first optical flow map is the optical flow map of the large-scale image. Then, the first-moment sub-image and the second-moment sub-image are input into the value image frame interpolation model to obtain a second result at the target moment and a second optical flow map. Among them, the second result is a small-scale frame interpolation result at the target moment, and the second optical flow map is the optical flow map of the small-scale image.

[0071] At this time, the server 104 inputs the obtained first result, the first optical flow map, the second result, and the second optical flow map into a pre-trained attention model to obtain the attention weight value at the target moment. Then, the server 104 performs multi-scale fusion processing based on the obtained attention weight value at the target moment, the first result at the target moment, and the second result at the target moment to obtain the target image at the target moment. Among them, the target image at the target moment is an image at a certain moment between the i-th frame image and the (i + 1)-th frame image.

[0072] In the embodiment of the present specification, for two consecutive frame images, first perform downsampling processing to obtain small-scale images with a scale lower than the original scale images, then input the large-scale two consecutive frame images and the small-scale two consecutive frame images into an image interpolation model respectively to obtain the large-scale interpolation result and the small-scale interpolation result at the target moment, and fuse the large-scale interpolation result and the small-scale interpolation result through the attention weight value obtained by the attention model to obtain the target image at the target moment after fusion, and insert the target image in the middle of the two consecutive frame images, so that the number of frames of the video image increases, making the video image smoother and improving the visual effect.

[0073] See Figure 2 , Figure 2 shows a flowchart of a first image processing method provided according to an embodiment of the present specification, including the following steps:

[0074] Step 202: Obtain the image at the first moment and the image at the second moment, and determine the sub-image at the first moment and the sub-image at the second moment based on the image at the first moment and the image at the second moment.

[0075] Among them, the image at the first moment and the image at the second moment can be any two consecutive frame images in any video. Among them, any video includes entertainment videos, movie videos, news videos, etc. For example, in an entertainment video, the image at the first moment and the image at the second moment obtained are two consecutive frame images in the entertainment video.

[0076] Specifically, when implementing, the determining the sub-image at the first moment and the sub-image at the second moment based on the image at the first moment and the image at the second moment includes:

[0077] Perform image scale processing on the image at the first moment and the image at the second moment to obtain the sub-image at the first moment and the sub-image at the second moment.

[0078] In practical applications, image scale processing is performed on the image at the first moment and the image at the second moment. The image at the first moment and the image at the second moment are downsampled to obtain the sub-image at the first moment and the sub-image at the second moment. Among them, the image scale processing can be achieved by downsampling based on image scaling, or can be achieved by bilinear interpolation. The downsampling process is performed on the image at the first moment and the image at the second moment based on the above two image scale processing methods. That is to say, the image at the first moment and the image at the second moment obtained from the video acquisition terminal are continuous images of the original video scale, while the sub-image at the first moment and the sub-image at the second moment obtained by downsampling are continuous images of a small scale. Among them, the continuous images of the original scale can also be called large-scale images relative to the continuous images of the small scale.

[0079] In specific implementation, among the two consecutive frames of images obtained, the image at the first moment is the i-th frame image, the image at the second moment is the (i + 1)-th frame image, and the target image at the target moment is the image in the middle of the i-th frame image and the (i + 1)-th frame image.

[0080] In practical applications, the i-th frame image and the (i + 1)-th frame image are obtained, and the i-th frame image and the (i + 1)-th frame image are downsampled to obtain the i-th frame sub-image and the (i + 1)-th frame sub-image. Among them, the i-th frame sub-image and the (i + 1)-th frame sub-image are small-scale images corresponding to the i-th frame image and the (i + 1)-th frame image obtained. For example, the 0-th frame image and the 1-st frame image are obtained, and the 0-th frame image and the 1-st frame image are respectively downsampled to obtain the small-scale 0-th frame sub-image and 1-st frame sub-image. Among them, the downsampling process can be to perform scaling processing on the length and width of the image. For example, an image with a resolution of 1080P is scaled by half to obtain an image with a resolution of 540P, or the original image is processed by bilinear interpolation to obtain the downsampled image.

[0081] The image processing method provided in the embodiments of this specification downsamples two consecutive frames of images in the original video to obtain small-scale images after downsampling, which is beneficial to subsequently inputting the original two frames of images and the small-scale images into the image interpolation model to obtain the first result, the first optical flow map, the second result, and the second optical flow map. The small-scale images after downsampling will greatly improve the processing speed during the image interpolation process.

[0082] Step 204: Determine the first result at the target moment and the first optical flow map at the target moment based on the image at the first moment and the image at the second moment, and determine the second result at the target moment and the second optical flow map at the target moment based on the sub-image at the first moment and the sub-image at the second moment.

[0083] Among them, the target image at the target moment is the image in the middle of the i-th frame image and the (i + 1)-th frame image, the first result at the target moment is the first result of the image at the target moment in the middle of the i-th frame image and the (i + 1)-th frame image, and the first optical flow map at the target moment is the first optical flow map of the image at the target moment in the middle of the i-th frame image and the (i + 1)-th frame image.

[0084] Specifically, the determining of the first result at the target moment and the first optical flow map at the target moment based on the image at the first moment and the image at the second moment includes:

[0085] Input the image at the first moment and the image at the second moment into an image processing model to obtain the first-scale interpolation result at the target moment and the first-scale optical flow map at the target moment.

[0086] In specific implementation, the image processing model can be a variety of different models, as long as it is a model based on an optical flow estimation-based interpolation algorithm. Inputting the image at the first moment and the image at the second moment into the image processing model for interpolation calculation can obtain the first-scale interpolation result of the interpolated image at any moment between the image at the first moment and the image at the second moment and the first-scale optical flow map. The first-scale interpolation result is the large-scale interpolation result based on the image at the first moment and the image at the second moment, and the first-scale optical flow map is the large-scale optical flow estimation result based on the image at the first moment and the image at the second moment.

[0087] The image processing method provided by the embodiments of this specification obtains the first-scale interpolation result at the target moment and the first-scale optical flow map. Among them, the first-scale optical flow map is the motion estimation map of the large-scale image in a complex motion scene. Based on the first-scale interpolation result at the target moment and the first-scale optical flow map, subsequent calculations are carried out to obtain a more accurate target image for interpolation processing, increasing the number of frames of the video image, making the video image frame smoother, and presenting a better visual effect.

[0088] In addition, specifically, the determining of the second result at the target moment and the second optical flow map at the target moment based on the sub-image at the first moment and the sub-image at the second moment includes:

[0089] Input the sub-image at the first moment and the sub-image at the second moment into the image processing model to obtain the second-scale interpolation result at the target moment and the second-scale optical flow map at the target moment.

[0090] In specific implementation, the image processing model can be a variety of different models, as long as it is a model based on the optical flow estimation-based frame interpolation algorithm. Input the sub-images at the first moment and the sub-images at the second moment into the image processing model for frame interpolation calculation, and the second-scale frame interpolation result and the second-scale optical flow map of the frame interpolation image at any moment between the sub-images at the first moment and the sub-images at the second moment can be obtained. The second-scale frame interpolation result is based on the small-scale frame interpolation results of the sub-images at the first moment and the sub-images at the second moment, and the second-scale optical flow map is based on the small-scale optical flow estimation results of the sub-images at the first moment and the sub-images at the second moment.

[0091] In practical applications, following the above example, input the i-th frame image and the (i + 1)-th frame image into the image interpolation model for frame interpolation calculation, and the large-scale frame interpolation result and the large-scale optical flow map of the frame interpolation image at any moment between the i-th frame image and the (i + 1)-th frame image can be obtained. Among them, the frame interpolation image at any moment is the target image at the target moment between the i-th frame image and the (i + 1)-th frame image; input the i-th sub-image and the (i + 1)-th sub-image into the image interpolation model for frame interpolation calculation, and the small-scale frame interpolation result and the small-scale optical flow map of the frame interpolation image at any moment between the i-th sub-image and the (i + 1)-th sub-image can be obtained.

[0092] The image processing method provided by the embodiments of this specification, by obtaining the first-scale frame interpolation result and the first-scale optical flow map at the target moment, where the first-scale optical flow map is the motion estimation map of the large-scale image in a complex motion scene, based on the first-scale frame interpolation result and the first-scale optical flow map at the target moment, to achieve subsequent calculations to obtain a more accurate target image for frame interpolation processing, increase the frame rate of the video image, make the video image more smooth, and show a better visual effect.

[0093] Step 206: Obtain the image weight value at the target moment according to the first result, the first optical flow map, the second result, and the second optical flow map.

[0094] Among them, the first result is the large-scale frame interpolation result at the target moment obtained based on the image at the first moment and the image at the second moment, the first optical flow map is the first optical flow estimation motion map of the target moment image, the second result is the small-scale frame interpolation result at the target moment obtained based on the sub-image at the first moment and the sub-image at the second moment, and the second optical flow map is the second optical flow estimation motion map of the target moment image.

[0095] The image processing method provided by the embodiments of this specification, based on the image weight value at the target moment, more accurately determines the specific result of the frame interpolation image at the target moment.

[0096] Specifically, obtaining the image weight value at the target moment based on the first result, the first optical flow map, the second result, and the second optical flow map includes:

[0097] Input the first-scale interpolation result, the first-scale optical flow map, the second-scale interpolation result, and the second-scale optical flow map into an attention model to obtain the image weight value at the target moment.

[0098] In specific implementation, input the first result, the first optical flow map, the second result, and the second optical flow map into a pre-trained attention model. Among them, the attention model can be implemented through deep network learning or other methods. As long as it predicts a weight map, it can be considered an attention model. In the attention model, the image weight value at the target moment can be obtained according to the above four parameters, where the image weight value is the predicted weight value.

[0099] In practical applications, input the large-scale interpolation result and large-scale optical flow map of the interpolation image at any moment between the i-th frame image and the (i + 1)-th frame image, and the small-scale interpolation result and small-scale optical flow map of the interpolation image at any moment between the i-th sub-image and the (i + 1)-th sub-image into a pre-trained attention model. Through inference calculation, the attention weight value can be obtained. This attention weight value can represent the relative proportion value of the target image at the target moment relative to the i-th frame image and the (i + 1)-th frame image; in relatively complex motion scenarios and transitions, there may be no corresponding relationship between the i-th frame image and the (i + 1)-th frame image before and after, and the image differences between two consecutive frames may be very large. If the existing technology is used to copy frames for image interpolation processing, or at positions such as shot switching and transitions, since there is no corresponding relationship between the front and back frames, generally video interpolation is not required, and traditional algorithms cannot detect it, which will cause the image to become very blurred and the visual experience is extremely poor. Therefore, the image processing method provided in the embodiments of this specification obtains the attention weight value through the attention model, which can more accurately determine the target image after interpolation at the target moment. Inserting the target image into two consecutive frames of images can effectively improve the image interpolation effect in complex motion scenarios.

[0100] Step 208: Determine the target image at the target moment based on the image weight value, the first result, and the second result.

[0101] Among them, the image weight value is the image weight value of the interpolation image at the target moment, the first result is the large-scale interpolation result at the target moment obtained from the first-moment image and the second-moment image, the second result is the small-scale interpolation result at the target moment obtained from the first-moment sub-image and the second-moment sub-image, and the determined target image at the target moment is the image to be interpolated at any moment between the first moment and the second moment.

[0102] In specific implementation, the first result is the large-scale interpolation result of the (i + t)-th frame image at the target time t obtained from the i-th frame image and the (i + 1)-th frame image, the second result is the small-scale interpolation result of the (i + t)-th sub-image at the target time t obtained from the i-th sub-image and the (i + 1)-th sub-image, and the image weight value is the image weight value of the (i + t)-th frame image. Here, the image weight value can be represented by a numerical value A, and the numerical value A can be the vector matrix value of the target image. First, perform upsampling on the small-scale interpolation result of the (i + t)-th sub-image to obtain an interpolation result with the same resolution as the large-scale interpolation result of the (i + t)-th frame image. Then, multiply the interpolation result with the same resolution as the large-scale interpolation result of the (i + t)-th frame image by the value obtained by adding 1 to the opposite of the image weight value A, and then add the product of the large-scale interpolation result of the (i + t)-th frame image and the image weight value A, thereby obtaining the interpolation result of the target image at the target time.

[0103] In practical applications, for example, the formula for fusing interpolation results of different scales using the image weight value can be I t = R t * A + UP[R - S t * (1 - A), where the numerical value A is the image weight value, R t is the first result of the large-scale image, R - S t is the second result of the small-scale image, UP[R - S t is the upsampling process for the second result of the large-scale image, and I t is the final interpolation result, and thus the interpolation image at the target time t interpolated between the image at the first moment and the image at the second moment can be obtained.

[0104] The image processing method provided in the embodiments of this specification uses the fusion of interpolation results of different scales through an attention model to obtain the target image at the target time, inserts the target image between two consecutive frames of images, increases the frame rate of the video image, and effectively improves the interpolation effect in cases of large-scale motion, lens switching, transitions, etc.

[0105] Further, determining the target image at the target time based on the image weight value, the first result, and the second result includes:

[0106] Judging whether the image weight value at the target time meets a first preset threshold;

[0107] If so, determine the initial image at the target time based on the image weight value at the target time, the first result at the target time, and the second result at the target time, and use the initial image as the target image at the target time;

[0108] Otherwise, determine the initial image of the target moment based on the image weight value of the target moment, the first result of the target moment, and the second result of the target moment, and determine the target image of the target moment based on the initial image of the target moment, the image of the first moment, and the image of the second moment.

[0109] Specifically, the image weight value can be a weight value calculated based on a vector matrix. The first preset threshold can be preset according to multiple experimental situations. Determine whether the image weight value of the target moment meets the first preset threshold. If so, determine the initial image of the target moment according to the image weight value of the target image, the first result of the target moment, and the second result of the target moment, and use the initial image as the target image of the target moment for the final output and the interpolation image for the final image processing; if not, after determining the initial image of the target moment, determine the target image of the target moment based on the initial image of the target moment, the image of the first moment, and the image of the second moment as the final interpolation image.

[0110] The image processing method provided by the embodiments of this specification determines whether there are defects in the interpolation result by judging the image weight value, so as to effectively improve the interpolation effect under large-amplitude motion and effectively filter out the situation of interpolation failure caused by lens switching.

[0111] Further, the determining the initial image of the target moment based on the image weight value of the target moment, the first result of the target moment, and the second result of the target moment includes:

[0112] Perform image scale processing on the second result of the target moment to obtain the image processing result of the target moment;

[0113] Determine the initial image of the target moment based on the image weight value of the target moment, the first result of the target moment, and the image processing result of the target moment.

[0114] Specifically, the second result of the target moment is the interpolation result of the target moment of the downsampled small-scale image. Perform upsampling processing on the interpolation result to obtain the image processing result of the target moment. The image processing result of the target moment is the interpolation result of the scale corresponding to the large scale of the original video image. Determine the initial image of the target moment based on the image weight value of the target moment, the first result of the target moment, and the image processing result of the target moment through the implementation formula.

[0115] The image processing method provided by the embodiments of this specification realizes the fusion of interpolation results at different scales based on image weight values, obtains the target image at the target moment accurately and inserts it between two consecutive frames, improves the frame rate of video images, and can effectively improve the video interpolation effect in complex motion scenes.

[0116] Further, determining the target image at the target moment based on the initial image at the target moment, the first image at the target moment, and the second image at the target moment includes:

[0117] Judging whether the target moment in the values of the initial image at the target moment meets a second preset threshold,

[0118] If so, determining the second image as the target image at the target moment;

[0119] If not, determining the first image as the target image at the target moment.

[0120] Among them, when inserting the obtained target image at the target moment between two frames, if the two frames are discontinuous frames under a transition shot, it will cause the interpolation of the target image at the target moment to fail. In order to further reduce the occurrence of interpolation failure, a defect detection module for the target image at the target moment after image interpolation is added. The defect detection module can detect whether the multiple frames of images obtained after inserting the target image at the target moment between two frames are smooth. If the multiple frames of images are smooth, the target image at the target moment can be output. If the multiple frames of images are blurred, one of the two frames is selected as the target image at the target moment. The defect detection module can effectively filter out the situation of interpolation failure caused by lens switching. The image weight value at the target moment can be a value calculated based on a vector matrix. Taking the calculation of the image weight value as an average value as an example, the image weight value is compared with a preset threshold.

[0121] Specifically, it is judged whether the target image obtained after fusion has defects. If there are no defects, the target image at the target moment is inserted between the image at the first moment and the image at the second moment as the final interpolated image; if it is judged that the target image has defects, or inserting the target image will cause the image to be blurred or deformed, the target image at the target moment inserted between the image at the first moment and the image at the second moment can be the image at the first moment or the image at the second moment.

[0122] In specific implementation, for example, the image at the first moment is the 0th frame image, the image at the second moment is the 1st frame image, and finally, an interpolated image at any moment t between the 0th frame and the 1st frame is output. If the image weight value is A, it is determined whether the image weight value A is greater than a first preset threshold. If the image weight value A is greater than the first preset threshold, the fused target image described above is used as the interpolated image at moment t of the final output. If the image weight value is less than or equal to the first preset threshold, it indicates that the fused target image is defective and unavailable. Further, it is determined whether moment t is greater than a second preset threshold. If the intermediate moment t is greater than the second preset threshold, the image at the second moment is finally output as the final interpolated image. If the intermediate moment t is less than or equal to the second preset threshold, the image at the first moment is finally output as the final interpolated image.

[0123] The image processing method provided in the embodiments of this specification is based on an optical flow-based interpolation model. According to the pixel correspondence relationship between the front and back frames, the fusion of interpolation results at different scales is realized through an attention model, effectively improving the interpolation effect under large-scale motion. In order to ensure the high resolution of the video, a defect detection module is introduced to effectively filter out the situation of interpolation failure caused by lens switching, so as to increase the frame rate of the existing high-definition or standard-definition video.

[0124] See Figure 3A , Figure 3A FIG. is a flowchart for determining the first result and the first optical flow map at the target moment in a video interpolation scenario for an image processing method provided in an embodiment of this specification, which specifically includes the following steps:

[0125] Step 302: Obtain the image at the first moment and the image at the second moment.

[0126] Specifically, the image at the first moment is the ith frame image, and the image at the second moment is the (i + 1)th frame image.

[0127] Step 304: Input the obtained image at the first moment and the image at the second moment into the image processing model.

[0128] Specifically, the image processing model can be various different models, as long as it is a model based on an optical flow estimation-based interpolation algorithm. The obtained image at the first moment and the image at the second moment are input into the image processing model for interpolation calculation.

[0129] Step 306: Calculate and output the first result and the first optical flow map at the target moment.

[0130] Specifically, based on the optical flow estimation-based interpolation algorithm, calculate the first-scale interpolation result and the first optical flow map at the target moment between the image at the first moment and the image at the second moment.

[0131] The image processing method provided by the embodiments of this specification obtains a first-scale interpolation result and a first optical flow map by inputting the image at the first moment and the image at the second moment into an image processing model for interpolation calculation, so as to facilitate subsequent input into an attention model to obtain an image weight value, and determine the target image at the target moment based on the image weight value, and can output a smoother interpolated image.

[0132] See Figure 3B , Figure 3B FIG. is a flowchart of determining a second result and a second optical flow map at a target moment in a video interpolation scenario for an image processing method provided by an embodiment of this specification, which specifically includes the following steps:

[0133] Step 312: Obtain the image at the first moment and the image at the second moment.

[0134] Specifically, the image at the first moment is the i-th frame image, and the image at the second moment is the (i + 1)-th frame image.

[0135] Step 314: Perform image scale processing on the image at the first moment and the image at the second moment to obtain a sub-image at the first moment and a sub-image at the second moment.

[0136] Specifically, the image scale processing can perform image scaling by downsampling or perform image scale processing by bilinear interpolation, and perform downsampling processing on the image at the first moment and the image at the second moment to obtain corresponding small-scale sub-images at the first moment and sub-images at the second moment.

[0137] Step 316: Input the sub-image at the first moment and the sub-image at the second moment into the image processing model.

[0138] Specifically, the sub-image at the first moment and the sub-image at the second moment are respectively small-scale images corresponding to the image at the first moment and the image at the second moment, and the image processing model can support various different types of models, as long as it is an interpolation algorithm based on optical flow estimation can be implemented.

[0139] Step 318: Calculate and output a second result and a second optical flow map at the target moment.

[0140] Specifically, the second result at the target moment is a small-scale interpolation result at the target moment, and the second optical flow map is a small-scale optical flow map at the target moment.

[0141] The image processing method provided by the embodiments of this specification obtains a small-scale interpolation result and a small-scale optical flow map through an image processing model, which is beneficial to subsequent obtaining of the attention weight value at the target moment, so as to facilitate subsequent determination of the target image at the final target moment according to the attention weight value, and improve the interpolation effect of video images.

[0142] See Figure 3C , Figure 3C is a flowchart for determining an image weight value in a video frame interpolation scenario for an image processing method provided by an embodiment of this specification, specifically including the following steps:

[0143] Step 322: Obtain the first result, the first optical flow map, the second result, and the second optical flow map at the target moment.

[0144] Step 324: Input the first result, the first optical flow map, the second result, and the second optical flow map into the attention model.

[0145] Specifically, input the first result, the first optical flow map, the second result, and the second optical flow map into a pre-trained attention model. Among them, the attention model can be implemented through deep network learning or other means. As long as it predicts a weight map, it can be considered an attention model.

[0146] Step 326: Obtain the image weight value through inference calculation.

[0147] Specifically, in the attention model, the image weight value at the target moment can be obtained according to the above four parameters. Among them, the image weight value is the value of the predicted weight.

[0148] For the image processing method provided by the embodiment of this specification, by obtaining the attention weight value through the attention model, it can further determine the target image after frame interpolation at the target moment, and can effectively improve the image frame interpolation effect in complex motion scenarios.

[0149] See Figure 4A , Figure 4A shows a flowchart for defect detection in a video frame interpolation scenario for an image processing method provided by an embodiment of this specification, specifically including the following steps:

[0150] Step 402: Obtain the image weight value.

[0151] Specifically, based on the first-scale frame interpolation result, the first-scale optical flow map, the second-scale frame interpolation result, and the second-scale optical flow map input into the attention model, obtain the image weight value at the target moment.

[0152] Step 404: Determine whether the image weight value exceeds a first preset threshold.

[0153] Specifically, the image weight value can be the image weight value of the interpolated image at the target moment. Among them, the image weight value can be the weight value calculated based on the vector matrix. Determine whether the image weight value exceeds the first preset threshold, and the first preset threshold can be preset according to multiple experimental situations.

[0154] Step 406: If so, the target image is flawless, and output the target image at the target time.

[0155] Specifically, if so, determine the initial image at the target time according to the image weight value of the target image, the first result at the target time, and the second result at the target time, and use the initial image as the target image at the target time for final output, which is the interpolation image for final image processing.

[0156] Step 408: If not, the target image has a flaw, and determine whether the target time exceeds the second preset threshold.

[0157] Specifically, if not, after determining the initial image at the target time, determine the target image at the target time based on the initial image at the target time, the image at the first time, and the image at the second time, as the final interpolation image.

[0158] Step 410: Output the image at the second time.

[0159] Specifically, if it is determined that the target image has a flaw, or inserting the target image will cause the image to be blurred or deformed, then the target image at the target time inserted between the image at the first time and the image at the second time can adopt the image at the first time or the image at the second time.

[0160] In specific implementation, if the target time of the intermediate time is greater than the second preset threshold, finally output the image at the second time as the final interpolation image.

[0161] Step 412: Output the image at the first time.

[0162] Specifically, if the target time of the intermediate time is less than or equal to the second preset threshold, finally output the image at the first time as the final interpolation image.

[0163] The image processing method provided by the embodiments of this specification is based on an optical flow-based interpolation model. According to the pixel correspondence relationship between the front and back frames, the fusion of interpolation results at different scales is realized through an attention model. The obtained fusion frame is inserted into two consecutive frames of images, effectively improving the interpolation effect under large-scale motion. By introducing a flaw detection module, the situation of interpolation failure caused by lens switching is effectively filtered to increase the frame rate of the existing high-definition or standard-definition video.

[0164] See Figure 4B , Figure 4B shows a flowchart of an image processing method in a video interpolation scenario according to an embodiment of this specification, which specifically includes the following steps:

[0165] Step 412: Input video frames, where the video frames are two consecutive frames in a video image.

[0166] Specifically, the input video frames correspond to obtaining the image at the first moment and the image at the second moment in the embodiments of this specification.

[0167] Step 414: Perform downsampling processing on the video frames.

[0168] Specifically, the downsampling processing correspondingly performs image scaling scale processing on the obtained image at the first moment and the image at the second moment in the embodiments of this specification, or performs downsampling processing through bilinear interpolation to obtain the sub-image at the first moment and the sub-image at the second moment with a small scale.

[0169] Step 416: Input the downsampled sub-image at the first moment and the sub-image at the second moment into the video frame interpolation model.

[0170] Specifically, the video frame interpolation model corresponds to the image processing model in the embodiments of this specification. Input the sub-image at the first moment and the sub-image at the second moment into the image processing model to obtain the interpolation result at the second scale and the second optical flow map. The optical flow map is the second optical flow map in the embodiments of this specification, and the interpolation result at the small scale is the interpolation result at the second scale in the embodiments of this specification.

[0171] Step 418: Input the video frames into the video frame interpolation model.

[0172] Specifically, the video frame interpolation model correspondingly is the image processing model in the embodiments of this specification. Input the image at the first moment and the image at the second moment into the image processing model to obtain the interpolation result at the first scale and the first optical flow map. The optical flow map is the first optical flow map in the embodiments of this specification, and the interpolation result at the large scale is the interpolation result at the first scale in the embodiments of this specification.

[0173] Step 420: Input the small-scale optical flow map, the small-scale interpolation result, the large-scale optical flow map, and the large-scale interpolation result into the attention model to obtain the attention weight map.

[0174] Step 422: Perform multi-scale fusion on the attention weight map, the large-scale interpolation result, and the small-scale interpolation result to obtain the fused frame at the target moment.

[0175] Specifically, the attention weight map corresponds to the attention weight value in the embodiments of this specification, the large-scale interpolation result is the first result in the embodiments of this specification, and the small-scale interpolation result is the second result in the embodiments of this specification.

[0176] Step 424: Perform defect detection and judgment according to the attention weight map.

[0177] Specifically, the attention weight map corresponds to the attention weight value of the embodiment of this specification. According to the distribution of the attention weight value, it is determined whether there are defects in the fused target image.

[0178] Step 426: If there are no defects in the fused frame at the obtained target time, output the fused frame at the target time.

[0179] Specifically, the fused frame at the target time corresponds to the target image at the target time in the embodiment of this specification.

[0180] Step 428: If there are defects in the fused frame at the obtained target time, output the current frame.

[0181] Specifically, the current frame corresponds to the first moment image or the second moment image in the embodiment of this specification.

[0182] The image processing method provided by the embodiment of this specification is based on an optical flow interpolation model. According to the pixel correspondence relationship between the front and back frames, the fusion of interpolation results at different scales is realized through an attention model. The obtained fused frame is inserted into two consecutive frames of images, effectively improving the interpolation effect under large-scale motion. By introducing a defect detection module, the situation of interpolation failure caused by lens switching is effectively filtered to increase the frame rate of the existing high-definition or standard-definition video.

[0183] See Figure 5 , Figure 5 shows a flowchart of a second image processing method provided according to an embodiment of this specification, including the following steps:

[0184] Step 502: Display an image input interface for the user based on the user's image processing request.

[0185] Step 504: Obtain the first moment image and the second moment image input by the user based on the image input interface, and determine the first moment sub-image and the second moment sub-image based on the first moment image and the second moment image.

[0186] Step 506: Determine the first result and the first optical flow map at the target time based on the first moment image and the second moment image, and determine the second result and the second optical flow map at the target time based on the first moment sub-image and the second moment sub-image.

[0187] Step 508: Obtain the image weight value at the target time according to the first result, the first optical flow map, the second result, and the second optical flow map.

[0188] Step 510: Determine the target image at the target moment based on the image weight value, the first result, and the second result, and return the target image at the target moment to the user.

[0189] It should be noted that for the parts corresponding to the embodiments of the first image processing method in the second image processing method provided in the embodiments of this specification, reference may be made to the detailed descriptions in the embodiments of the first image processing method above, and details will not be repeated here.

[0190] The image processing method provided in the embodiments of this specification is based on an optical flow interpolation model. According to the pixel correspondence relationship between the front and rear frames, the fusion of interpolation results at different scales is achieved through an attention model, effectively improving the interpolation effect under large-scale motion. By introducing a defect detection module, the situation of interpolation failure caused by lens switching is effectively filtered to increase the frame rate of existing high-definition or standard-definition videos.

[0191] See Figure 6 , Figure 6 shows a flowchart of a third image processing method provided according to an embodiment of this specification, including the following steps:

[0192] Step 602: Receive an image processing request sent by the user carrying the image at the first moment and the image at the second moment.

[0193] Step 604: Obtain the image at the first moment and the image at the second moment, and determine the sub-image at the first moment and the sub-image at the second moment based on the image at the first moment and the image at the second moment.

[0194] Step 606: Determine the first result at the target moment and the first optical flow map at the target moment based on the image at the first moment and the image at the second moment, and determine the second result at the target moment and the second optical flow map at the target moment based on the sub-image at the first moment and the sub-image at the second moment.

[0195] Step 608: Obtain the image weight value at the target moment according to the first result, the first optical flow map, the second result, and the second optical flow map.

[0196] Step 610: Determine the target image at the target moment based on the image weight value, the first result, and the second result, and return the target image at the target moment to the user.

[0197] It should be noted that for the parts corresponding to the embodiments of the first image processing method in the third image processing method provided in the embodiments of this specification, reference may be made to the detailed descriptions in the embodiments of the first image processing method above, and details will not be repeated here.

[0198] The image processing method provided by the embodiments of this specification is based on an optical flow interpolation model. According to the pixel correspondence between the front and rear frames, the fusion of interpolation results at different scales is achieved through an attention model, effectively improving the interpolation effect under large-scale motion. And by introducing a defect detection module, the situation of interpolation failure caused by lens switching is effectively filtered to increase the frame rate of existing high-definition or standard-definition videos.

[0199] Corresponding to the above method embodiments, this specification also provides embodiments of a detection device. Figure 7 The structural schematic diagram of the first detection device provided by an embodiment of this specification is shown. As Figure 7 shown, the device includes:

[0200] An acquisition module 702, configured to acquire an image at a first moment and an image at a second moment, and determine a sub-image at the first moment and a sub-image at the second moment based on the image at the first moment and the image at the second moment;

[0201] A first determination module 704, configured to determine a first result at a target moment and a first optical flow map at the target moment based on the image at the first moment and the image at the second moment, and determine a second result at the target moment and a second optical flow map at the target moment based on the sub-image at the first moment and the sub-image at the second moment;

[0202] An obtaining module 706, configured to obtain an image weight value at the target moment according to the first result, the first optical flow map, the second result, and the second optical flow map;

[0203] A second determination module 708, configured to determine a target image at the target moment based on the image weight value, the first result, and the second result.

[0204] Optionally, the acquisition module 702 is further configured to:

[0205] Perform image scale processing on the image at the first moment and the image at the second moment to obtain a sub-image at the first moment and a sub-image at the second moment.

[0206] Optionally, the first determination module 704 is further configured to:

[0207] Input the image at the first moment and the image at the second moment into an image processing model to obtain a first-scale interpolation result at the target moment and a first-scale optical flow map at the target moment.

[0208] Optionally, the first determination module 704 is further configured to:

[0209] Input the sub-images at the first moment and the sub-images at the second moment into the image processing model to obtain the second-scale interpolation result and the second-scale optical flow map at the target moment.

[0210] Optionally, the obtaining module 706 is further configured to:

[0211] Input the first-scale interpolation result, the first-scale optical flow map, the second-scale interpolation result, and the second-scale optical flow map into the attention model to obtain the image weight value at the target moment.

[0212] Optionally, the second determination module 708 is further configured to:

[0213] Determine whether the image weight value at the target moment meets a first preset threshold;

[0214] If so, determine the initial image at the target moment based on the image weight value at the target moment, the first result at the target moment, and the second result at the target moment, and use the initial image as the target image at the target moment;

[0215] If not, determine the initial image at the target moment based on the image weight value at the target moment, the first result at the target moment, and the second result at the target moment, and determine the target image at the target moment based on the initial image at the target moment, the first image at the target moment, and the second image at the target moment.

[0216] Optionally, the second determination module 708 is further configured to:

[0217] Perform image scale processing on the second result at the target moment to obtain the image processing result at the target moment;

[0218] Determine the initial image at the target moment based on the image weight value at the target moment, the first result at the target moment, and the image processing result at the target moment.

[0219] Optionally, the second determination module 708 is further configured to:

[0220] Perform image scale processing on the second result at the target moment to obtain the image processing result at the target moment;

[0221] Determine the initial image at the target moment based on the image weight value at the target moment, the first result at the target moment, and the image processing result at the target moment.

[0222] Optionally, the device further includes:

[0223] A judgment module, configured to judge whether the target time in the values of the initial image at the target time meets a second preset threshold.

[0224] If so, determine that the target image at the target time is the second image.

[0225] If not, determine that the target image at the target time is the first image.

[0226] Optionally, the image at the first time is the i-th frame image, the image at the second time is the (i + 1)-th frame image, and the target image at the target time is the image in the middle of the i-th frame image and the (i + 1)-th frame image.

[0227] The above is a schematic solution of an image processing device in this embodiment. It should be noted that the technical solution of this image processing device and the technical solution of the first image processing method above belong to the same concept. For the details not described in the technical solution of the image processing device, reference can be made to the description of the technical solution of the first image processing method above.

[0228] Corresponding to the above method embodiment, this specification also provides an embodiment of a detection device. Figure 8 The structural schematic diagram of the first detection device provided by an embodiment of this specification is shown. As Figure 8 shown, the device includes:

[0229] A display module 802, configured to display an image input interface for the user based on the user's image processing request.

[0230] An acquisition module 804, configured to acquire the image at the first time and the image at the second time input by the user based on the image input interface, and determine a sub-image at the first time and a sub-image at the second time based on the image at the first time and the image at the second time.

[0231] A first determination module 806, configured to determine a first result at the target time and a first optical flow map at the target time based on the image at the first time and the image at the second time, and determine a second result at the target time and a second optical flow map at the target time based on the sub-image at the first time and the sub-image at the second time.

[0232] An obtaining module 808, configured to obtain an image weight value at the target time according to the first result, the first optical flow map, the second result, and the second optical flow map.

[0233] A second determination module 810, configured to determine the target image at the target time based on the image weight value, the first result, and the second result, and return the target image at the target time to the user.

[0234] The above is a schematic solution of an image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the second image processing method described above belong to the same concept. For the details not described in detail in the technical solution of the image processing apparatus, reference can be made to the description of the technical solution of the second image processing method above.

[0235] Corresponding to the above method embodiment, this specification also provides an embodiment of an image processing apparatus. Figure 9 It shows a schematic structural diagram of a third image processing apparatus provided by an embodiment of this specification. As Figure 9 shown, the apparatus includes:

[0236] A receiving module 902, configured to receive an image processing request sent by a user and carrying an image at a first moment and an image at a second moment;

[0237] An obtaining module 904, configured to obtain the image at the first moment and the image at the second moment, and determine a sub-image at the first moment and a sub-image at the second moment based on the image at the first moment and the image at the second moment;

[0238] A first determining module 906, configured to determine a first result at a target moment and a first optical flow map at the target moment based on the image at the first moment and the image at the second moment, and determine a second result at the target moment and a second optical flow map at the target moment based on the sub-image at the first moment and the sub-image at the second moment;

[0239] An obtaining module 908, configured to obtain an image weight value at the target moment according to the first result, the first optical flow map, the second result, and the second optical flow map;

[0240] A second determining module 910, configured to determine a target image at the target moment based on the image weight value, the first result, and the second result, and return the target image at the target moment to the user.

[0241] The above is a schematic solution of an image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the third image processing method described above belong to the same concept. For the details not described in detail in the technical solution of the image processing apparatus, reference can be made to the description of the technical solution of the third image processing method above.

[0242] Figure 10A structural block diagram of a computing device 1000 provided according to an embodiment of this specification is shown. The components of the computing device 1000 include, but are not limited to, a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and a database 1050 is used to store data.

[0243] The computing device 1000 further includes an access device 1040, which enables the computing device 1000 to communicate via one or more networks 1060. Examples of these networks include the Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 1040 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, Worldwide Interoperability for Microwave Access (Wi-MAX) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, Near Field Communication (NFC) interface, and so on.

[0244] In an embodiment of this specification, the above components of the computing device 1000 and Figure 10 other components not shown may also be connected to each other, for example, via a bus. It should be understood that Figure 10 the shown structural block diagram of the computing device is for illustrative purposes only and is not a limitation on the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0245] The computing device 1000 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 1000 can also be a mobile or stationary server.

[0246] Among them, the processor 1020 is used to execute the following computer-executable instructions. When the processor executes the computer-executable instructions, the steps of the image processing method are implemented.

[0247] The above is a schematic solution of a computing device in this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above image processing method belong to the same concept. For the details not described in the technical solution of the computing device, reference can be made to the description of the technical solution of the above image processing method.

[0248] An embodiment of this specification also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the image processing method.

[0249] The above is a schematic solution of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above image processing method belong to the same concept. For the details not described in the technical solution of the storage medium, reference can be made to the description of the technical solution of the above image processing method.

[0250] The specific embodiments of this specification are described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0251] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0252] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the embodiments of this specification are not limited by the described order of actions, because according to the embodiments of this specification, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments of this specification.

[0253] In the above embodiments, the descriptions of the various embodiments have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0254] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of the embodiments of the present specification. These embodiments are selected and specifically described in the present specification to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and utilize the present specification. The present specification is only limited by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: Obtaining an image at a first moment and an image at a second moment, and determining a sub-image at the first moment and a sub-image at the second moment based on the image at the first moment and the image at the second moment; Determining a first result at a target moment and a first optical flow map at the target moment based on the image at the first moment and the image at the second moment, and determining a second result at the target moment and a second optical flow map at the target moment based on the sub-image at the first moment and the sub-image at the second moment; Obtaining an image weight value at the target moment according to the first result, the first optical flow map, the second result, and the second optical flow map; Determining a target image at the target moment based on the image weight value, the first result, and the second result includes: determining whether the image weight value at the target moment meets a first preset threshold; If so, determining an initial image at the target moment based on the image weight value at the target moment, the first result at the target moment, and the second result at the target moment, and using the initial image as the target image at the target moment.

2. The image processing method according to claim 1, wherein the determining a sub-image at the first moment and a sub-image at the second moment based on the image at the first moment and the image at the second moment includes: Performing image scale processing on the image at the first moment and the image at the second moment to obtain a sub-image at the first moment and a sub-image at the second moment.

3. The image processing method according to claim 1, wherein the determining a first result at a target moment and a first optical flow map at the target moment based on the image at the first moment and the image at the second moment includes: Inputting the image at the first moment and the image at the second moment into an image processing model to obtain a first scale interpolation result at the target moment and a first scale optical flow map at the target moment.

4. The image processing method according to claim 3, wherein the determining a second result at a target moment and a second optical flow map at the target moment based on the sub-image at the first moment and the sub-image at the second moment includes: Inputting the sub-image at the first moment and the sub-image at the second moment into the image processing model to obtain a second scale interpolation result at the target moment and a second scale optical flow map at the target moment.

5. The image processing method according to claim 4, wherein the obtaining an image weight value at the target moment according to the first result, the first optical flow map, the second result, and the second optical flow map includes: Inputting the first scale interpolation result, the first scale optical flow map, the second scale interpolation result, and the second scale optical flow map into an attention model to obtain an image weight value at the target moment.

6. The image processing method according to claim 1, after determining whether the image weight value at the target moment meets a first preset threshold, further comprising: If not, determine the initial image of the target moment based on the image weight value of the target moment, the first result of the target moment, and the second result of the target moment, and determine the target image of the target moment based on the initial image of the target moment, the image of the first moment, and the image of the second moment.

7. The image processing method according to claim 1, wherein the determining the initial image of the target moment based on the image weight value of the target moment, the first result of the target moment, and the second result of the target moment comprises: Performing image scale processing on the second result of the target moment to obtain the image processing result of the target moment; Determine the initial image of the target moment based on the image weight value of the target moment, the first result of the target moment, and the image processing result of the target moment.

8. The image processing method according to claim 7, wherein the determining the target image of the target moment based on the initial image of the target moment, the image of the first moment, and the image of the second moment comprises: Determine whether the target moment in the values of the initial image of the target moment satisfies a second preset threshold; If so, determine that the image of the second moment is the target image of the target moment; If not, determine that the image of the first moment is the target image of the target moment.

9. The image processing method according to any one of claims 1-8, wherein the image of the first moment is the i-th frame image, the image of the second moment is the (i + 1)-th frame image, and the target image of the target moment is the image between the i-th frame image and the (i + 1)-th frame image.

10. An image processing method, comprising: Displaying an image input interface for the user based on the user's image processing request; Obtaining the image of the first moment and the image of the second moment input by the user based on the image input interface, and determining the sub-image of the first moment and the sub-image of the second moment based on the image of the first moment and the image of the second moment; Determining the first result of the target moment and the first optical flow map of the target moment based on the image of the first moment and the image of the second moment, and determining the second result of the target moment and the second optical flow map of the target moment based on the sub-image of the first moment and the sub-image of the second moment; Obtaining the image weight value of the target moment according to the first result, the first optical flow map, the second result, and the second optical flow map; Determine the target image of the target moment based on the image weight value, the first result, and the second result, and return the target image of the target moment to the user, wherein the determining the target image of the target moment based on the image weight value, the first result, and the second result comprises: determining whether the image weight value of the target moment satisfies a first preset threshold; if so, determining the initial image of the target moment based on the image weight value of the target moment, the first result of the target moment, and the second result of the target moment, and using the initial image as the target image of the target moment.

11. An image processing method, comprising: Receive an image processing request sent by a user, which carries an image of a first moment and an image of a second moment; Obtain the image of the first moment and the image of the second moment, and determine a sub-image of the first moment and a sub-image of the second moment based on the image of the first moment and the image of the second moment; Determine a first result at a target moment and a first optical flow map at the target moment based on the image of the first moment and the image of the second moment, and determine a second result at the target moment and a second optical flow map at the target moment based on the sub-image of the first moment and the sub-image of the second moment; Obtain an image weight value at the target moment according to the first result, the first optical flow map, the second result and the second optical flow map; Determine a target image at the target moment based on the image weight value, the first result and the second result, and return the target image at the target moment to the user, where determining the target image at the target moment based on the image weight value, the first result and the second result includes: judging whether the image weight value at the target moment meets a first preset threshold; if so, determine an initial image at the target moment based on the image weight value at the target moment, the first result at the target moment and the second result at the target moment, and use the initial image as the target image at the target moment.

12. An image processing apparatus, comprising: An obtaining module, configured to obtain an image of a first moment and an image of a second moment, and determine a sub-image of the first moment and a sub-image of the second moment based on the image of the first moment and the image of the second moment; A first determining module, configured to determine a first result at a target moment and a first optical flow map at the target moment based on the image of the first moment and the image of the second moment, and determine a second result at the target moment and a second optical flow map at the target moment based on the sub-image of the first moment and the sub-image of the second moment; An obtaining module, configured to obtain an image weight value at the target moment according to the first result, the first optical flow map, the second result and the second optical flow map; A second determining module, configured to determine a target image at the target moment based on the image weight value, the first result and the second result; The second determining module is further configured to judge whether the image weight value at the target moment meets a first preset threshold; If so, determine an initial image at the target moment based on the image weight value at the target moment, the first result at the target moment and the second result at the target moment, and use the initial image as the target image at the target moment.

13. An image processing apparatus, comprising: A display module, configured to display an image input interface for the user based on the user's image processing request; An obtaining module, configured to obtain an image of a first moment and an image of a second moment input by the user based on the image input interface, and determine a sub-image of the first moment and a sub-image of the second moment based on the image of the first moment and the image of the second moment; A first determination module, configured to determine a first result at a target time and a first optical flow map at the target time based on the image at the first time and the image at the second time, and determine a second result at the target time and a second optical flow map at the target time based on the sub-image at the first time and the sub-image at the second time; An obtaining module, configured to obtain an image weight value at the target time according to the first result, the first optical flow map, the second result, and the second optical flow map; A second determination module, configured to determine a target image at the target time based on the image weight value, the first result, and the second result, and return the target image at the target time to the user; The second determination module is further configured to determine whether the image weight value at the target time meets a first preset threshold; If so, determine an initial image at the target time based on the image weight value at the target time, the first result at the target time, and the second result at the target time, and use the initial image as the target image at the target time.

14. An image processing apparatus, comprising: A receiving module, configured to receive an image processing request carrying an image at a first time and an image at a second time sent by a user; An obtaining module, configured to obtain the image at the first time and the image at the second time, and determine a sub-image at the first time and a sub-image at the second time based on the image at the first time and the image at the second time; A first determination module, configured to determine a first result at a target time and a first optical flow map at the target time based on the image at the first time and the image at the second time, and determine a second result at the target time and a second optical flow map at the target time based on the sub-image at the first time and the sub-image at the second time; An obtaining module, configured to obtain an image weight value at the target time according to the first result, the first optical flow map, the second result, and the second optical flow map; A second determination module, configured to determine a target image at the target time based on the image weight value, the first result, and the second result, and return the target image at the target time to the user; The second determination module is further configured to determine whether the image weight value at the target time meets a first preset threshold; If so, determine an initial image at the target time based on the image weight value at the target time, the first result at the target time, and the second result at the target time, and use the initial image as the target image at the target time.

15. A computing device, comprising: A memory and a processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the processor executes the computer-executable instructions, the steps of the image processing method according to any one of claims 1-9 or 10 or 11 are implemented.

16. A computer-readable storage medium, which stores computer instructions, and when the instructions are executed by a processor, the steps of the image processing method according to any one of claims 1-9 or 10 or 11 are implemented.

Images