Image processing method and device

Abstract

The present application discloses an image processing method and device. When inserting frames according to the optical flow method, this method not only performs optical flow calculations but also depth calculations. Since the depth can indicate the distance between the subject and the lens, the greater the depth, the farther the distance, and the lower the pixel corresponding to the depth should be. Using this principle, the pixels in the optical flow can be strengthened or weakened by the depth, so that the inserted frame image is clearer and more reasonable, and closer to the real frame image, making the video picture smoother and the viewing experience better.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method and device. Background Art

[0002] The optical flow method uses the changes in pixels in the time domain in an image sequence and the correlation between adjacent frames to find the correspondence between the previous frame and the current frame, thereby calculating the motion information of objects between adjacent frames.

[0003] Through the optical flow method, the frame sequence collected at a fixed time can be interpolated (the frame at a certain moment between the previous moment and the current moment is predicted based on the frame at the previous moment and the frame at the current moment), and the video can be restored and synthesized into a smooth and complete frame stream.

[0004] Existing optical flow frame prediction neural networks typically use a linear method to calculate the optical flow passing through a point at time t. This optical flow is then used to wrap the image at time t-1 and the image at time t+1, and finally, the image at time t is fused. However, this method requires high optical flow prediction accuracy. If this accuracy is not high enough, the predicted frame will be very blurry and may even cause the image in the predicted frame to be distorted. Summary of the Invention

[0005] The applicant creatively provides an image processing method and device.

[0006] According to a first aspect of an embodiment of the present application, an image processing method is provided, the method comprising: performing an optical flow operation on a first frame image at time t-1 and a second frame image at time t+1 to obtain a first optical flow and a second optical flow, respectively, where t is a natural number, wherein the first optical flow is the optical flow from the frame image at time t-1 to the frame image at time t+1, and the second optical flow is the optical flow from the frame image at time t+1 to the frame image at time t-1; performing a depth operation on the first frame image and the second frame image to obtain a first depth and a second depth, respectively; determining a first predicted frame image based on the first frame image, the first optical flow and the first depth; determining a second predicted frame image based on the second frame image, the second optical flow and the second depth; and synthesizing the first predicted frame image and the second predicted frame image to obtain a third predicted frame image.

[0007] According to an implementation manner of an embodiment of the present application, a depth operation is performed based on the first frame image and the second frame image to obtain a first depth and a second depth respectively, including: obtaining the first depth and the second depth based on the first frame image at time t-1, the second frame image at time t+1 and a depth network model.

[0008] According to an implementation manner of an embodiment of the present application, a first predicted frame image is determined based on a first frame image, a first optical flow and a first depth, including: obtaining a fourth predicted frame image based on the first frame image and the first optical flow; performing a monotonically opposite function operation on the first depth to obtain the probability of occurrence of each pixel point; and determining the first predicted frame image based on the fourth predicted frame image and the probability of occurrence of each pixel point.

[0009] According to an embodiment of the present application, the process of performing optical flow calculation based on the first frame image at time t-1 and the second frame image at time t+1, and synthesizing the first predicted frame image and the second predicted frame image to obtain the third predicted frame image is implemented by the first frame prediction model.

[0010] According to an implementation manner of an embodiment of the present application, after obtaining the third predicted frame image, the method also includes: determining the fifth predicted frame image based on the third predicted frame image and the second frame prediction model, the second frame prediction model is a generative adversarial neural network model, and the generative model and the discriminative model of the generative adversarial neural network model are based on the same image generation network model.

[0011] According to an implementation manner of an embodiment of the present application, before obtaining the third predicted frame image, the method further includes: pre-training the first frame prediction model to obtain a pre-trained first frame prediction model.

[0012] According to an implementation of the first embodiment of the present application, before determining the sixth predicted frame image according to the third frame image and the second frame prediction model, the method further includes:

[0013] A generative adversarial neural network model is established based on the pre-trained first frame prediction model to obtain a second frame prediction model; and adversarial training is performed on the second frame prediction model to obtain the second frame prediction model after adversarial training.

[0014] According to an implementation manner of an embodiment of the present application, adversarial training is performed on the second frame prediction model, including: using the result of the first frame prediction model as the input of the generative model of the second frame prediction model, setting the training label to 1, and training the generative model of the second frame prediction model; using the result of the first frame prediction model as the input of the discriminative model of the second frame prediction model, setting the training label to 0, and training the discriminative model of the second frame prediction model.

[0015] According to an implementation of an embodiment of the present application, after performing adversarial training on the second frame prediction model to obtain the adversarially trained second frame prediction model, the method further includes: jointly training the first frame prediction model and the adversarially trained second frame prediction model.

[0016] According to a second aspect of an embodiment of the present application, an image processing device is provided, which includes: an optical flow prediction module, used to perform optical flow operations based on a first frame image at time t-1 and a second frame image at time t+1, to obtain a first optical flow and a second optical flow respectively, where t is a natural number, wherein the first optical flow is the optical flow from the frame image at time t-1 to the frame image at time t+1, and the second optical flow is the optical flow from the frame image at time t+1 to the frame image at time t-1; a depth prediction module, used to perform depth operations based on the first frame image and the second frame image, to obtain a first depth and a second depth respectively; a first predicted frame image prediction module, used to determine a first predicted frame image based on the first frame image, the first optical flow and the first depth; a second predicted frame image prediction module, used to determine a second predicted frame image based on the second frame image, the second optical flow and the second depth; and a frame synthesis module, used to synthesize the first predicted frame image and the second predicted frame to obtain a third predicted frame image.

[0017] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, the storage medium comprising a set of computer-executable instructions, which are used to execute any one of the above-mentioned image processing methods when the instructions are executed.

[0018] The embodiment of the present application provides an image processing method and device, which performs not only optical flow calculations but also depth calculations when inserting frames according to the optical flow method. Since the depth can indicate the distance between the subject and the lens, the greater the depth, the farther the distance, and the lower the pixel corresponding to the depth should be. Using this principle, the pixels in the optical flow can be strengthened or weakened by the depth, so that the inserted frame image is clearer and more reasonable, and closer to the real frame image, making the video picture smoother and the viewing experience better.

[0019] It should be understood that the implementation of this application does not require the realization of all the above beneficial effects, but a specific technical solution can achieve a specific technical effect, and other embodiments of this application can also achieve beneficial effects not mentioned above. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the exemplary embodiments of the present application will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present application are shown in an illustrative and non-limiting manner, in which:

[0021] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.

[0022] Figure 1 This is a flowchart of using the existing optical flow method for frame insertion;

[0023] Figure 2This is a schematic diagram of the implementation flow of the image processing method according to an embodiment of the present application;

[0024] Figure 3 This is a schematic diagram of a specific flow chart of an image processing method applied in another embodiment of the present application;

[0025] Figure 4 This is a schematic diagram of a specific flow chart of an image processing method applied in another embodiment of the present application;

[0026] Figure 5 for Figure 4 A schematic diagram of the process of training the model used in the embodiment shown;

[0027] Figure 6 Schematic diagram of the structure of an image processing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, features, and advantages of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.

[0029] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0030] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0031] Typically, a continuously shot video consists of a series of images captured at a specific time point over a continuous period of time. These frames are sorted by time to create a continuous time sequence. During playback, the original time points are restored and each frame is played back in sequence to form a continuous dynamic video.

[0032] The frame image at time t represents the frame image displayed at a certain moment, and the time t+1 represents the frame image displayed at the next moment, where t represents the sequence number of the frame image at that moment in the continuous frame image time sequence, which is a natural number.

[0033] Optical flow is the movement speed and direction of each pixel in a continuous frame image time series, from the frame image at time t to the frame image at time t+1.

[0034] For example, in the frame image at time t, the position of pixel A is (x1, y1); in the frame image at time t+1, the position of pixel A is (x2, y2), which means that pixel A has moved from (x1, y1) to (x2, y2). If (x2, y2)-(x1, y1)=(ux, vy), then the optical flow from the frame image at time t to the frame image at time t+1 is: (ux, vy).

[0035] Once the optical flow from the frame image at time t to the frame image at time t+1 is determined, the frame image at time t+1 can be warped with the optical flow to obtain the frame image at time t; or the frame image at time t can be warped with the optical flow to obtain the frame image at time t+1. Using this method, the optical flow method, it is possible to interpolate the original video frame image time series to obtain a denser frame image time series, thus making the video playback smoother.

[0036] Figure 1 A flowchart for frame interpolation using existing optical flow methods is shown. First, an optical flow calculation is performed based on the frame image at time t-1 and the frame image at time t+1, obtaining the optical flow from the frame image at time t-1 to the frame image at time t+1 and the optical flow from the frame image at time t+1 to the frame image at time t-1, respectively. Next, a linear estimation is performed based on the optical flow from the frame image at time t-1 to the frame image at time t+1 and the frame image at time t-1 to obtain the first frame image at time t. A linear estimation is performed based on the optical flow from the frame image at time t+1 to the frame image at time t-1 and the frame image at time t+1 to obtain the second frame image at time t. Subsequently, the first frame image at time t and the second frame image at time t are synthesized to obtain a synthesized frame image at time t. This synthesized frame image at time t can be inserted between the frame image at time t-1 and the frame image at time t+1, making the video smoother.

[0037] However, the above-mentioned interpolation optical flow method requires high accuracy of optical flow prediction. When the accuracy is insufficient, the image will be blurry. Moreover, the optical flow method requires constant brightness. Once the lighting conditions change, the image will be deformed.

[0038] To this end, this application provides Figure 2An image processing method shown in FIG. Figure 2 The method includes: operation 210, performing optical flow calculation based on the first frame image at time t-1 and the second frame image at time t+1 to obtain a first optical flow and a second optical flow respectively, where t is a natural number, wherein the first optical flow is the optical flow from the frame image at time t-1 to the frame image at time t+1, and the second optical flow is the optical flow from the frame image at time t+1 to the frame image at time t-1; operation 220, performing depth calculation based on the first frame image and the second frame image to obtain a first depth and a second depth respectively; operation 230, determining a first predicted frame image based on the first frame image, the first optical flow and the first depth; operation 240, determining a second predicted frame image based on the second frame image, the second optical flow and the second depth; operation 250, synthesizing the first predicted frame image and the second predicted frame image to obtain a third predicted frame image.

[0039] Among them, operation 210 is similar to the existing optical flow method in which the optical flow calculation is performed based on the frame image at time t-1 and the frame image at time t+1, and the optical flow from the frame image at time t-1 to the frame image at time t+1 and the optical flow from the frame image at time t+1 to the frame image at time t-1 are obtained respectively, so it will not be repeated.

[0040] However, unlike existing optical flow methods, in addition to performing operation 210, the embodiment of the present application also performs operation 220 to perform a depth calculation based on the first frame image at time t-1 and the second frame image at time t+1 to obtain a first depth and a second depth. The first depth is the depth obtained by performing the depth calculation on the first frame image; the second depth is the depth obtained by performing the depth calculation on the first frame image.

[0041] When shooting video, a three-dimensional image is projected onto a two-dimensional plane, forming a two-dimensional frame image. Depth is a numerical value indicating the distance between the object being filmed and the lens. Depth calculation refers to the process of performing computer operations on a two-dimensional image (for example, depth calculation based on a formula or depth prediction based on a depth estimation model) to determine pixel depth. This is equivalent to the inverse of the shooting process.

[0042] Typically, when the object is farther from the lens, the shadow cast by it is smaller and the grayscale is lower; when the object is closer, the shadow cast by it is larger and the grayscale is higher. Therefore, the probability of a pixel's presence can be determined based on its depth value. For example, if a pixel has a larger depth (meaning it was farther from the lens when it was captured), the probability of its presence is lower. Conversely, if a pixel has a smaller depth (meaning it was closer to the lens when it was captured), the probability of its presence is higher.

[0043] In operation 230, a conversion can be performed based on the first frame image and the first optical flow to obtain an estimated frame image at time t, i.e., the first frame image at time t. Then, the occurrence probability of each pixel point is obtained through depth calculation of the first frame image, and the pixel points of the estimated frame image at time t are strengthened or weakened according to the occurrence probability of each pixel point, so that a first predicted frame image that is clearer, more reasonable and closer to the real frame image can be obtained.

[0044] Similarly, in operation 240, a conversion can be performed based on the second frame image and the second optical flow to obtain an estimated frame image at time t, that is, the second frame image at time t. Then, the occurrence probability of each pixel point is obtained through depth calculation of the second frame image, and the pixel points of the estimated frame image at time t are strengthened or weakened according to the occurrence probability of each pixel point, so that a second predicted frame image that is clearer, more reasonable and closer to the real frame image can be obtained.

[0045] Thereafter, according to operation 250 , the first predicted frame image and the second predicted frame image are synthesized to obtain a third predicted frame image.

[0046] Figure 3 A specific flow chart showing another embodiment of the present application applying the above-mentioned image processing method is shown. Figure 3 The operation shown on the left is similar to the existing optical flow method and will not be described in detail. Figure 3 The depth operation shown in the right part (see operation 220 for details), and the first depth and second depth obtained by the depth operation are used to perform probability operation on the two t-time frame images (the first t-time frame image and the second t-time frame image) at time t obtained according to the optical flow method to obtain a predicted frame image processed by the probability operation (the first predicted frame image and the second predicted frame image), see operations 203 and 204 for details; thereafter, frame synthesis is performed through operation 205 to obtain a synthesized predicted frame image (the third predicted frame image).

[0047] In this embodiment, the first depth and the second depth are obtained based on the first frame image at time t-1, the second frame image at time t+1, and a deep network model. The deep network can use machine learning methods to estimate depth from a single image quickly and effectively.

[0048] Furthermore, as we gain depth through deep networks, we can also control the granularity (coarse or fine) of the deep network by defining the learning rate during training.

[0049] By adding a depth calculation operation and converting the depth into pixel probability, a probability calculation is then performed on the frame image at time t (also defined as the fourth predicted frame image) based on the pixel probability to produce a processed predicted frame image. The processed first and second predicted frame images are clearer, more reasonable, and closer to the actual frame image than the first and second frame images at time t before the calculation.

[0050] Since the larger the depth value, the farther the object is from the camera, the lower the probability of the pixel appearing; and the smaller the depth value, the closer the object is to the camera, the higher the probability of the pixel appearing. In this embodiment, a value between (0, 1) is obtained by performing a monotonically inverse function operation on the first depth to represent the probability of each pixel appearing.

[0051] Among them, a function with opposite monotonicity represents a function in which the relationship between the change of the function value and the change of the independent variable is opposite, for example, an inverse proportional function.

[0052] The probability calculation is performed by multiplying the occurrence probability of each pixel point with the fourth predicted frame image.

[0053] Accordingly, in Figure 3 The predicted frame image synthesized in Figure 1 The frame image at time t after synthesizing the first frame image at time t and the second frame image at time t is closer to the real frame image. Figure 3 The synthesized predicted frame image is inserted as the frame image at time t between the frame image at time t-1 and the frame image at time t-2, which can make the video picture smoother and the viewing experience better.

[0054] Figure 4 A specific flow chart of another embodiment using the above-mentioned image processing method is shown. This embodiment is used to draw the test operations and operation results in the automated test process as screenshots, and send the screenshots to the background every n milliseconds (for example, 1000 milliseconds). After receiving the above screenshots, the background uses the image processing method of this application to insert frames and restore them into a smooth and complete user operation video for R&D personnel to locate the problem scene.

[0055] Since the image processing method of this application is based on optical flow and depth estimation, it is particularly suitable for test scenarios that include the following operations: mouse movement operations; keyboard input operations and option checking operations.

[0056] exist Figure 4 In the embodiment shown, the first frame prediction model is used to implement Figure 3 All operations shown are performed to complete frame prediction, where the optical flow calculation is performed by the optical flow prediction model and the depth calculation is performed by the depth prediction model.

[0057] like Figure 4 As shown, the process of applying the image processing method of the embodiment of the present application to perform frame interpolation includes: inputting the frame image at time t-1 and the frame image at time t-2 into the first frame prediction model, and the first frame prediction model completes Figure 3 The frame prediction process shown outputs a third predicted frame image (the synthesized predicted frame image); then, the third predicted frame image is input into the second frame prediction model, where the second frame prediction model is a generative adversarial neural network model, and the second frame prediction model generates a fifth predicted frame image based on the third predicted frame image; then, the fifth predicted frame image is inserted as the frame image at time t between the frame image at time t-1 and the frame image at time t-2.

[0058] Among them, since in the application scenario of this embodiment, the frame image needs to be sent to the background server through the communication network, before the frame synthesis is performed, the predicted frame image processed by the probability calculation is compressed and decompressed through the frame compression and decompression network, so as to reduce the amount of data transmitted by the network as much as possible to save transmission bandwidth.

[0059] In addition, due to factors such as unstable screen display voltage, brightness changes may occur during the test, which will cause frame image deformation or even unreasonable frame images, causing the picture to jump, jitter, etc., and even unable to capture the real picture at the moment of error in the test.

[0060] To this end, Figure 4 The interpolation method shown further adds a second frame prediction model after the first prediction model outputs the third prediction frame image, so as to determine a fifth prediction frame image based on the third frame image and the second frame prediction model.

[0061] The second-frame prediction model is a generative adversarial neural network model, whose generative and discriminative models are based on the same image generation network model. This ensures that the fifth frame generated by the second-frame prediction model conforms to pixel motion patterns and image logic, making the inserted frame more reasonable and seamless with the previous and next frames.

[0062] In order to make Figure 4 The embodiment shown can predict the frame image with higher accuracy. Figure 5 Three-stage model training shown.

[0063] Among them, pre-trained weights are also used in the training process of the optical flow prediction model, depth prediction model and generative adversarial neural network model. Figure 5 As shown in Figure 2, the specific training process includes:

[0064] The first training stage is used to pre-train the optical flow prediction model and the depth prediction model in the first frame prediction model.

[0065] The learning rate during pre-training can be set to a smaller learning rate to perform coarse-grained depth prediction.

[0066] The training data (the frame image at time t-1 and the frame image at time t+1 annotated with the real frame image at time t) are input into the first frame prediction model, which performs model calculation and outputs the predicted frame image. The predicted frame image is compared with the annotated real frame image to calculate the loss function and optimize the first frame prediction model.

[0067] Among them, the loss function can use EPE to calculate the average value of the difference distance (Euclidean distance) between the true value and the predicted value of all pixels.

[0068] After the first stage of training is completed, the model parameters of the optical flow prediction model and the depth prediction model are locked.

[0069] The second training stage is used to perform adversarial training on the generative adversarial neural network model in the second frame prediction model.

[0070] The training data used for adversarial training of the generative adversarial neural network model in the second frame prediction model includes:

[0071] The real frame image at time t, the frame image at time t-1, the frame image at time t+1, and other real frame images in the continuous frame image time series are used as inputs of the generative model of the second frame prediction model, the training label is set to 1, and the generative model of the second frame prediction model is trained;

[0072] The result of the first frame prediction model is used as the input of the generative model of the second frame prediction model, the training label is set to 1, and the generative model of the second frame prediction model is trained;

[0073] The result of the first frame prediction model is used as the input of the discriminant model of the second frame prediction model, the training label is set to 0, and the discriminant model of the second frame prediction model is trained.

[0074] The generative model and discriminative model of the second frame prediction model are alternately trained using the above training data. After each training, the generative adversarial neural network loss function is calculated, and the second frame prediction model is optimized until convergence.

[0075] Among them, when training the second frame prediction model, binary cross entropy (BCE) can be used as the loss function of the generative adversarial neural network.

[0076] As mentioned earlier, because the generative adversarial neural network's generation model and discrimination stage are based on the same image generation network model, when the training reaches equilibrium, that is, when the discrimination module can no longer distinguish that the generated image is fake, the generated predicted frame image must be an image frame that is reasonably connected with the previous and next frames.

[0077] The third training stage is used to jointly train the first frame prediction model that has been pre-trained in the first training stage and the second frame prediction model that has been adversarially trained in the second training stage.

[0078] During this training phase, all parameters of all models can be unlocked, and the frame image at time t-1 and the frame image at time t+1 marked with the real frame image at time t are used as training data. The predicted frame image output by the second true prediction model is compared with the real frame image at time t to calculate the loss function and optimize the second frame prediction model.

[0079] It should be noted that Figures 4 and 5 The embodiments shown are merely illustrative of the embodiments of the image processing method of the present application, and are not limitations on the implementation methods and application scenarios. Implementers can flexibly adopt any applicable implementation methods and apply them to any applicable application scenarios based on specific implementation requirements and conditions.

[0080] Furthermore, the embodiment of the present application also provides an image processing device. Figure 6 As shown, the device 60 includes: an optical flow prediction module 601, which is used to perform optical flow calculations based on the first frame image at time t-1 and the second frame image at time t+1 to obtain a first optical flow and a second optical flow respectively, where t is a natural number, wherein the first optical flow is the optical flow from the frame image at time t-1 to the frame image at time t+1, and the second optical flow is the optical flow from the frame image at time t+1 to the frame image at time t-1; a depth prediction module 602, which is used to perform depth calculations based on the first frame image and the second frame image to obtain a first depth and a second depth respectively; a first predicted frame image prediction module 603, which is used to determine a first predicted frame image based on the first frame image, the first optical flow and the first depth; a second predicted frame image prediction module 604, which is used to determine a second predicted frame image based on the second frame image, the second optical flow and the second depth; and a frame synthesis module 605, which is used to synthesize the first predicted frame image and the second predicted frame to obtain a third predicted frame image.

[0081] According to an implementation of an embodiment of the present application, the depth prediction module 602 is specifically configured to obtain a first depth and a second depth based on a first frame image at time t-1, a second frame image at time t+1, and a depth network model.

[0082] According to an implementation manner of an embodiment of the present application, the first prediction frame image prediction module 603 includes: a fourth prediction frame image prediction submodule, used to obtain the fourth prediction frame image based on the first frame image and the first optical flow; a monotonically opposite function operation submodule, used to perform a monotonically opposite function operation on the first depth to obtain the probability of occurrence of each pixel point; a first prediction frame image determination submodule, used to determine the first prediction frame image based on the fourth prediction frame image and the probability of occurrence of each pixel point.

[0083] According to an embodiment of the present application, the device 60 includes: a first frame prediction model operation module, which is used to implement a process from performing optical flow calculation based on the first frame image at time t-1 and the second frame image at time t+1, to synthesizing the first predicted frame image and the second predicted frame image to obtain a third predicted frame image.

[0084] According to an embodiment of the present application, the device 60 also includes: a second frame prediction model operation module, which is used to determine the fifth prediction frame image based on the third prediction frame image and the second frame prediction model. The second frame prediction model is a generative adversarial neural network model, and the generative model and the discriminative model of the generative adversarial neural network model are based on the same image generation network model.

[0085] According to an implementation of an embodiment of the present application, the device 60 further includes: a first frame prediction model training module, which is used to pre-train the first frame prediction model to obtain a pre-trained first frame prediction model.

[0086] According to an implementation of an embodiment of the present application, the device 60 also includes: a second frame prediction model training module, which is used to establish a generative adversarial neural network model based on the pre-trained first frame prediction model to obtain a second frame prediction model; and perform adversarial training on the second frame prediction model to obtain a second frame prediction model after adversarial training.

[0087] According to an implementation manner of an embodiment of the present application, the second frame prediction model training module includes: a training submodule of a generation model, which is used to use the result of the first frame prediction model as the input of the generation model of the second frame prediction model, set the training label to 1, and train the generation model of the second frame prediction model; a training submodule of a discriminant model, which is used to use the result of the first frame prediction model as the input of the discriminant model of the second frame prediction model, set the training label to 0, and train the discriminant model of the second frame prediction model.

[0088] According to an implementation of an embodiment of the present application, the device 60 further includes: a joint training module for jointly training the first frame prediction model and the second frame prediction model after adversarial training.

[0089] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, the storage medium comprising a set of computer-executable instructions, which are used to execute any one of the above-mentioned image processing methods when the instructions are executed.

[0090] It should be noted that the above descriptions of the image processing embodiment and the computer storage medium embodiment are similar to the descriptions of the aforementioned method embodiment and have similar beneficial effects as the aforementioned method embodiment, and therefore are not described in detail here. For technical details not yet disclosed in the descriptions of the image processing device embodiment and the computer storage medium embodiment of this application, please refer to the descriptions of the aforementioned method embodiment of this application for understanding. To save space, they are not described in detail here.

[0091] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another device, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0093] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0094] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0095] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage media, read-only memory (ROM), magnetic disk or optical disk, and other media that can store program codes.

[0096] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage media, ROM, magnetic disk or optical disk.

[0097] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image processing method, comprising: Perform optical flow calculation on the first frame image at time t-1 and the second frame image at time t+1 to obtain the first optical flow and the second optical flow respectively, where t is a natural number, wherein the first optical flow is the optical flow from the frame image at time t-1 to the frame image at time t+1, and the second optical flow is the optical flow from the frame image at time t+1 to the frame image at time t-1; Performing depth calculation on the first frame image and the second frame image to obtain a first depth and a second depth respectively; Determine a first predicted frame image according to the first frame image, the first optical flow, and the first depth; Determine a second predicted frame image according to the second frame image, the second optical flow, and the second depth; synthesizing the first predicted frame image and the second predicted frame image to obtain a third predicted frame image; The determining a first predicted frame image according to the first frame image, the first optical flow, and the first depth includes: Obtain a fourth predicted frame image according to the first frame image and the first optical flow; Performing a monotonic opposite function operation on the first depth to obtain an occurrence probability of each pixel; A first predicted frame image is determined according to the fourth predicted frame image and the occurrence probability of each pixel point.

2. The method according to claim 1, wherein performing depth calculation based on the first frame image and the second frame image to obtain the first depth and the second depth respectively comprises: The first depth and the second depth are obtained according to the first frame image at time t-1, the second frame image at time t+1 and the deep network model.

3. According to the method of claim 1, the process from performing optical flow calculation based on the first frame image at time t-1 and the second frame image at time t+1 to synthesizing the first predicted frame image and the second predicted frame image to obtain the third predicted frame image is realized by the first frame prediction model.

4. The method according to claim 3, after obtaining the third predicted frame image, further comprising: A fifth predicted frame image is determined based on the third predicted frame image and the second frame prediction model, wherein the second frame prediction model is a generative adversarial neural network model, and the generative model and the discriminative model of the generative adversarial neural network model are based on the same image generation network model.

5. The method according to claim 4, before obtaining the third predicted frame image, the method further comprises: The first frame prediction model is pre-trained to obtain a pre-trained first frame prediction model.

6. The method according to claim 5, before determining the fifth predicted frame image based on the third predicted frame image and the second frame prediction model, the method further comprises: Establishing a generative adversarial neural network model based on the pre-trained first frame prediction model to obtain the second frame prediction model; Perform adversarial training on the second frame prediction model to obtain an adversarially trained second frame prediction model.

7. The method according to claim 6, wherein the performing adversarial training on the second frame prediction model comprises: Using the result of the first frame prediction model as the input of the generative model of the second frame prediction model, setting the training label to 1, and training the generative model of the second frame prediction model; The result of the first frame prediction model is used as the input of the discriminant model of the second frame prediction model, the training label is set to 0, and the discriminant model of the second frame prediction model is trained.

8. The method according to claim 6, after performing adversarial training on the second frame prediction model to obtain the adversarially trained second frame prediction model, the method further comprises: The first frame prediction model and the second frame prediction model after the adversarial training are jointly trained.

9. An image processing device, comprising: An optical flow prediction module is configured to perform an optical flow calculation based on the first frame image at time t-1 and the second frame image at time t+1 to obtain a first optical flow and a second optical flow, respectively, where t is a natural number, wherein the first optical flow is the optical flow from the frame image at time t-1 to the frame image at time t+1, and the second optical flow is the optical flow from the frame image at time t+1 to the frame image at time t-1; a depth prediction module, configured to perform a depth calculation based on the first frame image and the second frame image to obtain a first depth and a second depth, respectively; A first predicted frame image prediction module, configured to determine a first predicted frame image according to the first frame image, the first optical flow, and the first depth; A second predicted frame image prediction module, configured to determine a second predicted frame image according to the second frame image, the second optical flow, and the second depth; A frame synthesis module, configured to synthesize the first predicted frame image and the second predicted frame image to obtain a third predicted frame image; The first prediction frame image prediction module includes: a fourth predicted frame image prediction submodule, configured to obtain a fourth predicted frame image according to the first frame image and the first optical flow; a monotonic opposite function operation submodule, configured to perform a monotonic opposite function operation on the first depth to obtain an occurrence probability of each pixel; The first predicted frame image determination submodule is configured to determine a first predicted frame image according to the fourth predicted frame image and the occurrence probability of each pixel point.

Images