Model training method and device

Abstract

The embodiment of the invention provides a model training method, and the method comprises the steps: determining the similarity of two frames of images of a current input model, the two frames of images being adjacent in time sequence or being separated by a preset number of frame images in time sequence; predicting a middle frame of the two frames of images based on the similarity; under the condition that the similarity is greater than a first threshold value, updating the gradient of the model by adopting a first loss function based on the intermediate frame; under the condition that the similarity is smaller than a second threshold value, a second loss function is adopted to update the gradient of the model based on the middle frame, and the first loss function and the second loss function are both obtained through construction based on the similarity. According to the technical scheme provided by the embodiment of the invention, the model can determine whether the two frames of images currently input into the model have transition or not, and the intermediate frame is generated according to the transition or not, so that a wrong intermediate frame is prevented from being synthesized, and the accuracy of intermediate frame generation is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a model training method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Video interpolation, as the name suggests, is to add several frames between every two consecutive frames in the video sequence to shorten the display time between frames and improve the frame rate and smoothness of the video.

[0003] Among the current interpolation methods, the interpolation method based on the optical flow model synthesizes intermediate frames that are more reasonable than other interpolation methods, and the interpolation effect is more ideal. However, the interpolation method based on the optical flow model is only suitable for interpolating continuous frames. If the two frames to be interpolated are the previous and next frames in a scene switch, the interpolation method based on the optical flow model will synthesize an incorrect intermediate frame.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0005] The embodiments of the present application provide a model training method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0006] One aspect of an embodiment of the present application provides a model training method, the method comprising: Determine the similarity of two frames of images of the current input model, where the two frames of images are adjacent in time sequence or separated by a predetermined number of frames in time sequence; Predicting an intermediate frame of the two frames of image based on the similarity; When the similarity is greater than a first threshold, updating the gradient of the model using a first loss function based on the intermediate frame; When the similarity is less than a second threshold, a second loss function is used to update the gradient of the model based on the intermediate frame, wherein both the first loss function and the second loss function are constructed based on the similarity.

[0007] Optionally, extract each frame image of the target video, and obtain a first training data set based on the extraction result; Determine whether there is a transition between every two adjacent frame images in the first training data set in time sequence, and if there is a transition, mark the transition between the current two adjacent frame images; In the case of obtaining frame images from the first training data set and inputting them into the model for training, determining whether the transition identifier exists in the frame images corresponding to time t, time t+1, and time t+2 in the first training data set; In the absence of the transition mark, the frame images corresponding to the time t and the time t+2 are input into the model for training.

[0008] Optionally, the method further comprises: In the case where the transition mark does not exist and the similarity is greater than a first threshold, taking the frame image corresponding to the time t+1 as the first expected output result of the current training of the model; Correspondingly, updating the gradient of the model using a first loss function based on the intermediate frame includes: A first loss function is used to update the gradient of the model based on the intermediate frame and the first expected output result.

[0009] Optionally, the method further comprises: In the presence of the transition mark, the target frame image and the frame image adjacent to the target frame image are input into the model for training, and the target frame image is the first frame image with the transition mark among the frame images corresponding to the time t, the time t+1 and the time t+2.

[0010] Optionally, the method further comprises: In the case where the transition mark exists and the similarity is less than a second threshold, taking the target frame image and one of the frame images adjacent to the target frame image as a second expected output result of the current training of the model; Correspondingly, updating the gradient of the model using a second loss function based on the intermediate frame includes: A second loss function is used to update the gradient of the model based on the intermediate frame and the second expected output result.

[0011] Optionally, the two frames of images are a first frame of image and a second frame of image; Correspondingly, predicting the intermediate frame of the two frames of images based on the similarity includes: Predicting a first optical flow field from the intermediate frame to the first frame image, a second optical flow field from the intermediate frame to the second frame image, and a fusion weight based on an optical flow prediction algorithm; Predicting an intermediate frame between the two frames of images based on the first optical flow field, the second optical flow field, the fusion weight and the similarity.

[0012] Optionally, the first loss function is constructed based on the intermediate frame and the similarity, and the second loss function is constructed based on the intermediate frame, the similarity, the first optical flow field, the second optical flow field and the fusion weight.

[0013] Another aspect of an embodiment of the present application provides a model training device, the device comprising: A determination module, used to determine the similarity of two frames of images of the current input model, wherein the two frames of images are adjacent in time sequence or separated by a predetermined number of frames in time sequence; A prediction module, used for predicting the middle frame of the two frames of images based on the similarity; A first updating module, configured to update the gradient of the model using a first loss function based on the intermediate frame when the similarity is greater than a first threshold; A second updating module is used to update the gradient of the model using a second loss function based on the intermediate frame when the similarity is less than a second threshold, wherein the first loss function and the second loss function are both constructed based on the similarity.

[0014] Another aspect of an embodiment of the present application provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein: the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0015] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0016] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0017] The above technical solution adopted in the embodiment of the present application may have the following advantages: By determining the similarity of the two frames of the current input model, the middle frame of the two frames is predicted based on the similarity. When the similarity is greater than a first threshold, the gradient of the model is updated based on the predicted middle frame using a first loss function; when the similarity is less than a second threshold, the gradient of the model is updated based on the middle frame using a second loss function. The first loss function and the second loss function are both constructed based on the similarity of the two frames. The model can be trained using different loss functions according to the similarity between the input frame images, so that the model can adapt to two situations with high similarity (corresponding to continuous frames) and low similarity (corresponding to transition frames), respectively. The intermediate frame can be generated according to whether there is a transition, so as to avoid synthesizing erroneous intermediate frames and improve the accuracy of intermediate frame generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 The flowchart of the model training method according to the first embodiment of the present application is schematically shown; Figure 2 The newly added process of the model training method according to the first embodiment of the present application is schematically shown; Figure 3 Schematically shows Figure 1 Flow chart of sub-steps of step S102; Figure 4 A block diagram of a model training device according to Embodiment 2 of the present application is schematically shown; and Figure 5 The hardware architecture diagram of the computer device according to the third embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0021] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0022] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0023] First, the following terms are explained: Video interpolation: It is a video processing technology that increases the frame rate of the video by inserting newly generated intermediate frames between existing frames, making the video playback smoother and more delicate.

[0024] Optical flow: refers to the rate of change of each pixel in an image sequence over time, usually represented by a two-dimensional vector.

[0025] Transition: It is a transition method between two different pictures in a video. It is like a shot switching in a movie, making the connection between different pictures more natural and smooth, and enhancing the visual effect of the video.

[0026] Artifacts: In image and video processing, artifacts refer to distortions or noise that are not originally present in an image or video but are introduced by the image processing process or equipment. These distortions usually affect the quality of the image or video, making the image look unnatural or distorted.

[0027] Peak signal-to-noise ratio (PSNR): It is a commonly used objective indicator for evaluating image quality, especially in the fields of image compression and restoration. It measures the degree of noise pollution of an image by comparing the difference between the original image and the processed image.

[0028] Structural Similarity Index (SSIM): is a measure of the similarity between two images. Unlike traditional indicators such as mean square error (MSE), SSIM is more in line with the human visual system's perception of image quality, and it takes into account the brightness, contrast, and structural information of the image.

[0029] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: There are three main methods for interpolation: 1. Directly copy the previous or next frame as the interpolation frame; 2. Perform a double-exposure blur process on the previous and next frames to obtain the intermediate frame; 3. Interpolation based on the optical flow model, by analyzing and modeling the previous and next frames to generate optical flow to obtain the linear mapping relationship between frames, and finally combine the intermediate frame. Among these three methods, the intermediate frame produced by the third method is more reasonable and has a better interpolation effect.

[0030] Although the interpolation method based on the optical flow model works well, it is only applicable to interpolation of continuous frames. If there is a transition (scene switching) between frame images, the two frames before and after the scene switching are often discontinuous in content and image, and there are large differences. At this time, if the interpolation method based on the optical flow model is used for interpolation, the intermediate frame will have very serious artifacts. In order to avoid this situation, the common solution in the industry is to calculate the PSNR or SSIM of the two frames to determine whether the two frames are at the scene switching moment. If so, copy one of the frames as the intermediate frame instead of using the interpolation method based on the optical flow model.

[0031] However, both PSNR and SSIM are pixel-based image quality evaluation indicators that cannot reflect the semantic information or visual perception of the image. Moreover, PSNR and SSIM are global evaluation indicators that give the same weight to each pixel of the image regardless of its location and importance. More importantly, these indicators do not incorporate the temporal information of the video, and video interpolation is often based on temporal information. In scenes involving large dynamic changes, these indicators are often unreliable.

[0032] In common optical flow-based interpolation models, when the previous and next frames are continuous frames, the generated intermediate frames will basically have no artifacts. If the previous two frames are transition frames, then in theory, there is no need to generate intermediate frames, and the previous or next frame can be directly copied as the intermediate frame. However, the model cannot determine whether it is a transition frame, resulting in the model generating non-existent optical flow and synthesizing erroneous intermediate frames.

[0033] To this end, the embodiment of the present application provides a technical solution for model training. In this technical solution, the model can determine whether there is a transition between the two frames of the current input model, and generate an intermediate frame based on whether there is a transition, thereby avoiding the synthesis of incorrect intermediate frames and improving the accuracy of intermediate frame generation. See below for details.

[0034] The technical solutions of the present application are described below through multiple embodiments. It should be noted that these embodiments can be implemented in a variety of different forms and should not be construed as being limited to the embodiments described here.

[0035] Embodiment 1 Figure 1The flowchart of the model training method according to the first embodiment of the present application is schematically shown.

[0036] like Figure 1 As shown, the model training method may include steps S100 to S106, wherein: Step S100, determining the similarity of two frame images of the current input model, wherein the two frame images are adjacent in time sequence or separated in time sequence by a predetermined number of frame images.

[0037] Step S102: predicting an intermediate frame between the two image frames based on the similarity.

[0038] Step S104: when the similarity is greater than a first threshold, updating the gradient of the model by using a first loss function based on the intermediate frame.

[0039] Step S106, when the similarity is less than a second threshold, updating the gradient of the model by using a second loss function based on the intermediate frame, wherein both the first loss function and the second loss function are constructed based on the similarity.

[0040] The model training method provided in this embodiment determines the similarity between two frames of images of the current input model, predicts the middle frame of the two frames of images based on the similarity, and when the similarity is greater than a first threshold, uses a first loss function to update the gradient of the model based on the predicted middle frame; when the similarity is less than a second threshold, uses a second loss function to update the gradient of the model based on the middle frame, wherein the first loss function and the second loss function are both constructed based on the similarity of the two frames of images, so that the model can be trained using different loss functions according to the similarity between the input frame images, thereby making the model adapt to two situations with high similarity (corresponding to continuous frames) and low similarity (corresponding to transition frames), respectively, and can generate intermediate frames according to whether there is a transition, avoid synthesizing erroneous intermediate frames, and improve the accuracy of intermediate frame generation.

[0041] The following combination Figure 1 , each step in steps S100~S106 and other optional steps are explained in detail.

[0042] Step S100 , determine the similarity of two frame images of the current input model, where the two frame images are adjacent in time sequence or separated by a predetermined number of frame images in time sequence.

[0043] Specifically, the similarity of the two frames of images can be determined by determining the Euclidean distance, Manhattan distance, Bhattacharyya distance, cosine similarity, SSIM, etc. of the two frames of images of the current input model, or by a deep learning model. Optionally, it can be implemented by a module in the model (which can be called a transition discrimination module), for example, the module can be used to obtain the feature maps of the two frames of images respectively, and then the similarity s of the two feature maps can be judged by the fully connected layer, and the result of 0 to 1 is output. If the similarity is closer to 1, the two frames of images are more similar, and they are non-continuous frames; if the similarity is closer to 0, the two frames of images are less similar, and they are transition frames.

[0044] When preparing a training data set for the model, frame images can be extracted from multiple videos, and then the extracted frame images are marked as continuous frames and / or transition frames, etc., and it is determined in advance how to take two frames of images from the training data set to input the model, as well as the model output corresponding to the two frames of images, and then the frame images are taken from the training data set to input the model for training. Some exemplary schemes are provided below.

[0045] In an optional embodiment, if Figure 2 As shown, the model training method of the embodiment of the present application may also include: Step S200: extract each frame image of the target video, and obtain a first training data set based on the extraction result.

[0046] Step S202, determining whether there is a transition between every two adjacent frame images in the first training data set in time sequence, and if there is a transition, marking the transition between the current two adjacent frame images.

[0047] Step S204, when frame images are obtained from the first training data set and input into the model for training, it is determined whether the transition mark exists in the frame images corresponding to time t, time t+1 and time t+2 in the first training data set.

[0048] Step S206, when the transition mark does not exist, the frame images corresponding to the time t and the time t+2 are input into the model for training.

[0049] There may be multiple target videos, that is, frame images are extracted from multiple videos as the first training data set. Time t may be the time corresponding to the current frame image when the frame image is taken from the first training data set in a predetermined order. For example, if the frame image is taken from the 1st frame, the 2nd frame, and the 3rd frame when the frame image is acquired for the first time, then time t is the time corresponding to the 1st frame; if the frame image is taken from the 4th frame, the 5th frame, and the 6th frame when the frame image is acquired for the second time, then time t is the time corresponding to the 4th frame, and so on. Time t+1 is the time corresponding to the first frame image after the frame image corresponding to time t, and time t+2 is the time corresponding to the second frame image after the frame image corresponding to time t.

[0050] Specifically, each frame image of the target video can be extracted, and a first training data set can be formed according to the extracted frame images; when forming the first training data set, the scene switching judgment can be performed on every two adjacent frame images in the first training data set in time sequence to determine whether there is a transition, and in the case of a transition, the two adjacent frame images currently judged are marked as transitions; optionally, in the case of no transition, the two adjacent frame images currently judged can be marked as continuous frames, and of course, the continuous frames can also be marked. After all the frame images of the first training data set are determined and marked, the model can be trained according to the first training data set. In the case of obtaining frame images from the first training data set to input the model for training, a part of the frame images can be taken from the first training data set in a certain order. Before taking the frame images, it can be determined whether there are transition marks for the corresponding frame images at time t, time t+1 and time t+2; in the case of no transition mark, it indicates that the corresponding frame images at time t, time t+1 and time t+2 are continuous frames, and the corresponding frame images at time t and time t+2 are input into the model for training. Among them, the frame image corresponding to the time t+1 can be used as the intermediate frame expected to be generated, and the intermediate frame can be labeled accordingly when forming the first training data set. When determining whether there is a transition, the absolute difference method, mean square error, SSIM, histogram, optical flow method or machine learning method can be used to determine whether there is a transition. When the transition mark is performed on the two adjacent frame images currently judged, the previous frame image belongs to the scene before the transition, and the next frame image belongs to the scene after the transition.

[0051] In this embodiment, by extracting each frame image of the target video, a first training data set is obtained based on the extraction result, and it is determined whether there is a transition between every two adjacent frame images in the first training data set in time sequence. If there is a transition, the current two adjacent frame images are marked with a transition mark; when the frame images are obtained from the first training data set and input into the model for training, it is determined whether there is a transition mark for the frame images corresponding to time t, time t+1 and time t+2 in the first training data set. If there is no transition mark, the frame images corresponding to time t and time t+2 are input into the model for training. It can be automatically determined whether the frame images in the training data set are continuous frames, and in the case of continuous frames, suitable frame images are automatically obtained and input into the model for training, thereby improving the efficiency of model training.

[0052] In the presence of a transition marker, in theory, two frame images can be taken one frame apart to input into the model for training. Because in the case of a transition, there are large differences between the frame image of the previous scene and the frame image of the next scene. Therefore, the above two frame images can be two frame images separated by a predetermined number of frame images in time sequence, where the predetermined number can be 1, 2 or 3, etc.

[0053] In an optional embodiment, the model training method of the embodiment of the present application may also include: in the presence of a transition marker, inputting the target frame image and the frame images adjacent to the target frame image into the model for training, the target frame image being the first frame image with a transition marker among the frame images corresponding to time t, time t+1, and time t+2.

[0054] For example, if the target frame image is the frame image corresponding to time t+1, the frame images corresponding to time t+1 and time t+2 are input into the model for training; for another example, if the target frame image is the frame image corresponding to time t+2, the frame images corresponding to time t+2 and time t+3 are input into the model for training.

[0055] In the case of a transition, one of the two frames of images input to the model may be used as the expected output of the model, and may be specifically labeled in forming the first training data set.

[0056] It is understandable that, in the case of transitions, although the two frame images input to the model can be separated by a predetermined number of frame images in time sequence, this will cause a waste of resources in the training data set; in the present embodiment, in the presence of a transition marker, the target frame image and the frame image adjacent to the target frame image are input into the model for training, which can reduce the waste of resources in the training data set and improve the utilization rate of training resources; at the same time, in the case of transitions, data different from the continuous frame situation can be automatically obtained for training, thereby improving the flexibility of training.

[0057] It should be noted that in practical applications, training data sets can also be constructed for continuous frames and transition frames respectively, and then different training data frame images can be obtained for training. However, this will increase the complexity of constructing training data sets and model training operations. And through the method of the above-mentioned embodiment of the present application, the same training data set can be used, and continuous and transition frames can be automatically distinguished to obtain frame images, which can effectively reduce the complexity of constructing training data sets and model training operations, and improve the efficiency of model training.

[0058] Step S102 , predicting an intermediate frame between the two image frames based on the similarity.

[0059] Specifically, the optical flow prediction method can be used to predict the middle frame of two frames of images based on the similarity. Of course, in practical applications, the optical flow prediction method can also be used to predict the middle frame of two frames of images when the similarity is high; and other methods (such as the copy method) can be used to predict the middle frame of two frames of images when the similarity is low. Some exemplary solutions are provided below.

[0060] In an optional embodiment, the two frames of images input to the model are a first frame of image and a second frame of image. Figure 3 As shown, step S102 may include: Step S300: predicting a first optical flow field from the intermediate frame to the first frame image, a second optical flow field from the intermediate frame to the second frame image, and a fusion weight based on an optical flow prediction algorithm.

[0061] Step S302: predicting an intermediate frame between the two frames of images based on the first optical flow field, the second optical flow field, the fusion weight and the similarity.

[0062] Assuming that the first frame image is the frame image corresponding to time 0, the second frame image is the frame image corresponding to time 1, and the intermediate frame corresponds to time t, the first optical flow field from the intermediate frame to the first frame image, that is, the optical flow field from t→0, can be predicted based on the optical flow prediction algorithm; the second optical flow field from the intermediate frame to the second frame image, that is, the optical flow field from t→1, can be predicted, and the fusion weight (fusionMap, which can be recorded as f_map) can be obtained at the same time. For a general optical flow interpolation model, the intermediate frame can be expressed as: I_t=f_map (warp(I_0,f_t→0))+((1-f_map) (warp(I_1,f_t→1))); Among them, I_t represents the intermediate frame, I_0 represents the first frame image, I_1 represents the second frame image, f_t→0 represents the first optical flow field, and f_t→1 represents the second optical flow field. In the case of continuous frames, the intermediate frame can be generated correctly, but in the case of transition frames, the optical flow of t→0 and t→1 that should not exist will be generated, and the wrong intermediate frame will be synthesized.

[0063] In this embodiment, the intermediate frame of the two frames is predicted based on the first optical flow field, the second optical flow field, the fusion weight and the similarity. The similarity can be used as the adjustment weight so that in the case of transition, the intermediate frame can be generated based on the first frame or the second frame. For example, the expression of the intermediate frame can be rewritten as: I_t=f_map (warp(I_0,s (f_t→0)))+((1-f_map) (warp(I_1,s (f_t→1)))); When the input is a continuous frame, under the training optimization of the model, s can tend to 1, and the expression of the intermediate frame will become I_t=f_map (warp(I_0,f_t→0))+((1-f_map) (warp(I_1,f_t→1))). When the input is a transition frame, under the training optimization of the model, s will tend to 0, so f_t→0 and f_t→1 will also tend to 0, then the expression of the intermediate frame I_t=f_map (warp(I_0)+(1-f_map) (warp(I_1); At the same time, the loss function can be constructed according to f_map to make it tend to 0 or 1, so that the intermediate frame will eventually be generated according to the first frame image or the second frame image.

[0064] In this embodiment, by predicting the first optical flow field from the intermediate frame to the first frame image, the second optical flow field from the intermediate frame to the second frame image and the fusion weight based on the optical flow prediction algorithm, the intermediate frame of the two frames of images of the input model is predicted based on the first optical flow field, the second optical flow field, the fusion weight and the similarity. The similarity can be used as the adjustment weight to enable the model to automatically distinguish between continuous frames and transition frames to generate intermediate frames. While improving the accuracy of intermediate frame generation, the unification of intermediate frame generation for continuous frames and transition frames is achieved.

[0065] Step S104 , when the similarity is greater than a first threshold, updating the gradient of the model using a first loss function based on the intermediate frame.

[0066] When the similarity is greater than the first threshold, it indicates that the two frames of images input to the model training are continuous frames, and the gradient of the model can be updated according to the first loss function corresponding to the continuous frames. When constructing the first loss function, the goal can be that the similarity tends to 1. In addition, the first loss function can be constructed based on the predicted intermediate frame and some pre-labeled data. Some exemplary schemes are provided below.

[0067] In an optional embodiment, in the aforementioned embodiment, when there is no transition marker and the similarity is greater than a first threshold, the frame image corresponding to time t+1 is used as the first expected output result of the current training of the model; correspondingly, in step S104, updating the gradient of the model based on the intermediate frame using the first loss function may include: updating the gradient of the model based on the intermediate frame and the first expected output result using the first loss function.

[0068] Exemplarily, the first loss function may be: L_total=L1_loss(I_t,I_0.5)+L1_loss(s,1); Among them, I_0.5 corresponds to the frame image corresponding to the above time t+1. In addition, the L1 loss function can also be replaced by the L2 loss function, the Smooth L1 loss function and the Huber loss function.

[0069] When there is no transition marker and the similarity is greater than the first threshold, the input to the model is continuous frames, corresponding to the frame images at time t and time t+2 respectively. At this time, the frame image corresponding to time t+1 is used as the first expected output result of the current training of the model. Since the frame image corresponding to time t+1 is actually the intermediate frame image between the frame images corresponding to time t and time t+2, the frame image corresponding to time t+1 is used as the first expected output to train the model, which can enable the model to obtain better performance.

[0070] In this embodiment, when there is no transition marker and the similarity is greater than the first threshold, the frame image corresponding to time t+1 is used as the first expected output result of the current training of the model, and the gradient of the model is updated using the first loss function based on the intermediate frame and the first expected output result. In the case of continuous frames, the frame image corresponding to time t+1 between the frame images corresponding to time t and time t+2 can be used as the expected output to train the model, which can facilitate the acquisition of the expected output value of the model from the first training data set, thereby improving the efficiency of model training.

[0071] Step S106 , when the similarity is less than a second threshold, the gradient of the model is updated based on the intermediate frame using a second loss function, wherein both the first loss function and the second loss function are constructed based on the similarity.

[0072] After the model is trained according to the first loss function and the second loss function, if the model reaches a certain accuracy, a trained model can be obtained, and subsequently a corresponding intermediate frame can be generated for the two input image frames based on the trained model.

[0073] When the similarity is less than the second threshold, it indicates that the two frames of images input to the model training are transition frames, and the gradient of the model can be updated according to the second loss function corresponding to the transition frame. When constructing the second loss function, the goal can be that the similarity tends to 0. In addition, the second loss function can also be constructed based on the predicted intermediate frame and some pre-labeled data. Some exemplary schemes are provided below.

[0074] In an optional embodiment, in the aforementioned embodiment, when there is a transition marker and the similarity is less than a second threshold, the target frame image and one of the frame images adjacent to the target frame image are used as the second expected output result of the current training of the model, wherein the target frame image is the first frame image with a transition marker among the frame images corresponding to time t, time t+1, and time t+2; correspondingly, in step S106, the gradient of the model is updated using the second loss function based on the intermediate frame and the second expected output result.

[0075] As mentioned above, since it is hoped that when the frame is transitioned, under the optimization of the model, the first optical flow field and the second optical flow field tend to 0, and the fusion weight tends to 1, the second loss function can also include the first optical flow field, the second optical flow field and the fusion weight, so that the model can optimize these elements during the training process.

[0076] In an optional embodiment, the first loss function is constructed based on the intermediate frame and the similarity, and the second loss function is constructed based on the intermediate frame, the similarity, the first optical flow field, the second optical flow field and the fusion weight.

[0077] That is, the first loss function can be constructed based only on the two elements of the intermediate frame and the similarity, while the second loss function is constructed based on the first optical flow field, the second optical flow field and the fusion weight in addition to the intermediate frame and the similarity. The expression of the first loss function can be as described above, and the expression of the second loss function can be: L_total=L1_loss(I_t,I_0)+L1_loss(s,0)+L1_loss(f_t→0,0)+L1_loss(f_t→1,0)+L1_loss(f_map,1); Among them, I_0 represents the target frame image, and the L1 loss function can also be replaced by the L2 loss function, Smooth L1 loss function, Huber loss function, etc. In addition, f_map aims to be close to 1.

[0078] It should be understood that the above expression takes the target frame image as the expected output result of the model. In fact, the expected output result of the model can also be the frame image adjacent to the target frame image. When the frame image adjacent to the target frame image is taken as the expected output result of the model, the expression of the second loss function can be: L_total=L1_loss(I_t,I_1)+L1_loss(s,0)+L1_loss(f_t→0,0)+L1_loss(f_t→1,0)+L1_loss(f_map,0); Among them, I_1 represents the frame image adjacent to the target frame image. In addition, f_map aims to be close to 0.

[0079] When there is a transition marker and the similarity is less than the second threshold, the input to the model is a transition frame. At this time, the target frame image and one of the subsequent adjacent frame images of the target frame image are trained as the expected output results of the model. The intermediate frame generated by the model can be equivalent to the method of copying the frame, thereby achieving the generation of the intermediate frame by the method equivalent to the copying frame in the case of the transition frame. In this way, the model can be applicable to both the case of continuous frames and the case of transition frames.

[0080] In this embodiment, when there is a transition marker and the similarity is less than a second threshold, the target frame image and one of the frame images adjacent to the target frame image are used as the second expected output result of the current training of the model, and the gradient of the model is updated using a second loss function based on the intermediate frame and the second expected output result. When the input is a transition frame, the intermediate frame can be automatically generated using a method equivalent to a copy frame, so that the model is applicable not only to continuous frames but also to transition frames, thereby improving the flexibility and performance of the model, and also improving the accuracy of intermediate frame generation.

[0081] Embodiment 2 Figure 4 The block diagram of the model training device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 4 As shown, the apparatus 400 may include: a determination module 410, a prediction module 420, a first update module 430 and a second update module 440, wherein: A determination module 410 is used to determine the similarity of two frames of images of the current input model, where the two frames of images are adjacent in time sequence or separated by a predetermined number of frames in time sequence; A prediction module 420, configured to predict an intermediate frame between the two frames of images based on the similarity; A first updating module 430, configured to update the gradient of the model using a first loss function based on the intermediate frame when the similarity is greater than a first threshold; The second updating module 440 is used to update the gradient of the model using a second loss function based on the intermediate frame when the similarity is less than a second threshold, wherein the first loss function and the second loss function are both constructed based on the similarity.

[0082] In an optional embodiment, the device 400 is further used for: Extracting each frame image of the target video, and obtaining a first training data set based on the extraction result; Determine whether there is a transition between every two adjacent frame images in the first training data set in time sequence, and if there is a transition, mark the transition between the current two adjacent frame images; In the case of obtaining frame images from the first training data set and inputting them into the model for training, determining whether the transition identifier exists in the frame images corresponding to time t, time t+1, and time t+2 in the first training data set; In the absence of the transition mark, the frame images corresponding to the time t and the time t+2 are input into the model for training.

[0083] In an optional embodiment, the device 400 is further used for: In the case where the transition mark does not exist and the similarity is greater than a first threshold, taking the frame image corresponding to the time t+1 as the first expected output result of the current training of the model; Correspondingly, the first updating module 430 is further used for: A first loss function is used to update the gradient of the model based on the intermediate frame and the first expected output result.

[0084] In an optional embodiment, the device 400 is further used for: In the presence of the transition mark, the target frame image and the frame image adjacent to the target frame image are input into the model for training, and the target frame image is the first frame image with the transition mark among the frame images corresponding to the time t, the time t+1 and the time t+2.

[0085] In an optional embodiment, the device 400 is further used for: In the case where the transition mark exists and the similarity is less than a second threshold, taking the target frame image and one of the frame images adjacent to the target frame image as a second expected output result of the current training of the model; Correspondingly, the second updating module 440 is further used for: A second loss function is used to update the gradient of the model based on the intermediate frame and the second expected output result.

[0086] In an optional embodiment, the two frames of images are a first frame of image and a second frame of image. Correspondingly, the prediction module 420 is further configured to: Predicting a first optical flow field from the intermediate frame to the first frame image, a second optical flow field from the intermediate frame to the second frame image, and a fusion weight based on an optical flow prediction algorithm; Predicting an intermediate frame between the two frames of images based on the first optical flow field, the second optical flow field, the fusion weight and the similarity.

[0087] In an optional embodiment, the first loss function is constructed based on the intermediate frame and the similarity, and the second loss function is constructed based on the intermediate frame, the similarity, the first optical flow field, the second optical flow field and the fusion weight.

[0088] Embodiment 3 Figure 5 The schematic diagram of the hardware architecture of a computer device 10000 suitable for implementing the model training method according to the third embodiment of the present application is schematically shown. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Figure 5 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other through a system bus. Among them: The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 can be an internal storage module of the computer device 10000, such as a hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 can also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed on the computer device 10000, such as the program code of the model training method, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or are to be output.

[0089] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0090] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0091] It should be pointed out that Figure 5 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0092] In this embodiment, the model training method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0093] Embodiment 4 An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the model training method in the embodiment are implemented.

[0094] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the computer-readable storage medium may also include both an internal storage unit of a computer device and an external storage device thereof. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on a computer device, such as the program code of the model training method in the embodiment, etc. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.

[0095] Embodiment 5 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0096] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0097] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Determine the similarity of two frames of images of the current input model, where the two frames of images are adjacent in time sequence or separated by a predetermined number of frames in time sequence; Predicting an intermediate frame of the two frames of image based on the similarity; When the similarity is greater than a first threshold, updating the gradient of the model using a first loss function based on the intermediate frame; When the similarity is less than a second threshold, a second loss function is used to update the gradient of the model based on the intermediate frame, wherein both the first loss function and the second loss function are constructed based on the similarity.

2. The method according to claim 1, characterized in that The method further comprises: Extracting each frame image of the target video, and obtaining a first training data set based on the extraction result; Determine whether there is a transition between every two adjacent frame images in the first training data set in time sequence, and if there is a transition, mark the transition between the current two adjacent frame images; In the case of obtaining frame images from the first training data set and inputting them into the model for training, determining whether the transition identifier exists in the frame images corresponding to time t, time t+1, and time t+2 in the first training data set; In the absence of the transition mark, the frame images corresponding to the time t and the time t+2 are input into the model for training.

3. The method according to claim 2, characterized in that The method further comprises: In the case where the transition mark does not exist and the similarity is greater than a first threshold, taking the frame image corresponding to the time t+1 as the first expected output result of the current training of the model; Correspondingly, updating the gradient of the model using a first loss function based on the intermediate frame includes: A first loss function is used to update the gradient of the model based on the intermediate frame and the first expected output result.

4. The method according to claim 2, characterized in that: The method further comprises: In the presence of the transition mark, the target frame image and the frame image adjacent to the target frame image are input into the model for training, and the target frame image is the first frame image with the transition mark among the frame images corresponding to the time t, the time t+1 and the time t+2.

5. The method according to claim 4, characterized in that The method further comprises: In the case where the transition mark exists and the similarity is less than a second threshold, taking the target frame image and one of the frame images adjacent to the target frame image as a second expected output result of the current training of the model; Correspondingly, updating the gradient of the model using a second loss function based on the intermediate frame includes: A second loss function is used to update the gradient of the model based on the intermediate frame and the second expected output result.

6. The method according to any one of claims 1 to 5, characterized in that: The two frames of images are a first frame of image and a second frame of image; Correspondingly, predicting the intermediate frame of the two frames of images based on the similarity includes: Predicting a first optical flow field from the intermediate frame to the first frame image, a second optical flow field from the intermediate frame to the second frame image, and a fusion weight based on an optical flow prediction algorithm; Predicting an intermediate frame between the two frames of images based on the first optical flow field, the second optical flow field, the fusion weight and the similarity.

7. The method according to claim 6, characterized in that The first loss function is constructed based on the intermediate frame and the similarity, and the second loss function is constructed based on the intermediate frame, the similarity, the first optical flow field, the second optical flow field and the fusion weight.

8. A model training device, characterized in that: The device comprises: A determination module, used to determine the similarity of two frames of images of the current input model, wherein the two frames of images are adjacent in time sequence or separated by a predetermined number of frames in time sequence; A prediction module, used for predicting the middle frame of the two frames of images based on the similarity; A first updating module, configured to update the gradient of the model using a first loss function based on the intermediate frame when the similarity is greater than a first threshold; A second updating module is used to update the gradient of the model using a second loss function based on the intermediate frame when the similarity is less than a second threshold, wherein the first loss function and the second loss function are both constructed based on the similarity.

9. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images