Method, device and electronic device for generating video from image

The generation of flow parameters through the target image processing model solves the problem of high complexity in the generation of flow effect videos in the prior art, and the conversion from static images to flow videos is realized, and the efficiency and effect of video generation are improved.

CN114049384BActive Publication Date: 2024-11-19DOUYIN VISION CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111318900.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-09
Publication Date
2024-11-19
Estimated Expiration
2041-11-09

AI Technical Summary

Technical Problem

It is difficult to effectively generate videos with flow effects in the prior art, especially when the image contains objects such as hair and clothes. The flow effects of these objects need to be presented, but the implementation is complex and depends on artificially selected areas and flow directions.

Method used

Through the target image processing model, flow parameters (region, flow direction) corresponding to the static image are generated, and multiple frame images are generated based on these parameters to combine them into video. The accuracy and robustness of the model are improved by using neural network models and data augmentation technology. .

Benefits of technology

The complexity of generating flow effect videos is reduced, and videos with flow effects are generated through static images are realized, the accuracy of flow area recognition and flow direction prediction is improved, and the efficiency and effect of video generation is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114049384B_ABST
    Figure CN114049384B_ABST
Patent Text Reader

Abstract

The present disclosure relates to a method, device and electronic device for generating a video from an image, and relates to the field of image processing technology. The method comprises: obtaining a first image; inputting the first image into a target image processing model, obtaining a first flow parameter for the first image output by the target image processing model, performing flow processing on the first image based on the first flow parameter to generate multiple frames of second images, and combining the multiple frames of second images to obtain a video; wherein the first flow parameter includes: at least one region and the flow direction of each region, and the target image processing model is a neural network model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method, device and electronic device for generating video from images. Background Art

[0002] At present, some images may contain objects such as hair and clothes. These objects are in a flowing state in actual scenes. When the flowing effect of these objects needs to be presented, a method for generating videos with flowing effects is urgently needed. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a method, device and electronic device for generating videos from images. A video with a flow effect can be generated from a static image.

[0004] In order to achieve the above objectives, the technical solutions provided by the embodiments of the present disclosure are as follows:

[0005] In a first aspect, a method for generating a video from an image is provided, comprising:

[0006] acquiring a first image;

[0007] Inputting the first image into the target image processing model, obtaining the first flow parameter for the first image output by the target image processing model; wherein the first flow parameter includes: at least one region, and the flow direction of each region, and the target image processing model is a neural network model;

[0008] The first image is processed based on the first flow parameter to generate a plurality of frames of second images, and the plurality of frames of second images are combined to obtain a video.

[0009] As an optional implementation of the embodiment of the present disclosure, the processing parameters also include: flow velocity in each area.

[0010] As an optional implementation of the embodiment of the present disclosure, the target image processing model is a neural network model trained based on sample information, and the sample information includes: a plurality of sample images, and a standard flow parameter of each sample image;

[0011] Before inputting the first image into the target image processing model and obtaining the first flow parameter output by the target image processing model for the first image, the method further includes:

[0012] Get sample information;

[0013] The following steps are executed at least once in a loop to obtain the target image processing model:

[0014] Acquire a target sample image from a plurality of sample images, and input the target sample image into an initial image processing model;

[0015] Acquire a second flow parameter of the target sample image output by the initial image processing model;

[0016] Determining a target loss function according to the second flow parameter and the standard flow parameter;

[0017] Based on the target loss function, the initial image processing model is modified.

[0018] As an optional implementation of the embodiment of the present disclosure, the target loss function includes at least one of the following:

[0019] Cross entropy loss function, total variation loss function, dice loss function, focal loss function, L1 regular loss function.

[0020] As an optional implementation of the embodiment of the present disclosure, obtaining sample information includes:

[0021] Get the original image;

[0022] Performing geometric transformation and / or color transformation on the original image to obtain at least one transformed image;

[0023] The original image and at least one transformed image are used as sample images in the sample information.

[0024] As an optional implementation of the embodiment of the present disclosure, the geometric transformation includes: at least one of flipping, rotating, cropping, deforming, and scaling;

[0025] As an optional implementation of the embodiment of the present disclosure, the color transformation includes: adding at least one of noise and color disturbance.

[0026] As an optional implementation of the embodiment of the present disclosure, inputting the first image into the target image processing model includes:

[0027] Performing a downsampling operation on the first image to obtain a downsampled first image;

[0028] The downsampled first image is input to the target image processing model.

[0029] As an optional implementation of the embodiment of the present disclosure, the flow velocity in the edge area of ​​each area is smaller than the flow velocity in the center area.

[0030] As an optional implementation of the embodiment of the present disclosure, the target image processing model includes: multiple downsampling operations, and / or multiple convolution operations,

[0031] The operation-related parameters for adjacent downsampling operations and / or adjacent convolution operations are different;

[0032] The operation-related parameters include at least one of the following:

[0033] Kernel size, dilation factor, step size.

[0034] As an optional implementation of the embodiment of the present disclosure, the target image processing model is a semantic segmentation model based on a high-resolution network model.

[0035] In a second aspect, a device for generating a video from an image is provided, comprising:

[0036] An acquisition module is used to acquire a first image; input the first image into a target image processing model, and acquire a first flow parameter for the first image output by the target image processing model; wherein the first flow parameter includes: at least one region, and a flow direction of each region, and the target image processing model is a neural network model;

[0037] The generating module is used to perform flow processing on the first image based on the first flow parameter to generate multiple frames of second images, and combine the multiple frames of second images to obtain a video.

[0038] As an optional implementation of the embodiment of the present disclosure, the processing parameters also include: flow velocity in each area.

[0039] As an optional implementation of the embodiment of the present disclosure, the target image processing model is a neural network model trained based on sample information, and the sample information includes: a plurality of sample images, and a standard flow parameter of each sample image;

[0040] The acquisition module is further used to: input the first image into the target image processing model, and before acquiring the first flow parameter output by the target image processing model for the first image, acquire the sample information;

[0041] The following steps are executed at least once in a loop to obtain the target image processing model:

[0042] Acquire a target sample image from a plurality of sample images, and input the target sample image into an initial image processing model;

[0043] Acquire a second flow parameter of the target sample image output by the initial image processing model;

[0044] Determining a target loss function according to the second flow parameter and the standard flow parameter;

[0045] Based on the target loss function, the initial image processing model is modified.

[0046] As an optional implementation of the embodiment of the present disclosure, the target loss function includes at least one of the following:

[0047] Cross entropy loss function, total variation loss function, dice loss function, focal loss function, L1 regular loss function.

[0048] As an optional implementation of the embodiment of the present disclosure, the acquisition module is specifically used to:

[0049] Get the original image;

[0050] Performing geometric transformation and / or color transformation on the original image to obtain at least one transformed image;

[0051] The original image and at least one transformed image are used as sample images in the sample information.

[0052] As an optional implementation of the embodiment of the present disclosure, the geometric transformation includes: at least one of flipping, rotating, cropping, deforming, and scaling;

[0053] As an optional implementation of the embodiment of the present disclosure, the color transformation includes: adding at least one of noise and color disturbance.

[0054] As an optional implementation of the embodiment of the present disclosure, the acquisition module is specifically used to:

[0055] Performing a downsampling operation on the first image to obtain a downsampled first image;

[0056] The downsampled first image is input to the target image processing model.

[0057] As an optional implementation of the embodiment of the present disclosure, the flow velocity in the edge area of ​​each area is smaller than the flow velocity in the center area.

[0058] As an optional implementation of the embodiment of the present disclosure, the target image processing model includes: multiple downsampling operations, and / or multiple convolution operations,

[0059] The operation-related parameters for adjacent downsampling operations and / or adjacent convolution operations are different;

[0060] The operation-related parameters include at least one of the following:

[0061] Kernel size, dilation factor, step size.

[0062] As an optional implementation of the embodiment of the present disclosure, the target image processing model is a semantic segmentation model based on a high-resolution network model.

[0063] According to a third aspect, an electronic device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, a method for generating a video from an image is implemented according to the first aspect or any one of its optional embodiments.

[0064] In a fourth aspect, a computer-readable storage medium is provided, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, a method for generating a video from an image as in the first aspect or any one of its optional embodiments is implemented.

[0065] In a fifth aspect, a computer program product is provided, comprising: when the computer program product is run on a computer, the computer implements a method for generating a video from an image as in the first aspect or any one of its optional embodiments.

[0066] The technical solution provided by the embodiment of the present disclosure has the following advantages compared with the prior art: a first image can be obtained; the first image is input into the target image processing model to obtain the first flow parameters for the first image output by the target image processing model; wherein the first flow parameters include: at least one region, the flow direction of each region, the target image processing model is a neural network model trained based on sample information, and the sample information includes: multiple sample images, and standard flow parameters for each sample image; the first image is flow processed based on the first flow parameters to generate multiple frames of second images, and the multiple frames of second images are combined to obtain a video. Through this solution, since the flow parameters (region, flow direction) corresponding to the first image can be generated based on the target image processing model, a video with a flow effect can be further generated based on the generated flow parameters and the first image, thereby realizing the generation of a video with a flow effect through a static image. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0069] Figure 1 A schematic diagram of a method for generating a video from an image provided in an embodiment of the present disclosure Figure 1 ;

[0070] Figure 2A schematic diagram of a method for generating a video from an image provided in an embodiment of the present disclosure Figure 2 ;

[0071] Figure 3 A structural block diagram of a device for generating video from an image provided by an embodiment of the present disclosure;

[0072] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0073] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0074] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0075] The terms "first" and "second" in the specification and claims of the present invention are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first image and a second image are used to distinguish different images rather than to describe a specific order of the images.

[0076] Some images may contain objects such as hair and clothes. When the dynamic effects of these objects need to be presented, a video with a flowing effect for the hair, clothes and other objects in the static image is required. Therefore, a method for generating a video with a flowing effect from a static image is urgently needed.

[0077] In some embodiments, a display effect of making a part of an image flow can be realized. In the specific implementation process, the user needs to manually select the area in the image and set the flow direction and other information before generating the video. In the process of generating a video with a flow effect from an image, since it is necessary to manually select the area and set the flow direction for these areas, the implementation complexity of the process of generating a video with a flow effect from an image is high.

[0078] The disclosed embodiments provide a method for generating videos from images. Since the method can generate flow parameters (area, flow direction) corresponding to a first image based on a target image processing model, a video with a flow effect can be further generated based on the generated flow parameters and the first image. Compared with manually selecting an area and setting a flow direction to generate a video with a flow effect, the implementation complexity of the process of generating a video with a flow effect from an image can be reduced.

[0079] The above-mentioned method of generating video from images can be applied to an apparatus or electronic device for generating video from images. The apparatus for generating video from images can be a functional module or functional entity in the electronic device that can implement the method of generating video from images.

[0080] The above-mentioned electronic device can be a server, a tablet computer, a mobile phone, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), a personal computer (PC), etc., and the embodiments of the present disclosure do not specifically limit this.

[0081] like Figure 1 As shown, it is a flowchart of a method for generating videos from images provided in an embodiment of the present disclosure. The method may include two stages: a model training stage and a practical application stage.

[0082] The model training phase includes the following steps 101 to 106.

[0083] 101. Obtain sample information.

[0084] The sample information includes: a plurality of sample images, and standard flow parameters of each sample image; the standard flow parameters may include at least one region, the flow direction of each region, and the flow speed of each region.

[0085] The above-mentioned area may refer to a flowable area. Some subjects in the image may flow in the actual scene, such as water flow, hair, clothing, etc. In the embodiment of the present disclosure, the water flow, hair, clothing, etc. in the image may be identified as a flowable area. In the embodiment of the present disclosure, it is also referred to as a flow area.

[0086] In some embodiments, obtaining sample information includes: obtaining an original image; performing a geometric transformation and / or a color transformation on the original image to obtain at least one transformed image; and using the original image and the at least one transformed image as sample images in the sample information.

[0087] In the actual model training process, in order to characterize the accuracy of the model, a large amount of image data is required as training samples, so it is necessary to make full use of the existing images for data enhancement to obtain more training samples. The meaning of data enhancement in this disclosure is to make limited data produce the value equivalent to more data without substantially increasing the data. When enhancing a sample image through data enhancement, the sample image can be geometrically transformed and / or color transformed to obtain multiple enhanced sample images.

[0088] The geometric transformation operation does not change the content of the image itself. The geometric transformation may include at least one of flipping, rotating, cropping, deforming, and scaling.

[0089] In the disclosed embodiment, geometric transformations such as random scaling and random cropping are performed on the image, and the transformed image is used as the sample image in the sample information. This can improve the accuracy of the target image processing model obtained by subsequent training in identifying flow areas of different sizes and locations.

[0090] In the disclosed embodiment, based on the situation where the flow direction is single in the hair area, clothing area, etc., it is proposed to randomly rotate the image, and use the randomly rotated image as the sample image in the sample information, which can improve the robustness of the target image processing model obtained by subsequent training to different flow directions.

[0091] The above-mentioned random flipping and random rotation will not change the size of the image, but random cropping will change the size of the image because it crops out part of the content of the original image. The image obtained after cropping will be smaller than the original image.

[0092] The color transformation may include: adding at least one of noise and color disturbance. The data enhancement of color transformation generally changes the content of the image.

[0093] In some embodiments, data enhancement based on adding noise is to randomly superimpose some noise on the original image. The most common is superimposing Gaussian noise. In some implementations, some pixels can be discarded in a rectangular area with a selectable size and random position, so that the image produces some colored noise.

[0094] Color perturbation is to change the color of the original image by adding or reducing certain color components or changing the order of color channels in a certain color space to obtain multiple images after color change.

[0095] In the disclosed embodiment, in view of the situation that light-colored hair and light-colored clothing may not be correctly segmented into flow areas, it is proposed to perform color perturbation on the original image to increase a variety of color-perturbed images as sample images. After training based on such image samples, the image processing model's misclassification of light-colored areas can be reduced.

[0096] In the disclosed embodiment, when acquiring sample information, the plurality of sample images may be images based on own image resources, and images obtained after data enhancement based on existing image resources. When acquiring the standard processing parameters of each sample image, the flow area mask may be annotated based on each sample image by manual annotation to obtain the flow area of ​​each sample image, and the vectors of the flow direction and flow speed in the flow area of ​​each sample image may be annotated by manual annotation to obtain the flow direction and flow speed in the flow area of ​​each sample image.

[0097] For example, Figure 2 As shown in the figure, it is a schematic diagram of the training process and application process of an image processing model. Figure 2 It can be seen that the flow area mask can be generated based on the original image through manual annotation, as well as the flow direction and flow speed of each flow area, and then this information can be used as sample information to train the image processing model.

[0098] The sample information may be a full-scene image flow dataset established based on own image resources.

[0099] In the embodiment of the present disclosure, different types of flow parameters are set for different scenes. That is, for each scene, there are some areas to be processed and flow parameters of these areas. That is, in the embodiment of the present invention, the first flow parameter may include: flow parameters corresponding to at least one scene, and the flow parameters corresponding to each scene include at least one similar area (i.e., the same type of flow area, such as 2 hair areas), and flow parameters of each area.

[0100] For the main coverage scenarios, the marking rules for flow area, flow direction and flow speed established by this patent are as follows:

[0101] (1) For scenes including human images: the hair and beard area of ​​the human is the flow area, the growth direction of the hair and beard is the flow direction, and the flow speed can be a uniform flow based on a certain fixed speed;

[0102] (2) For scenes including animal images: the hair area of ​​the animal is the flow area, the hair texture direction is the flow direction, and the flow speed can be a uniform flow based on a certain fixed speed.

[0103] (3) For scenes including animal images: the clothing area is the flow area, the direction of the wrinkles and curves of the clothing is the flow direction, the direction from top to bottom of the torso in the clothing area is the flow direction, the direction from shoulders to hands is the flow direction, or the direction from thighs to feet is the flow direction, and the flow speed can be a uniform flow based on a certain fixed speed.

[0104] (4) For scenes including sky images: for example, the cloud area is the flow area, the natural flow direction of the cloud is the flow direction, or the direction from left to right is not the flow direction, and the flow speed can be a uniform flow based on a certain fixed speed.

[0105] (5) For scenes involving water flow: the water flow area is the flow area, the direction of the natural flow of the water flow is the flow direction, or the direction of the water flow from high to low is the flow direction, and the flow speed can be a uniform flow based on a certain fixed speed.

[0106] For an image, using manual annotation information, it may eventually generate a mask including multiple categories of flow areas and a flow vector containing flow direction and flow speed information.

[0107] 102. Obtain a target sample image from a plurality of sample images, and input the target sample image into an initial image processing model.

[0108] The target sample image may be any sample image among multiple sample images.

[0109] 103. Obtain a second flow parameter of the target sample image output by the initial image processing model.

[0110] The second flow parameter is at least one flow area of ​​the target sample image, and the flow direction and flow speed of each flow area.

[0111] 104. Determine a target loss function according to the second flow parameter and the standard flow parameter.

[0112] 105. Based on the target loss function, correct the initial image processing model.

[0113] The objective loss function includes at least one of the following:

[0114] Cross entropy loss function, total variation loss function, dice loss function, focal loss function, L1 regular loss function.

[0115] In the disclosed embodiment, in order to ensure the accuracy of the algorithm, the cross entropy loss function, the total variation loss function and the L1 regularization loss function can be combined for weighted combination to supervise the prediction of the flow area, flow direction and flow speed in the image processing model.

[0116] The above-mentioned cross entropy loss function is mainly used to identify the accuracy of flow areas. Therefore, in some embodiments, setting the weight of the cross entropy loss function higher can improve the prediction accuracy of flow areas, while the L1 regularized loss function is mainly used to identify the accuracy of flow vectors (flow velocity and flow direction). Therefore, setting the weight of the L1 regularized loss function higher can improve the prediction accuracy of flow vectors.

[0117] Furthermore, since the total variation loss function acts on the smoothing of flow vectors within the flow area, in order to solve the problem of excessive differences in flow vectors in local areas leading to disordered flow effects, the disclosed embodiment introduces a total variation loss function to predict flow vectors, which can further improve the smoothness of the flow effect and make the picture effect smoother.

[0118] 106. Loop the above steps 102 to 105 at least once to obtain a target image processing model.

[0119] In the disclosed embodiment, in order to ensure the accuracy of the flow parameter prediction, the target image processing model adopts a semantic segmentation model based on a high resolution network model (High Resolution Network, HR Net), or the target image processing model is a semantic segmentation model based on a variant of the high resolution network model. Such models can have higher computational accuracy, with a larger amount of computation and more computational parameters, and are more suitable for being set on the service side, that is, being set in the server for use.

[0120] In some embodiments, the target image processing model includes: multiple downsampling operations, and / or multiple convolution operations.

[0121] In some embodiments, when setting multiple downsampling operations and operation-related parameters in the target image processing model, different operation-related parameters may be set for adjacent downsampling operations.

[0122] In some embodiments, when setting operation-related parameters of multiple convolution operations in the target image processing model, different operation-related parameters may be set for adjacent convolution operations.

[0123] The operation-related parameters include at least one of a kernel size, a dilation coefficient, and a stride.

[0124] That is to say, for adjacent downsampling operations, at least one of the downsampling kernel size, the downsampling dilation coefficient, and the downsampling step size may be set differently; for adjacent convolution operations, at least one of the convolution kernel size (kernel size), the convolution dilation coefficient (dilate), and the convolution step size may also be set differently.

[0125] In the disclosed embodiment, different operation-related parameters are set for adjacent downsampling operations or convolution operations in the model network, so as to avoid the gridding effect caused by processing image data at a fixed position each time when performing the downsampling operation or convolution operation, thereby improving the gridding effect problem in the predicted flow area mask.

[0126] The actual application stage includes the following steps 107 to 110.

[0127] 107. Acquire a first image.

[0128] like Figure 2 As shown in, a user can trigger an electronic device to generate a video with a flow effect based on a first image through user input. The user input process can trigger the use of an image flow service (which can be a special effects prop associated with an image processing model) to generate a video with a flow effect. When the image flow service is triggered, a trained image processing model (i.e., a target image processing model) will be called to process the first image and predict the corresponding flow parameters.

[0129] 108. Input the first image into a target image processing model, and obtain a first flow parameter for the first image output by the target image processing model.

[0130] The first flow parameter includes: at least one region (also referred to as a flow region in the embodiment of the present disclosure), a flow direction of each region, and a flow speed of each region.

[0131] In some embodiments, inputting the first image into the target image processing model includes: performing a downsampling operation on the first image to obtain a downsampled first image, and inputting the downsampled first image into the target image processing model.

[0132] Image downsampling can be understood as: for an image with a resolution of M*N, downsample it by s times, that is, obtain a resolution image of (M / s)*(N / s), where s is the common divisor of M and N. In the process of downsampling, each s*s pixel of the original image is converted into a pixel, and the value of this pixel can be the mean value of all pixels in the window.

[0133] Exemplarily, downsampling the first image can convert the first image into a small-size image, thereby reducing the amount of computation and time consumption of the target image processing model.

[0134] In some embodiments, the first flow parameter may not include the flow speed. In this case, the target image processing model may not predict the flow speed, and the flow speed may be a default fixed flow speed.

[0135] 109. Perform flow processing on the first image based on the first flow parameter to generate multiple frames of second images.

[0136] 110. Combine multiple frames of second images to obtain a video.

[0137] like Figure 2 As shown, based on the first image and the target image processing model, at least one flow area in the first image, as well as the flow direction and flow speed of each flow area can be predicted. Furthermore, based on these predicted flow parameters, the first image is subjected to flow processing, and multiple frames of second images can be obtained in the time sequence of the video frames. In this way, a video, i.e., a video with a flow effect, can be obtained.

[0138] When the first image is subjected to flow processing based on the first flow parameter, a plurality of frames of second images may be generated in a time sequence, and the plurality of second images may be combined according to the time sequence, thereby obtaining a video with a flow effect.

[0139] In order to solve the problem of afterimages at the boundaries of multiple flow areas and prevent flow in non-flow areas, the disclosed embodiments can utilize the object significance to reduce the flow vectors at the boundaries of multiple flow areas, avoid pixel cross-borders between different areas, and restrict the flow speed within the flow area in different regions and levels. Specifically, when the flow processing is performed on the first image based on the flow parameters, the flow speed in the edge area of ​​each flow area can be limited to be smaller than the flow speed in the center area.

[0140] Furthermore, different flow speed ranges may be set for edge areas and central areas in the flow areas, and the flow speed in the edge areas of each flow area may be limited based on the corresponding flow speed ranges, as may the flow speed in the central area of ​​each flow area.

[0141] The method for generating a video from an image provided by the embodiment of the present disclosure can obtain a first image; input the first image into a target image processing model, and obtain a first flow parameter for the first image output by the target image processing model; wherein the first flow parameter includes: at least one flow area, and the flow direction of each flow area, and the target image processing model is a neural network model trained based on sample information, and the sample information includes: multiple sample images, and standard flow parameters for each sample image; flow processing is performed on the first image based on the first flow parameter to generate multiple frames of second images, and the multiple frames of second images are combined to obtain a video. Through this solution, since the flow parameters (area, flow direction) corresponding to the first image can be generated based on the target image processing model, a video with a flow effect can be further generated based on the generated flow parameters and the first image, thereby realizing a video with a flow effect through a static image.

[0142] like Figure 3 As shown, an embodiment of the present disclosure provides a structural block diagram of a device for generating a video from an image, the device comprising:

[0143] The acquisition module 301 is used to acquire a first image; input the first image into a target image processing model, and acquire a first flow parameter for the first image output by the target image processing model; wherein the first flow parameter includes: at least one region, and a flow direction of each region, and the target image processing model is a neural network model;

[0144] The generating module 302 is used to perform flow processing on the first image based on the first flow parameter to generate multiple frames of second images, and combine the multiple frames of second images to obtain a video.

[0145] As an optional implementation of the embodiment of the present disclosure, the processing parameters also include: flow velocity in each area.

[0146] As an optional implementation of the embodiment of the present disclosure,

[0147] The target image processing model is a neural network model trained based on sample information, wherein the sample information includes: a plurality of sample images and a standard flow parameter of each sample image;

[0148] The acquisition module 301 is further used to: input the first image into the target image processing model, and before acquiring the first flow parameter output by the target image processing model for the first image, acquire sample information;

[0149] The following steps are executed at least once in a loop to obtain the target image processing model:

[0150] Acquire a target sample image from a plurality of sample images, and input the target sample image into an initial image processing model;

[0151] Acquire a second flow parameter of the target sample image output by the initial image processing model;

[0152] Determining a target loss function according to the second flow parameter and the standard flow parameter;

[0153] Based on the target loss function, the initial image processing model is modified.

[0154] As an optional implementation of the embodiment of the present disclosure, the target loss function includes at least one of the following:

[0155] Cross entropy loss function, total variation loss function, dice loss function, focal loss function, L1 regular loss function.

[0156] As an optional implementation of the embodiment of the present disclosure, the acquisition module is specifically used to:

[0157] Get the original image;

[0158] Performing geometric transformation and / or color transformation on the original image to obtain at least one transformed image;

[0159] The original image and at least one transformed image are used as sample images in the sample information.

[0160] As an optional implementation of the embodiment of the present disclosure, the geometric transformation includes: at least one of flipping, rotating, cropping, deforming, and scaling;

[0161] As an optional implementation of the embodiment of the present disclosure, color transformation includes: adding at least one of noise and color disturbance.

[0162] As an optional implementation of the embodiment of the present disclosure, the acquisition module 301 is specifically used to:

[0163] Performing a downsampling operation on the first image to obtain a downsampled first image;

[0164] The downsampled first image is input to the target image processing model.

[0165] As an optional implementation of the embodiment of the present disclosure, the flow velocity in the edge area of ​​each area is smaller than the flow velocity in the center area.

[0166] As an optional implementation of the embodiment of the present disclosure, the target image processing model includes: multiple downsampling operations, and / or multiple convolution operations,

[0167] The operation-related parameters for adjacent downsampling operations and / or adjacent convolution operations are different;

[0168] The operation-related parameters include at least one of the following:

[0169] Kernel size, dilation factor, step size.

[0170] As an optional implementation of the embodiment of the present disclosure, the target image processing model is a semantic segmentation model based on a high-resolution network model.

[0171] like Figure 4 As shown, an embodiment of the present disclosure provides an electronic device, including: a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401, wherein when the computer program is executed by the processor 401, the method for generating video from images involved in the above method embodiment is implemented.

[0172] The computer readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0173] The embodiment of the present disclosure provides a computer-readable storage medium, including: a computer program stored on the computer-readable storage medium, and when the computer program is executed by a processor, the method of generating a video from an image in the above method embodiment is implemented.

[0174] The embodiment of the present disclosure provides a computer program product, including: when the computer program product is run on a computer, the computer is enabled to implement the method of generating a video from an image in the above method embodiment.

[0175] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0176] In the present disclosure, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0177] In the present disclosure, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0178] In the present disclosure, computer-readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0179] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0180] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a video from an image, characterized in that: include: acquiring a first image; Input the first image into a target image processing model, and obtain a first flow parameter for the first image output by the target image processing model; wherein the first flow parameter includes: at least one region and a flow direction of each region, and the target image processing model is a neural network model; Processing the first image based on the first flow parameter to generate a plurality of frames of second images, and combining the plurality of frames of second images to obtain a video; The target image processing model is obtained by cyclically correcting the initial image processing model according to a target loss function determined by inputting a sample image of sample information into the initial image processing model to obtain a second flow parameter and a standard flow parameter of the sample image.

2. The method according to claim 1, characterized in that: The first flow parameter also includes: a flow velocity in each area.

3. The method according to claim 2, characterized in that The target image processing model is a neural network model trained based on sample information, wherein the sample information includes: a plurality of sample images and a standard flow parameter of each sample image; Before inputting the first image into the target image processing model and obtaining the first flow parameter output by the target image processing model for the first image, the method further includes: Obtaining the sample information; The following steps are executed at least once in a loop to obtain the target image processing model: Acquire a target sample image from the multiple sample images, and input the target sample image into an initial image processing model; acquire a second flow parameter of the target sample image output by the initial image processing model; determining a target loss function according to the second flow parameter and the standard flow parameter; Based on the target loss function, the initial image processing model is modified.

4. The method according to claim 3, characterized in that The objective loss function includes at least one of the following: Cross entropy loss function, total variation loss function, dice loss function, focal loss function, L1 regular loss function.

5. The method according to claim 3, characterized in that: The obtaining of the sample information includes: Get the original image; Performing geometric transformation and / or color transformation on the original image to obtain at least one transformed image; The original image and the at least one transformed image are used as sample images in the sample information.

6. The method according to claim 5, characterized in that The geometric transformation includes: at least one of flipping, rotating, cropping, deforming, and scaling; and / or, The color transformation includes: adding at least one of noise and color disturbance.

7. The method according to claim 1, characterized in that The step of inputting the first image into a target image processing model comprises: Performing a downsampling operation on the first image to obtain a downsampled first image; The downsampled first image is input into the target image processing model.

8. The method according to claim 2, characterized in that: The flow velocity in the edge region of each region is smaller than the flow velocity in the central region.

9. The method according to any one of claims 1 to 8, characterized in that: The target image processing model includes: multiple downsampling operations, and / or multiple convolution operations, The operation-related parameters of adjacent downsampling operations and / or adjacent convolution operations are different; The operation-related parameters include at least one of the following: Kernel size, dilation factor, step size.

10. The method according to claim 1, characterized in that The target image processing model is a semantic segmentation model based on a high-resolution network model.

11. A device for generating video from an image, characterized in that: include: An acquisition module, used for acquiring a first image; Inputting the first image into a target image processing model, and obtaining a first flow parameter for the first image output by the target image processing model; wherein the first flow parameter includes: at least one region and a flow direction of each region; A generating module, configured to perform flow processing on the first image based on the first flow parameter to generate a plurality of frames of second images, and combine the plurality of frames of second images to obtain a video; The target image processing model is obtained by cyclically correcting the initial image processing model according to a target loss function determined by inputting a sample image of sample information into the initial image processing model to obtain a second flow parameter and a standard flow parameter of the sample image.

12. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method for generating a video from an image according to any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating a video from an image according to any one of claims 1 to 10 is implemented.