Optical flow prediction method, device, electronic device and storage medium

Abstract

The present disclosure relates to an optical flow prediction method, device, electronic device and storage medium. The optical flow prediction method includes: acquiring an image including a sky area; acquiring texture features of the image, and predicting the optical flow in the image based on the acquired texture features of the image; determining the sky area in the image by segmenting the image; and obtaining the optical flow of the sky area based on the predicted optical flow and the determined sky area.

Description

Technical Field

[0001] The present disclosure relates to the field of signal processing, and in particular to an optical flow prediction method, device, electronic device and storage medium. Background Art

[0002] The popularity of smartphones allows everyone to take photos and videos anytime and anywhere to record the wonderful moments of daily life and travel. Among the materials people take daily, landscape photos and portrait photos with sky backgrounds account for a considerable proportion. At present, there are quite a lot of computer vision and computer graphics technologies that can help people edit sky photos and videos to create more attractive multimedia content. For example, people can replace the gray sky in the original image with the blue sky, or add interesting dynamic objects to the sky area in the photo, or generate sky gifs. Unlike other sky photo editing technologies, sky gif generation does not require the use of additional sky images, and the generated video is still based on the sky background in the original photo. As long as the motion pattern of the sky area is known, this technology can generate sky dynamic videos (also called "sky gifs"). However, in practice, the inability to accurately predict sky optical flow often results in poor results in some subsequent applications based on sky optical flow. For example, if the predicted sky optical flow is not accurate enough, the visual effect of the generated sky dynamic video will be poor (for example, the sky dynamic image is not realistic enough, or there are visual defects). In other words, how to better predict the sky optical flow largely determines the visual effect of the generated sky dynamic image. In view of this, a technology that can more accurately predict sky optical flow is needed. Summary of the invention

[0003] The present disclosure provides an optical flow prediction method, an apparatus electronic device and a storage medium to at least solve the problem of poor optical flow prediction accuracy in the related art.

[0004] According to a first aspect of an embodiment of the present disclosure, an optical flow prediction method is provided, wherein the optical flow prediction method comprises: acquiring an image including a sky area; acquiring texture features of the image, and predicting the optical flow in the image based on the acquired texture features of the image; determining the sky area in the image by segmenting the image; and obtaining the optical flow of the sky area based on the predicted optical flow and the determined sky area.

[0005] Optionally, obtaining the optical flow of the sky area based on the predicted optical flow and the determined sky area includes: filtering out the optical flow of the non-sky area by comparing the predicted optical flow with the determined sky area; determining a situation in which the remaining optical flow after filtering out the optical flow of the non-sky area covers the sky area, and obtaining the optical flow of the sky area based on the remaining optical flow according to the determined situation.

[0006] Optionally, the determining of whether the remaining optical flow after filtering out the optical flow of the non-sky area covers the sky area, and obtaining the optical flow of the sky area based on the remaining optical flow according to the determined situation, includes: determining whether the remaining optical flow after filtering out the optical flow of the non-sky area covers more than a predetermined proportion of the sky area; if it covers more than the predetermined proportion of the sky area, determining the remaining optical flow as the optical flow of the sky area; if it does not cover more than the predetermined proportion of the sky area, determining the completeness of the remaining optical flow based on the area of ​​the sky area covered by the remaining optical flow; if the completeness exceeds a preset threshold, obtaining the optical flow of the sky area through optical flow propagation based on the remaining optical flow; if the completeness does not exceed the preset threshold, obtaining the optical flow of the sky area by using a preset optical flow template and the determined sky area.

[0007] Optionally, the using a preset optical flow template and a determined sky area to obtain the optical flow of the sky area includes: based on the characteristics of the determined sky area, selecting an optical flow template with the highest matching degree with the sky area from multiple preset optical flow templates, and obtaining the optical flow of the sky area according to the selected optical flow template.

[0008] Optionally, obtaining the optical flow of the sky area according to the selected optical flow template includes: obtaining the optical flow of the sky area by performing random optical flow perturbation on the selected optical flow template.

[0009] Optionally, the feature of the sky area includes at least one of a position, a shape and a size of the sky area in the image.

[0010] Optionally, predicting the optical flow in the image based on the acquired texture features of the image includes: based on the texture features, predicting the optical flow in the image using a pre-trained deep generative adversarial network model, wherein the deep generative adversarial network model is trained based on a time-lapse video of the sky.

[0011] Optionally, determining the sky area in the image by segmenting the image includes: based on the image, using a pre-trained sky segmentation model to distinguish between the sky area and the non-sky area in the image, wherein the sky segmentation model is trained based on sky segmentation annotation data in the sky time-lapse video.

[0012] According to a second aspect of an embodiment of the present disclosure, an optical flow prediction device is provided, comprising: an image acquisition unit, configured to acquire an image including a sky area; an image optical flow prediction unit, configured to acquire texture features of the image, and predict the optical flow in the image based on the acquired texture features of the image; a sky area determination unit, configured to determine the sky area in the image by segmenting the image; and a sky optical flow acquisition unit, configured to acquire the optical flow of the sky area based on the predicted optical flow and the determined sky area.

[0013] Optionally, obtaining the optical flow of the sky area based on the predicted optical flow and the determined sky area includes: filtering out the optical flow of the non-sky area by comparing the predicted optical flow with the determined sky area; determining a situation in which the remaining optical flow after filtering out the optical flow of the non-sky area covers the sky area, and obtaining the optical flow of the sky area based on the remaining optical flow according to the determined situation.

[0014] Optionally, the determining of whether the remaining optical flow after filtering out the optical flow of the non-sky area covers the sky area, and obtaining the optical flow of the sky area based on the remaining optical flow according to the determined situation, includes: determining whether the remaining optical flow after filtering out the optical flow of the non-sky area covers more than a predetermined proportion of the sky area; if it covers more than the predetermined proportion of the sky area, determining the remaining optical flow as the optical flow of the sky area; if it does not cover more than the predetermined proportion of the sky area, determining the completeness of the remaining optical flow based on the area of ​​the sky area covered by the remaining optical flow; if the completeness exceeds a preset threshold, obtaining the optical flow of the sky area through optical flow propagation based on the remaining optical flow; if the completeness does not exceed the preset threshold, obtaining the optical flow of the sky area by using a preset optical flow template and the determined sky area.

[0015] Optionally, the using a preset optical flow template and a determined sky area to obtain the optical flow of the sky area includes: based on the characteristics of the determined sky area, selecting an optical flow template with the highest matching degree with the sky area from multiple preset optical flow templates, and obtaining the optical flow of the sky area according to the selected optical flow template.

[0016] Optionally, obtaining the optical flow of the sky area according to the selected optical flow template includes: obtaining the optical flow of the sky area by performing random optical flow perturbation on the selected optical flow template.

[0017] Optionally, the feature of the sky area includes at least one of a position, a shape and a size of the sky area in the image.

[0018] Optionally, predicting the optical flow in the image based on the acquired texture features of the image includes: based on the texture features, predicting the optical flow in the image using a pre-trained deep generative adversarial network model, wherein the deep generative adversarial network model is based on the sky time-lapse video and is trained by learning the mapping relationship between the sky texture in the sky time-lapse video and the sky optical flow.

[0019] Optionally, determining the sky area in the image by segmenting the image includes: based on the image, using a pre-trained sky segmentation model to distinguish between the sky area and the non-sky area in the image, wherein the sky segmentation model is trained based on sky segmentation annotation data in the sky time-lapse video.

[0020] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, characterized in that it includes: at least one processor; and at least one memory storing computer executable instructions, wherein when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the optical flow prediction method as described above.

[0021] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the optical flow prediction method as described above.

[0022] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product, comprising computer instructions, wherein the computer instructions implement the optical flow method as described above when executed by a processor.

[0023] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: According to the optical flow prediction method of the embodiments of the present disclosure, the optical flow in the image is predicted based on the texture features of the acquired image, and the sky area in the image is determined by segmenting the image, and then the optical flow of the sky area is obtained based on both the predicted optical flow and the determined sky area, so that the optical flow of the sky area can be predicted more accurately.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate exemplary embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0026] Figure 1is an exemplary system architecture in which exemplary embodiments of the present disclosure may be applied;

[0027] Figure 2 is a flowchart of an optical flow prediction method of an exemplary embodiment of the present disclosure;

[0028] Figure 3 is a schematic diagram illustrating an optical flow prediction method of an exemplary embodiment of the present disclosure;

[0029] Figure 4 is an example showing the optical flow of a sky area obtained by using the optical flow prediction method of an exemplary embodiment of the present disclosure;

[0030] Figure 5 is a block diagram showing an optical flow prediction apparatus of an exemplary embodiment of the present disclosure;

[0031] Figure 6 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0034] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0035] Figure 1 An exemplary system architecture 100 is shown in which exemplary embodiments of the present disclosure may be applied.

[0036] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. Users can use terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages (such as image or video data upload requests, image or video data download requests), etc. Various communication client applications may be installed on the terminal devices 101, 102, 103, such as audio and video communication software, audio and video recording software, instant communication software, conference software, mailbox client, social platform software, etc. In addition, various image or video shooting and editing applications may be installed on the terminal devices 101, 102 and 103. The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with display screens and capable of playing, recording, and editing audio and video, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers. When the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above, and can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made here.

[0037] The terminal devices 101, 102, and 103 may be equipped with image acquisition devices (such as cameras) to collect image or video data. In practice, the smallest visual unit that constitutes a video is a frame. Each frame is a static image. Synthesizing a temporally continuous sequence of frames together forms a dynamic video. In addition, the terminal devices 101, 102, and 103 may also be equipped with components for converting electrical signals into sounds (such as speakers) to play sounds, and may also be equipped with devices for converting analog audio signals into digital audio signals (such as microphones) to collect sounds. In addition, the terminal devices 101, 102, and 103 may communicate with each other by voice or video.

[0038] The server 105 may be a server that provides various services, such as a background server that provides support for multimedia applications installed on the terminal devices 101, 102, and 103. The background server may parse, store, and process the received audio and video data upload request and other data, and may also receive the audio and video data download request sent by the terminal devices 101, 102, and 103, and feed back the audio and video data indicated by the audio and video data download request to the terminal devices 101, 102, and 103.

[0039] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0040] It should be noted that the optical flow prediction method provided in the embodiments of the present disclosure is usually executed by a terminal device, but can also be executed by a server, or can also be executed by a terminal device and a server in collaboration. Accordingly, the optical flow prediction device can be set in the terminal device, in the server, or in both the terminal device and the server.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers, and the present disclosure has no limitation on this.

[0042] With the advancement of artificial intelligence technology, most of the current sky optical flow prediction methods mainly use deep neural network models to learn from massive sky time-lapse videos how to judge the dynamic pattern of the sky area based on the texture features in the picture. In other words, this method directly predicts the sky optical flow in the image. This method extracts real optical flow from the sky time-lapse video and uses it as annotation data for training the neural network. Due to the good learning ability of the deep neural network, when the model is trained with a large amount of data, it has the ability to predict optical flow based on the image scene. For example, these methods can judge the direction of movement of the entire sky area based on the shape and distribution of clouds, and at the same time, combine local texture features to fine-tune the dynamic change direction and amplitude of each sky pixel. The biggest advantage of these methods is that with the help of existing optical flow extraction algorithms and massive video data, the mapping relationship between static sky texture and optical flow (motion pattern) can be autonomously learned without time-consuming and labor-intensive manual annotation. When a new sky photo is given, the sky image optical flow can be obtained without manual intervention. However, the disadvantage of these methods is that the data for training the algorithm mainly comes from time-lapse videos. However, when the time-lapse video contains other dynamically changing areas, such as rivers and lakes, the sky optical flow obtained by the optical flow extraction algorithm will have labeling errors, causing the trained model to predict optical flow in some non-sky areas. Limited by the scale of training data, it is difficult to generate realistic dynamic images using the optical flow of these non-sky areas. In addition, the sky scenes contained in the sky time-lapse video are limited after all, and the diversity of sky images in real scenes is very high. This leads to the optical flow generated by this type of method containing more errors, which is manifested as the optical flow prediction of some sky pixels is zero, while the non-sky area obtains a larger optical flow. As a result, the problem brought to the generation of sky dynamic videos is that some sky areas are stationary, while the stationary objects undergo image displacement, which greatly affects the visual effect of the sky animation.

[0043] Compared with directly predicting the sky optical flow in an image, sky segmentation is a relatively easy computer vision problem. Therefore, obtaining the sky area in an image through sky segmentation and giving it a dynamic change pattern is considered to be a relatively robust method for generating sky optical flow. Based on the results of sky segmentation, this method first uses a neural network encoder to extract deep features from the sky area; then uses a pre-defined homography matrix to perform a projective transformation on the deep features; and finally generates the corresponding video frame through a decoder. Due to the use of the results of sky segmentation, this method can well suppress the optical flow in non-sky areas, and at the same time, most of the sky pixels can obtain relatively good dynamic textures. However, the problem with this method is that it needs to use a manually defined homography matrix to generate sky dynamics. Although changing the homography matrix can also change the motion pattern of the sky, this method ignores the importance of the image features of the sky area in determining the sky dynamic pattern, which easily leads to the sky dynamics generated by it not being consistent with the picture scene. For example, in some sky images, the texture displacement up and down is more reasonable visually, while this method is likely to give it horizontal dynamic changes.

[0044] To sum up, as mentioned in the background technology, in practice, the inability to accurately predict the sky optical flow often leads to poor effects of some subsequent applications based on the sky optical flow. For example, if the predicted sky optical flow is not accurate enough, the visual effect of the generated sky animation will be poor (for example, the sky animation is not realistic enough, or there are visual flaws).

[0045] The present invention proposes a robust method for predicting sky optical flow based on a single image, so that a harmonious and realistic motion pattern (optical flow) can be obtained on the vast majority of real sky images, thereby facilitating the subsequent generation of sky dynamic videos with good visual effects and few image quality issues based on the predicted optical flow.

[0046] Figure 2 is a flowchart of an optical flow prediction method according to an exemplary embodiment of the present disclosure.

[0047] Reference Figure 2 , in step S210, an image including a sky area is acquired. Here, the image may be a single image including a sky area. As an example, an image including a sky area may be acquired in response to an operation instruction of a user for acquiring an image. Here, the operation instruction may be, for example, a touch operation of a user on a user interface, but is not limited thereto. Any method of acquiring an image (manual or automatic) may be used to acquire an image including a sky area, and the present disclosure is not limited thereto.

[0048] In step S220, texture features of the image are obtained, and the optical flow in the image is predicted based on the obtained texture features of the image. Specifically, in step S220, after the texture features of the image are obtained, the optical flow in the image can be predicted based on the texture features using a pre-trained deep generative adversarial network model. Here, optical flow refers to the displacement of each pixel on the image in the image coordinate system, which corresponds to the motion pattern of the image. The deep generative adversarial network model is used to predict the optical flow in the entire image, and is hereinafter also referred to as an "optical flow prediction model."

[0049] The deep generative adversarial network model is a type of deep neural network model. Its working principle in the prediction stage is no different from that of the traditional deep neural network. It predicts the optical flow of each image pixel based on the texture features of the image input into the deep neural network. The main difference between it and the traditional deep neural network is the strategy used in model training. The traditional deep neural network only needs to learn the pixel-level mapping relationship between static texture and optical flow. In addition to considering the local pixel-level mapping relationship, the deep generative adversarial network model also introduces an adversarial loss function based on the entire image to make the predicted optical flow more compatible with the global image scene, thereby reducing the optical flow prediction distortion problem caused by local overfitting and improving the generalization ability of the model.

[0050] The deep generative adversarial network model can be based on the sky time-lapse video, and is trained by learning the mapping relationship between the sky texture and the sky optical flow in the sky time-lapse video. By collecting a large number of sky time-lapse videos to cover more sky textures and sky motion patterns (i.e., sky optical flow), the ability of the model to adapt to actual application scenarios can be improved. As is well known to those skilled in the art, the deep generative adversarial network model can generally include a generative network and a discriminant network, and when the deep generative adversarial network is used to predict the sky optical flow, the generative network can generate a predicted sky optical flow based on the sky texture in the picture of the sky time-lapse video, and the discriminant network can discriminate whether the predicted sky optical flow is the real sky optical flow in the sky time-lapse video. In other words, the output of the discriminant network is the probability that the sky optical flow predicted by the generative network is the real sky optical flow. By continuously using the adversarial loss function based on the entire image for training, the generative network can predict a sky optical flow that is sufficient to "make the fake look real", and for the discriminant network, it is basically difficult to discriminate whether the optical flow predicted by the generative network is real, and the training is completed. Through the above process, we realized the training of a deep generative adversarial network model based on sky time-lapse video by learning the mapping relationship between sky texture and sky optical flow in sky time-lapse video.

[0051] Therefore, the present disclosure uses a deep generative adversarial network model in step S220 to predict the optical flow in the image based on the texture features of the image, which can effectively reduce the optical flow prediction distortion caused by local overfitting and can adapt to actual application scenarios.

[0052] Next, in step S230, the sky area in the image is determined by segmenting the image. According to an exemplary embodiment, a pre-trained sky segmentation model (also referred to as a "sky cutout model") may be used based on the image to distinguish between the sky area and the non-sky area in the image. Here, the sky segmentation model is trained based on the sky segmentation annotation data in the sky time-lapse video. The sky time-lapse video here may be the sky time-lapse video used in training the optical flow prediction model above.

[0053] The sky segmentation model can distinguish between the sky area and the non-sky area in the image, so as to help us obtain the optical flow of the sky area based on the optical flow predicted in step S220 and the determined sky area (hereinafter, also referred to as optimizing the optical flow). According to an exemplary embodiment, in order to better cope with the subsequent application of generating sky animations, a large amount of sky segmentation annotation data can be used to train the sky segmentation model. The sky segmentation model is to perform binary classification on each pixel in the image, so as to distinguish between the sky area and the non-sky area in the image. Any pixel-level binary classification model can be used as the sky segmentation model here, and preferably, the sky segmentation model here can be a deep neural network model. The advantage of the deep neural network model is that it is possible to automatically learn which image features can represent the sky area from a large number of sky segmentation annotation images without manually designing image features and classification models. When using a deep network model with a full convolutional layer, the sky segmentation model can be applied to images of various scales and sizes. As an example, the sky segmentation model can be trained by performing the following operations: extracting sky segmentation annotation data from a sky time-lapse video, optimizing the fineness of sky annotations on complex edges, and training the sky segmentation model based on the optimized sky segmentation annotation data. In the sky time-lapse video, only the sky area has obvious dynamic changes, while the non-sky area has no or very slight image changes. Therefore, when extracting sky segmentation annotation data from the sky time-lapse video, the image motion change information can be used to distinguish the sky area and the non-sky area, thereby obtaining the sky segmentation annotation data.

[0054] After determining the sky area in the image by segmenting the image, in step S240, the optical flow of the sky area can be obtained based on the predicted optical flow and the determined sky area. Based on the result of sky segmentation, the dynamic changes of the non-sky area can be well suppressed, while allowing the sky area to present a more complete dynamic texture. Specifically, in step S240, the optical flow of the non-sky area can be first filtered out by comparing the predicted optical flow with the determined sky area, and then, it is determined whether the remaining optical flow after filtering out the optical flow of the non-sky area covers the sky area, and the optical flow of the sky area is obtained based on the remaining optical flow according to the determined situation.

[0055] As an example, determining whether the remaining optical flow after filtering out the optical flow of the non-sky area covers the sky area, and obtaining the optical flow of the sky area based on the remaining optical flow according to the determined situation may include: first, determining whether the remaining optical flow after filtering out the optical flow of the non-sky area covers more than a predetermined proportion of the sky area. Secondly, if the coverage exceeds the predetermined proportion of the sky area (in this case, the remaining optical flow can be considered to be a complete sky optical flow), the remaining optical flow is determined as the optical flow of the sky area. For example, using the segmented sky area, it is possible to determine the proportion of sky pixels of the predicted optical flow to the sky area in the image, and when this proportion is greater than a predetermined proportion (for example, 95%), the predicted sky optical flow can be considered to be complete. On the contrary, if it does not cover more than a predetermined proportion of the sky area (in this case, it can be considered that the remaining optical flow is not a complete sky optical flow), the completeness of the remaining optical flow is determined based on the area of ​​the sky area covered by the remaining optical flow. For example, the completeness can be the ratio between the area of ​​the sky area covered by the remaining optical flow and the area of ​​the sky area, or a value obtained based on the ratio (for example, a value obtained by multiplying the ratio by a predetermined value), or even the completeness can be the area of ​​the sky area covered by the remaining optical flow. It should be noted that the completeness can be defined in a variety of ways, and the present disclosure is not limited to this, as long as it is determined based on the area of ​​the sky area covered by the remaining optical flow. If the completeness exceeds the preset threshold, the optical flow of the sky area is obtained by optical flow propagation based on the remaining optical flow. That is to say, if the optical flow field at this time does not cover most of the sky area, we use the method of optical flow propagation to transmit the known optical flow to the entire sky area to obtain a more complete sky optical flow. Here, optical flow propagation is to transfer the predicted sky optical flow to the sky area where the optical flow is not predicted through the adjacent principle. For example, a breadth-first search algorithm can be used for optical flow propagation. On the contrary, if the completeness does not exceed the preset threshold, the optical flow of the sky area is obtained by using a preset optical flow template and a determined sky area. The reason is that in some extreme scenarios, sometimes only a very small area of ​​sky optical flow can be obtained, which can be regarded as very unreliable. At this time, a pre-set optical flow template can be used to fill the sky area in the image. For example, based on the characteristics of the determined sky area, an optical flow template with the highest matching degree with the sky area can be selected from multiple preset optical flow templates, and the optical flow of the sky area can be obtained according to the selected optical flow template. For example, the optical flow of the sky area can be obtained by performing random optical flow perturbations on the selected optical flow template, which can increase the local diversity of the optical flow in the image area. As an example, the characteristics of the sky area may include at least one of the position, shape and size of the sky area in the image, but are not limited to this.

[0056] Optionally, Figure 2The method shown may also include: generating a sky animation according to the obtained optical flow of the sky area. Figure 2 The optical flow prediction method shown combines optical flow prediction with sky segmentation to obtain the optical flow of the sky area, that is, the optical flow of the sky area is obtained based on the predicted optical flow and the determined sky area, so a more accurate sky optical flow can be obtained. Accordingly, the sky animation generated based on the more accurate sky optical flow is more realistic, more in line with the sky scene in the image, and has a better visual effect.

[0057] Figure 3 is a schematic diagram showing an optical flow prediction method of an exemplary embodiment of the present disclosure. Figure 2 The optical flow prediction method shown below refers to Figure 3 A brief description Figure 2 An example of the optical flow prediction method shown.

[0058] like Figure 3 As shown, after acquiring an image including the sky ( Figure 3 After the sky image is obtained (referred to as the "sky image" in the image), the sky image is input into the above-mentioned optical flow prediction model and sky cutout model respectively. Subsequently, the optical flow in the image is predicted by the optical flow prediction model, and the sky area is determined by the sky cutout model. Next, based on the optical flow predicted by the optical flow prediction model and the sky area determined by the sky cutout model, the optical flow of the sky area can be obtained.

[0059] Specifically, first, it is possible to determine whether the sky optical flow is complete in the manner mentioned in the description of step S240 above. If it is complete, it is determined as the final optical flow of the sky area. If it is incomplete, it is further determined whether the completeness exceeds a preset threshold (for example, X). If the completeness exceeds the preset threshold, the known optical flow is transmitted to the entire sky area through optical flow propagation, thereby obtaining the optical flow of the sky area. On the contrary, if the completeness does not exceed the preset threshold, the optical flow template with the highest matching degree with the sky area can be selected from multiple preset optical flow templates based on the characteristics of the determined sky area, and the optical flow of the sky area is obtained by performing random optical flow perturbations on the selected optical flow template.

[0060] By combining the deep learning-based optical flow prediction model with the sky cutout model, the sky optical flow can be robustly predicted. In addition, as mentioned in the description of step S240 above, for some extreme cases, a pre-set sky optical flow template can be applied to obtain the optical flow of the sky area. Therefore, for sky images in various real scenes, the optical flow prediction method of the exemplary embodiment of the present disclosure can predict the sky optical flow with good effect, thereby generating a dynamic short video of the sky with good visual effect and few image quality problems.

[0061] Figure 42 is an example showing an optical flow of a sky area obtained by using an optical flow prediction method of an exemplary embodiment of the present disclosure.

[0062] The optical flow prediction method using the exemplary embodiment of the present disclosure can accurately and robustly predict the sky optical flow from a single image. Figure 4 As shown in the figure, the first row is a single sky image (including images of the sky area), and the second row is a visualization of the sky optical flow obtained. Different colors represent different directions of the optical flow, and the depth of the color represents the magnitude of the optical flow. A blank indicates that there is no dynamic change in the image area (i.e., no optical flow). The ray direction of each grid cell is the specific direction of the optical flow. Figure 4 As can be seen from the example, the sky optical flow obtained by the optical flow prediction method of the exemplary embodiment of the present disclosure is more consistent with the sky scene in the image, which can capture the motion pattern of the entire sky area and well suppress the generation of dynamic textures in the non-sky area. In addition, the obtained sky optical flow has strong diversity in the local part of the image. Therefore, if a sky animation is generated based on such a sky optical flow, it can be guaranteed that the visual effect of the generated sky animation is more realistic and has fewer image quality problems.

[0063] Figure 5 is a block diagram illustrating an optical flow prediction apparatus according to an exemplary embodiment of the present disclosure.

[0064] Reference Figure 5 , the optical flow prediction device 500 may include an image acquisition unit 510, an image optical flow prediction unit 520, a sky area determination unit 530 and a sky optical flow acquisition unit 540. Specifically, the image acquisition unit 510 may be configured to acquire an image including a sky area. The image optical flow prediction unit 520 may be configured to acquire texture features of the image and predict the optical flow in the image based on the acquired texture features of the image. The sky area determination unit 530 may be configured to determine the sky area in the image by segmenting the image. The sky optical flow acquisition unit 540 may be configured to obtain the optical flow of the sky area based on the predicted optical flow and the determined sky area.

[0065] In addition, despite Figure 5 Although not shown in the figure, the optical flow prediction device 500 may optionally further include a sky dynamic image generation unit. The sky dynamic image generation unit may be configured to generate a sky dynamic image according to the obtained optical flow of the sky area.

[0066] because Figure 2 The optical flow prediction method shown can be obtained by Figure 5 The optical flow prediction device 500 shown in the figure is used for execution, and the image acquisition unit 510, the image optical flow prediction unit 520, the sky area determination unit 530 and the sky optical flow acquisition unit 540 can respectively execute the same Figure 2The operations corresponding to step S210, step S220, step S230 and step S240 in FIG. Figure 5 Any details on the operations performed by the various units in Figure 2 The corresponding description will not be repeated here.

[0067] In addition, it should be noted that, although the optical flow prediction device 500 is divided into units for performing corresponding processing respectively when it is introduced above, it is clear to those skilled in the art that the processing performed by the above-mentioned units can also be performed when the optical flow prediction device 500 does not perform any specific unit division or there is no clear boundary between the units. In addition, the optical flow prediction device 500 may also include other units, such as a storage unit, etc.

[0068] Figure 6 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

[0069] Reference Figure 6 The electronic device 600 may include at least one memory 601 and at least one processor 602, wherein the at least one memory stores computer executable instructions, and when the computer executable instructions are executed by the at least one processor, the at least one processor 602 executes the optical flow prediction method according to the embodiment of the present disclosure.

[0070] As an example, the electronic device may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instruction set. Here, the electronic device is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instruction (or instruction set) individually or in combination. The electronic device may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0071] In electronic devices, the processor may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0072] The processor can execute instructions or codes stored in the memory, wherein the memory can also store data. Instructions and data can also be sent and received through the network via the network interface device, wherein the network interface device can adopt any known transmission protocol.

[0073] The memory may be integrated with the processor, for example, RAM or flash memory is arranged within an integrated circuit microprocessor or the like. In addition, the memory may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory and the processor may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor can read files stored in the memory.

[0074] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0075] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the optical flow prediction method according to the exemplary embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The instructions or computer programs in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0076] According to an embodiment of the present disclosure, a computer program product may also be provided. The computer program includes computer instructions. When the computer instructions are executed by a processor, the optical flow prediction method according to the exemplary embodiment of the present disclosure is implemented.

[0077] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are defined by the claims.

Claims

1. An optical flow prediction method, comprising: Acquire an image including a sky region; Acquire texture features of the image, and predict optical flow in the image based on the acquired texture features of the image; Determine a sky area in the image by segmenting the image; Obtaining the optical flow of the sky area based on the predicted optical flow and the determined sky area, Wherein, obtaining the optical flow of the sky area based on the predicted optical flow and the determined sky area includes: By comparing the predicted optical flow with the determined sky area, filtering out the optical flow of the non-sky area; It is determined whether the remaining optical flow after filtering out the optical flow of the non-sky area covers the sky area, and the optical flow of the sky area is obtained based on the remaining optical flow according to the determined situation.

2. The optical flow prediction method according to claim 1, wherein: The determining whether the remaining optical flow after filtering out the optical flow of the non-sky area covers the sky area, and obtaining the optical flow of the sky area based on the remaining optical flow according to the determined situation, includes: Determine whether the remaining optical flow after filtering out the optical flow of the non-sky area covers more than a predetermined proportion of the sky area; If the coverage exceeds a predetermined proportion of the sky area, determining the remaining optical flow as the optical flow of the sky area; If the coverage does not exceed a predetermined proportion of the sky area, determining the completeness of the remaining optical flow according to the area of ​​the sky area covered by the remaining optical flow; If the completeness exceeds a preset threshold, obtaining the optical flow of the sky area through optical flow propagation based on the remaining optical flow; If the completeness does not exceed the preset threshold, the optical flow of the sky area is obtained by using a preset optical flow template and the determined sky area.

3. The optical flow prediction method according to claim 2, wherein: The method of obtaining the optical flow of the sky area by using a preset optical flow template and a determined sky area includes: Based on the determined features of the sky area, an optical flow template with the highest matching degree with the sky area is selected from a plurality of preset optical flow templates, and the optical flow of the sky area is obtained according to the selected optical flow template.

4. The optical flow prediction method according to claim 3, wherein: The step of obtaining the optical flow of the sky area according to the selected optical flow template comprises: The optical flow of the sky area is obtained by performing random optical flow perturbation on the selected optical flow template.

5. The optical flow prediction method according to claim 3, wherein: The characteristic of the sky region includes at least one of a position, a shape and a size of the sky region in the image.

6. The optical flow prediction method according to claim 1, wherein: The predicting the optical flow in the image based on the acquired texture feature of the image comprises: Based on the texture features, a pre-trained deep generative adversarial network model is used to predict the optical flow in the image, wherein the deep generative adversarial network model is based on the sky time-lapse video and is trained by learning the mapping relationship between the sky texture in the sky time-lapse video and the sky optical flow.

7. The optical flow prediction method according to claim 1, wherein: The step of determining a sky area in the image by segmenting the image comprises: Based on the image, a pre-trained sky segmentation model is used to distinguish between a sky area and a non-sky area in the image, wherein the sky segmentation model is trained based on sky segmentation annotation data in a sky time-lapse video.

8. An optical flow prediction device, comprising: An image acquisition unit configured to acquire an image including a sky area; an image optical flow prediction unit, configured to obtain texture features of the image, and predict the optical flow in the image based on the obtained texture features of the image; A sky area determination unit, configured to determine a sky area in the image by segmenting the image; A sky optical flow obtaining unit is configured to obtain the optical flow of the sky area based on the predicted optical flow and the determined sky area, Wherein, obtaining the optical flow of the sky area based on the predicted optical flow and the determined sky area includes: By comparing the predicted optical flow with the determined sky area, filtering out the optical flow of the non-sky area; It is determined whether the remaining optical flow after filtering out the optical flow of the non-sky area covers the sky area, and the optical flow of the sky area is obtained based on the remaining optical flow according to the determined situation.

9. The optical flow prediction device according to claim 8, wherein: The determining whether the remaining optical flow after filtering out the optical flow of the non-sky area covers the sky area, and obtaining the optical flow of the sky area based on the remaining optical flow according to the determined situation, includes: Determine whether the remaining optical flow after filtering out the optical flow of the non-sky area covers more than a predetermined proportion of the sky area; If the coverage exceeds a predetermined proportion of the sky area, determining the remaining optical flow as the optical flow of the sky area; If the coverage does not exceed a predetermined proportion of the sky area, determining the completeness of the remaining optical flow according to the area of ​​the sky area covered by the remaining optical flow; If the completeness exceeds a preset threshold, obtaining the optical flow of the sky area through optical flow propagation based on the remaining optical flow; If the completeness does not exceed the preset threshold, the optical flow of the sky area is obtained by using a preset optical flow template and the determined sky area.

10. The optical flow prediction device according to claim 9, wherein: The method of obtaining the optical flow of the sky area by using a preset optical flow template and a determined sky area includes: Based on the determined features of the sky area, an optical flow template with the highest matching degree with the sky area is selected from a plurality of preset optical flow templates, and the optical flow of the sky area is obtained according to the selected optical flow template.

11. The optical flow prediction device according to claim 10, wherein: The step of obtaining the optical flow of the sky area according to the selected optical flow template comprises: The optical flow of the sky area is obtained by performing random optical flow perturbation on the selected optical flow template.

12. The optical flow prediction device according to claim 10, wherein: The feature of the sky region includes at least one of a position, a shape and a size of the sky region in the image.

13. The optical flow prediction device according to claim 8, wherein: The predicting the optical flow in the image based on the acquired texture feature of the image comprises: Based on the texture features, a pre-trained deep generative adversarial network model is used to predict the optical flow in the image, wherein the deep generative adversarial network model is based on the sky time-lapse video and is trained by learning the mapping relationship between the sky texture in the sky time-lapse video and the sky optical flow.

14. The optical flow prediction device according to claim 8, wherein: The step of determining a sky area in the image by segmenting the image comprises: Based on the image, a pre-trained sky segmentation model is used to distinguish between a sky area and a non-sky area in the image, wherein the sky segmentation model is trained based on sky segmentation annotation data in a sky time-lapse video.

15. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to perform the optical flow prediction method according to any one of claims 1 to 7.

16. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one processor, the at least one processor is prompted to perform the optical flow prediction method according to any one of claims 1 to 7.

17. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the optical flow prediction method according to any one of claims 1 to 7 is implemented.

Images