Image processing method and device

Abstract

The present disclosure relates to an image processing method and device. The image processing method comprises: obtaining an average sky segmentation map of all image frames of a sky time-lapse video, wherein the sky time-lapse video is a video obtained by time-lapse photography of the sky; obtaining an average optical flow field of all image frames of the sky time-lapse video; generating a confidence map of the sky time-lapse video based on the average sky segmentation map and the average optical flow field, wherein the confidence map is a map indicating the confidence that a pixel of the sky time-lapse video is a sky pixel; generating a sky segmentation result of the sky time-lapse video based on the sky time-lapse video, the average sky segmentation map and the confidence map. According to the image processing method and device of the present disclosure, the accuracy of sky segmentation can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and more specifically, to an image processing method and device. Background Art

[0002] Sky segmentation, also known as sky cutout, is to classify each pixel in the image one by one to determine whether it belongs to the sky, and then identify the sky area in the image. Sky segmentation is an important sub-problem in image semantic segmentation and has a wide range of applications in scenarios such as image editing and content creation. For example, by identifying the sky in an image, the sky image area can be denoised, dark light enhanced, and automatically white balanced. The original sky background can also be replaced with a sky image taken by a professional photographer to enhance the beauty of the picture. In addition, the results of sky segmentation can also be used to create a refreshing short video of the sky.

[0003] In recent years, with the development of artificial intelligence technology, sky segmentation algorithms based on deep learning have achieved good results. However, training deep learning models often requires a large amount of labeled data; the more training data, the more accurate the labeling, and the higher the diversity of sky images, the better the trained segmentation model. However, unlike the category of labeled images, training sky segmentation algorithms requires pixel-level labeling. At present, manual labeling based on crowdsourcing is the mainstream solution in this field. It is time-consuming and labor-intensive, and there is a certain degree of subjective bias in its labeling. In addition, manual labeling tools mostly use polygon-based labeling methods, so for the edges of the sky with rich details, the labeling granularity is often not high; another problem with most sky segmentation training data is that the diversity of the sky images they contain is low, which is manifested as a single sky texture and lighting conditions. All these problems with labeled data have been plaguing the application of sky segmentation models in actual scenes. Summary of the invention

[0004] An exemplary embodiment of the present disclosure is to provide an image processing method and apparatus to at least solve the image processing problems in the related art, but may not solve any of the above problems.

[0005] According to an exemplary embodiment of the present disclosure, there is provided an image processing method, comprising: obtaining an average sky segmentation map of all image frames of a sky time-lapse video, wherein the sky time-lapse video is a video obtained by time-lapse photography of the sky; obtaining an average optical flow field of all image frames of the sky time-lapse video; generating a confidence map of the sky time-lapse video based on the average sky segmentation map and the average optical flow field, wherein the confidence map is a map representing the confidence that pixels of the sky time-lapse video are sky pixels; generating a sky segmentation result of the sky time-lapse video based on the sky time-lapse video, the average sky segmentation map and the confidence map.

[0006] Optionally, the step of obtaining the average sky segmentation map of all image frames of the sky time-lapse video may include: performing sky segmentation on each image frame of the sky time-lapse video through a sky segmentation model to obtain a sky segmentation result for each image frame of the sky time-lapse video; performing an averaging operation on the sky segmentation results of all image frames of the sky time-lapse video, and using the averaging operation result as the average segmentation map of all image frames of the sky time-lapse video.

[0007] Optionally, the step of acquiring the average optical flow field of all image frames of the sky time-lapse video may include: acquiring the average optical flow field of all image frames of the sky time-lapse video through an optical flow extraction model.

[0008] Optionally, before generating the confidence map of the sky time-lapse video based on the average sky segmentation map and the average optical flow field, the image processing method may further include: adjusting the average sky segmentation map based on the average optical flow field, and using the adjusted average sky segmentation map as the average sky segmentation map.

[0009] Optionally, the step of generating the confidence map of the sky time-lapse video based on the average sky segmentation map and the average optical flow field may include: determining the confidence of pixels in the sky time-lapse video based on the average sky segmentation map; determining the optical flow amplitude of pixels in the sky time-lapse video based on the average optical flow field; and generating the confidence map of the sky time-lapse video based on the confidence and optical flow amplitude of pixels in the sky time-lapse video.

[0010] Optionally, the step of generating the confidence map of the sky time-lapse video based on the confidence and optical flow amplitude of the pixels in the sky time-lapse video may include: classifying the pixels in the sky time-lapse video into multiple preset categories based on the confidence and optical flow amplitude of the pixels in the sky time-lapse video; assigning category confidence to each of the multiple categories; generating a multivariate map of the sky time-lapse video based on the confidence, category and category confidence of each pixel in the sky time-lapse video, and using the multivariate map as the confidence map of the sky time-lapse video.

[0011] Optionally, the step of classifying pixels in the sky time-lapse video into multiple preset categories based on the confidence and optical flow amplitude of the pixels in the sky time-lapse video may include: classifying pixels in the sky time-lapse video with a confidence higher than a first threshold as first-category pixels; classifying pixels in the sky time-lapse video with a confidence lower than or equal to the first threshold and an optical flow amplitude higher than a second threshold as second-category pixels; and classifying pixels in the sky time-lapse video other than first-category pixels and second-category pixels as third-category pixels.

[0012] Optionally, the step of generating a sky segmentation result of the sky time-lapse video based on the sky time-lapse video, the average sky segmentation map and the confidence map may include: based on the sky time-lapse video, the average sky segmentation map and the confidence map, using a preset weight-directed graph filtering optimization algorithm to obtain the sky segmentation result of the sky time-lapse video.

[0013] Optionally, the step of adjusting the average sky segmentation map based on the average optical flow field may include: selecting pixels whose optical flow amplitude is higher than a third threshold; determining whether the selected pixels are segmented as sky pixels in the average sky segmentation map; when the selected pixels are not segmented as sky pixels in the average sky segmentation map, adjusting the selected pixels to sky pixels in the average sky segmentation map.

[0014] Optionally, the generated multivariate graph of the sky time-lapse video is a ternary graph.

[0015] According to an exemplary embodiment of the present disclosure, there is provided an image processing apparatus, comprising: an initial segmentation unit, configured to obtain an average sky segmentation map of all image frames of a sky time-lapse video, wherein the sky time-lapse video is a video obtained by time-lapse photography of the sky; an optical flow acquisition unit, configured to obtain an average optical flow field of all image frames of the sky time-lapse video; a confidence map generation unit, configured to generate a confidence map of the sky time-lapse video based on the average sky segmentation map and the average optical flow field, wherein the confidence map is a map representing the confidence that a pixel of the sky time-lapse video is a sky pixel; and a segmentation optimization unit, configured to generate a sky segmentation result of the sky time-lapse video based on the sky time-lapse video, the average sky segmentation map and the confidence map.

[0016] Optionally, the initial segmentation unit can be configured to: perform sky segmentation on each image frame of the sky time-lapse video through a sky segmentation model to obtain a sky segmentation result for each image frame of the sky time-lapse video; perform an averaging operation on the sky segmentation results of all image frames of the sky time-lapse video, and use the averaging operation result as an average segmentation map of all image frames of the sky time-lapse video.

[0017] Optionally, the optical flow acquisition unit may be configured to: acquire the average optical flow field of all image frames of the sky time-lapse video through an optical flow extraction model.

[0018] Optionally, the image processing device may further include: a segmentation map adjustment unit configured to adjust the average sky segmentation map based on the average optical flow field, and use the adjusted average sky segmentation map as the average sky segmentation map.

[0019] Optionally, the confidence map generation unit can be configured to: determine the confidence of pixels in the sky time-lapse video based on the average sky segmentation map; determine the optical flow amplitude of pixels in the sky time-lapse video based on the average optical flow field; and generate the confidence map of the sky time-lapse video based on the confidence and optical flow amplitude of pixels in the sky time-lapse video.

[0020] Optionally, the confidence map generation unit can be configured to: classify the pixels in the sky time-lapse video into multiple preset categories based on the confidence and optical flow amplitude of the pixels in the sky time-lapse video; assign category confidence to each of the multiple categories; generate a multivariate map of the sky time-lapse video based on the confidence, category and category confidence of each pixel in the sky time-lapse video, and use the multivariate map as the confidence map of the sky time-lapse video.

[0021] Optionally, the confidence map generation unit can be configured to: classify pixels in the sky time-lapse video whose confidence is higher than a first threshold as first-category pixels; classify pixels in the sky time-lapse video whose confidence is lower than or equal to the first threshold and whose optical flow amplitude is higher than a second threshold as second-category pixels; and classify pixels in the sky time-lapse video other than first-category pixels and second-category pixels as third-category pixels.

[0022] Optionally, the segmentation optimization unit may be configured to obtain a sky segmentation result of the sky time-lapse video based on the sky time-lapse video, the average sky segmentation map and the confidence map using a preset weighted-guided graph filtering optimization algorithm.

[0023] Optionally, the segmentation optimization unit can be configured to: select pixels whose optical flow amplitude is higher than a third threshold; determine whether the selected pixels are segmented as sky pixels in the average sky segmentation map; when the selected pixels are not segmented as sky pixels in the average sky segmentation map, adjust the selected pixels to sky pixels in the average sky segmentation map.

[0024] Optionally, the generated multivariate graph of the sky time-lapse video is a ternary graph.

[0025] According to an exemplary embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement an image processing method according to an exemplary embodiment of the present disclosure.

[0026] According to an exemplary embodiment of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, enables the electronic device to perform an image processing method according to an exemplary embodiment of the present disclosure.

[0027] According to an exemplary embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction. When the computer program / instruction is executed by a processor, the image processing method according to the exemplary embodiment of the present disclosure is implemented.

[0028] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0029] It can automatically extract accurate pixel-level sky annotations from massive video data, thus eliminating time-consuming and labor-intensive manual work.

[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0032] Figure 1 An exemplary system architecture 100 is shown in which exemplary embodiments of the present disclosure may be applied.

[0033] Figure 2 A flowchart of an image processing method according to an exemplary embodiment of the present disclosure is shown.

[0034] Figure 3 An example of automatically extracting sky segmentation annotations using an image processing method according to an exemplary embodiment of the present disclosure is shown.

[0035] Figure 4 A flowchart of an image processing method according to another exemplary embodiment of the present disclosure is shown.

[0036] Figure 5 A block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure is shown.

[0037] Figure 6 A block diagram of an image processing apparatus according to another exemplary embodiment of the present disclosure is shown.

[0038] Figure 7 A block diagram of an electronic device 700 according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0039] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0041] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three types of parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three types of parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0042] The current sky area annotation acquisition scheme can be achieved by combining manual annotation with post-processing algorithms. In the combination of manual annotation and post-processing algorithms, first, multiple annotators will use polygonal annotation tools to identify relatively large sky areas from the image, thereby providing a coarse-grained, initial sky area annotation with imprecise edges. Such manual annotation often pays more attention to the annotation precision of sky pixels, and is prone to miss some sky areas with complex edge shapes, especially sky pixels between tree branches. Then, by using sky pixel density estimation and weighted guided filter, the post-processing algorithm can further optimize the manual annotation to obtain better annotation data for the sky and tree segmentation edges. It can be seen that the above process relies on manual participation to obtain the initial sky area annotation, and each image requires multiple people to participate in the annotation, making it difficult to obtain a large amount of annotation data in a short time. In addition, the post-processing algorithm has high requirements for the quality of the initial annotation: when the granularity of the manual annotation is poor and the precision of the sky pixels is low, the post-processing algorithm will be extremely sensitive to the optimization parameters, and the annotation quality of a considerable portion of the sky images optimized by the post-processing algorithm is even worse than the initial manual annotation.

[0043] In addition, since sky time-lapse videos contain more diverse sky images, using sky time-lapse video data to evaluate sky segmentation algorithms can more accurately measure the robustness of sky segmentation algorithms in real scenarios. However, the deep learning model trained on mainstream sky segmentation data cannot cope well with sky images under different lighting, different sky textures, and different climate conditions, and is prone to large segmentation errors. In addition, no automatic sky area labeling solution has been proposed, and manual methods are still used to obtain the labels for each sky time-lapse video.

[0044] The present disclosure is to propose a technology that can automatically extract sky area annotations from sky time-lapse videos. Figures 1 to 7 An image processing method and apparatus according to exemplary embodiments of the present disclosure are described in detail.

[0045] Figure 1 An exemplary system architecture 100 is shown in which exemplary embodiments of the present disclosure may be applied.

[0046] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. A user may use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages (e.g., video data upload request, video data download request), etc. Various image applications or video applications may be installed on the terminal devices 101, 102, 103. The terminal devices 101, 102, 103 may be hardware or software. When the terminal devices 101, 102, 103 are hardware, they may be various electronic devices having a display screen and capable of image display, video playback, recording, editing, etc., including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, etc. When the terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above, and can be implemented as multiple software or software modules (for example, to provide distributed services), or as a single software or software module. No specific limitation is made here.

[0047] The terminal devices 101, 102, and 103 may be equipped with image acquisition devices (e.g., cameras) to collect video data. In practice, the smallest visual unit that constitutes a video is a frame. Each frame is a static image. Synthesizing a temporally continuous sequence of frames together forms a dynamic video. In addition, the terminal devices 101, 102, and 103 may also be equipped with components for converting electrical signals into sounds (e.g., speakers) to play sounds, and may also be equipped with devices for converting analog audio signals into digital audio signals (e.g., microphones) to collect sounds.

[0048] The server 105 may be a server that provides various services, such as a background server that provides support for image applications or video applications installed on the terminal devices 101, 102, and 103. The background server may parse, store, and process received image data or video data upload requests and other data, and may also receive image data or video data download requests sent by the terminal devices 101, 102, and 103, and feed back the image data or video data indicated by the image data or video data download request to the terminal devices 101, 102, and 103.

[0049] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0050] It should be noted that the image processing method provided in the embodiments of the present disclosure is usually executed by a terminal device, but can also be executed by a server, or can also be executed by a terminal device and a server in collaboration. Accordingly, the image processing apparatus can be set in the terminal device, in the server, or in both the terminal device and the server.

[0051] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers, and the present disclosure has no limitation on this.

[0052] Figure 2 A flowchart of an image processing method according to an exemplary embodiment of the present disclosure is shown.

[0053] Reference Figure 2 In step S201, an average sky segmentation map of all image frames of the sky time-lapse video is obtained. Here, the sky time-lapse video is a video obtained by time-lapse photography of the sky.

[0054] In an exemplary embodiment of the present disclosure, when obtaining the average sky segmentation map of all image frames of the sky time-lapse video, sky segmentation can be first performed on each image frame of the sky time-lapse video through a sky segmentation model to obtain a sky segmentation result for each image frame of the sky time-lapse video, and then an averaging operation is performed on the sky segmentation results of all image frames of the sky time-lapse video, and the averaging operation result is used as the average segmentation map of all image frames of the sky time-lapse video.

[0055] When processing a time-lapse video of the sky by an image processing method according to an exemplary embodiment of the present disclosure, the sky segmentation model may be first used to obtain a sky segmentation result for each image frame. Since the corresponding sky area in a time-lapse video remains roughly unchanged, the average segmentation map of all video frames is used as the initial sky segmentation result to offset the large segmentation errors generated by the sky segmentation model on some frames. However, due to the limited data for training the sky segmentation model, the initial sky segmentation result still has many errors.

[0056] Figure 3An example of automatically extracting sky segmentation annotations using an image processing method according to an exemplary embodiment of the present disclosure is shown. Figure 3 middle, Figure 3 (a) is the image frame, Figure 3 (b) is the average sky segmentation map, that is, the initial sky segmentation result, Figure 3 (c) is the optical flow visualization diagram, Figure 3 (d) is the final sky segmentation annotation. Figure 3 In (c), the color depth indicates the amplitude of the optical flow, and the blank space indicates that there is no dynamic change in the image area.

[0057] like Figure 3 As shown in (b), under certain sky texture and lighting conditions, the sky segmentation model can easily identify some sky pixels as non-sky pixels. If such results are used as annotations to train the sky segmentation model, the effect of the sky segmentation model in real scenes will be affected. Therefore, other information in the video is needed to assist in optimizing the initial sky segmentation results. Considering that the sky area in the sky time-lapse video has significant dynamic changes, the optical flow of the video is used here.

[0058] In step S202, the average optical flow field of all image frames of the sky time-lapse video is obtained.

[0059] In an exemplary embodiment of the present disclosure, when acquiring the average optical flow field of all image frames of the sky time-lapse video, the average optical flow field of all image frames of the sky time-lapse video may be acquired through an optical flow extraction model.

[0060] When processing a time-lapse video of the sky by an image processing method according to an exemplary embodiment of the present disclosure, an optical flow extraction model can also be used to obtain an average optical flow field in the time-lapse video of the sky. In a time-lapse video of the sky, there is often no image displacement in the static foreground, while the texture of the sky area presents rapid dynamic changes. Therefore, using optical flow as auxiliary information, sky pixels that cannot be recognized by the sky segmentation model can be identified to achieve the purpose of optimizing the initial segmentation results. However, since the video optical flow also has a certain degree of noise and error, a simple fusion algorithm cannot obtain the best optimization result.

[0061] In the exemplary embodiments of the present disclosure, the execution order of step S201 and step S202 is not limited. That is, step S202 may be executed first, and then step S201, or step S201 and step S202 may be executed simultaneously.

[0062] In step S203, a confidence map of the sky time-lapse video is generated based on the average sky segmentation map and the average optical flow field. Here, the confidence map is a map indicating the confidence that the pixels of the sky time-lapse video are sky pixels.

[0063] In an exemplary embodiment of the present disclosure, when generating the confidence map of the sky time-lapse video based on the average sky segmentation map and the average optical flow field, the confidence of the pixels in the sky time-lapse video can be first determined based on the average sky segmentation map, and the optical flow amplitude of the pixels in the sky time-lapse video can be determined based on the average optical flow field, and then the confidence map of the sky time-lapse video can be generated based on the confidence and optical flow amplitude of the pixels in the sky time-lapse video.

[0064] In an exemplary embodiment of the present disclosure, when generating the confidence map of the sky time-lapse video based on the confidence and optical flow amplitude of the pixels in the sky time-lapse video, the pixels in the sky time-lapse video can be first classified into multiple preset categories based on the confidence and optical flow amplitude of the pixels in the sky time-lapse video, and a category confidence is assigned to each of the multiple categories. Then, based on the confidence, category and category confidence of each pixel in the sky time-lapse video, a multivariate map of the sky time-lapse video is generated, and the multivariate map is used as the confidence map of the sky time-lapse video.

[0065] In an exemplary embodiment of the present disclosure, when the pixels in the sky time-lapse video are classified into a plurality of preset categories based on the confidence and optical flow amplitude of the pixels in the sky time-lapse video, the pixels in the sky time-lapse video with a confidence higher than a first threshold value can be classified as first-category pixels; the pixels in the sky time-lapse video with a confidence lower than or equal to the first threshold value and an optical flow amplitude higher than a second threshold value can be classified as second-category pixels; the pixels in the sky time-lapse video other than the first-category pixels and the second-category pixels can be classified as third-category pixels. In this case, the multivariate graph of the sky time-lapse video generated is a ternary graph.

[0066] As an example, after obtaining the average sky segmentation map and the average optical flow field in the sky time-lapse video in step S201 and step S202, a trimap can be generated based on the average sky segmentation map and the average optical flow field. The specific method is to divide all pixels in the image into three categories according to the standard, and assign different confidence levels to each category: i) pixels with higher confidence in the initial sky segmentation result; ii) pixels with higher confidence in the initial sky segmentation result but larger optical flow amplitude; iii) other pixels.

[0067] In step S204, a sky segmentation result of the sky time-lapse video is generated based on the sky time-lapse video, the average sky segmentation map and the confidence map.

[0068] In an exemplary embodiment of the present disclosure, when generating the sky segmentation result of the sky time-lapse video based on the sky time-lapse video, the average sky segmentation map and the confidence map, the sky segmentation result of the sky time-lapse video can be obtained based on the sky time-lapse video, the average sky segmentation map and the confidence map using a preset weight-directed graph filtering optimization algorithm.

[0069] In an exemplary embodiment of the present disclosure, when adjusting the average sky segmentation map based on the average optical flow field, pixels whose optical flow amplitudes are higher than a third threshold value may be selected first, and it may be determined whether the selected pixels are segmented as sky pixels in the average sky segmentation map. Then, when the selected pixels are not segmented as sky pixels in the average sky segmentation map, the selected pixels may be adjusted to sky pixels in the average sky segmentation map.

[0070] Since the weighted graph filter optimization algorithm can ensure the smoothness of the segmentation results in the sky area while retaining the fineness and details of the sky contour at the edge, the weighted graph filter optimization algorithm is used to solve the information fusion problem of the video optical flow and the initial sky segmentation results, thereby correcting the segmentation errors in the initial results. The weighted graph filter optimization algorithm can further optimize and judge the category based on the similarity of the surrounding pixels. When the surrounding similar pixels give different judgments, the confidence map can guide the weighted graph filter optimization algorithm to believe in the pixels with high confidence more. By using the confidence map, the pixel category with high confidence can be transmitted to the pixel with low confidence through similarity, thereby optimizing the segmentation results. The sky annotation obtained in this way is smooth inside the contour, and the contour is more accurate.

[0071] The weighted graph filter optimization algorithm requires three inputs: 1) a sky image (e.g., the time-lapse video of the sky); 2) an initial sky segmentation result (e.g., the average sky segmentation map); and 3) a confidence map, where each value on the confidence map represents the probability that the pixel is the sky.

[0072] As an example, the weight-directed graph filtering optimization algorithm is divided into the following steps:

[0073] Step 1: Get the confidence map. The specific method is to divide all pixels in the image (sky time-lapse video) into three categories according to the following criteria and assign them different confidence levels: i) pixels with higher confidence in the initial sky segmentation result; ii) pixels with larger optical flow amplitude; iii) other pixels. Use the generated trimap as the confidence map.

[0074] Step 2: Input the sky image, the reset segmentation result, and the confidence map into the weighted graph filtering optimization algorithm, and output the segmentation result optimized by the algorithm.

[0075] like Figure 3 As shown in (b), when the sky area in the sky time-lapse video presents unusual texture colors or lighting conditions, the initial sky segmentation often has partial errors. Even with the current best optical flow extraction model, it is difficult to obtain accurate sky edges. In the present disclosure, by using video optical flow to assist in optimizing the initial sky segmentation results, high-precision sky area annotations can be automatically extracted from the time-lapse video, thereby achieving the automatic extraction of massive, highly diverse, and accurately labeled sky area annotation data from the sky time-lapse video without any human intervention, which is used to train the sky segmentation model and improve the robustness of the sky segmentation algorithm in actual scenes.

[0076] Figure 4 A flowchart of an image processing method according to another exemplary embodiment of the present disclosure is shown.

[0077] Reference Figure 4 In step S401, an average sky segmentation map of all image frames of the sky time-lapse video is obtained. Here, the sky time-lapse video is a video obtained by time-lapse photography of the sky.

[0078] In step S402, the average optical flow field of all image frames of the sky time-lapse video is obtained.

[0079] In step S403, the average sky segmentation map is adjusted based on the average optical flow field, and the adjusted average sky segmentation map is used as the average sky segmentation map.

[0080] If an image pixel has a large optical flow but is considered a non-sky pixel in the initial segmentation result, there is a high probability that the initial sky segmentation result is wrong. Therefore, the pixel can be reset to a sky pixel in the segmentation result.

[0081] In step S404, a confidence map of the sky time-lapse video is generated based on the average sky segmentation map and the average optical flow field. Here, the confidence map is a map indicating the confidence that the pixels of the sky time-lapse video are sky pixels.

[0082] In step S405, a sky segmentation result of the sky time-lapse video is generated based on the sky time-lapse video, the average sky segmentation map and the confidence map.

[0083] The weighted graph filter optimization algorithm requires three inputs: 1) a sky image (e.g., the time-lapse video of the sky); 2) an initial sky segmentation result (e.g., the average sky segmentation map); and 3) a confidence map, where each value on the confidence map represents the probability that the pixel is the sky.

[0084] As an example, the weight-directed graph filtering optimization algorithm is divided into the following steps:

[0085] Step 1: Adjust the initial segmentation result according to the optical flow. If a pixel has a large optical flow but is considered a non-sky pixel in the initial segmentation result, there is a high probability that the initial sky segmentation result is wrong. Therefore, the pixel is reset to a sky pixel in the segmentation result.

[0086] Step 2: Get the confidence map. The specific method is to divide all pixels in the image (sky time-lapse video) into three categories according to the following criteria and assign them different confidence levels: i) pixels with higher confidence in the initial sky segmentation result; ii) pixels with larger optical flow amplitude; iii) other pixels. Use the generated trimap as the confidence map.

[0087] Step 3: Input the sky image, the reset segmentation result, and the confidence map into the weighted-guided graph filtering optimization algorithm, and output the segmentation result optimized by the algorithm.

[0088] like Figure 3 As shown in (b), when the sky area in the sky time-lapse video presents unusual texture colors or lighting conditions, the initial sky segmentation often has partial errors. Even with the current best optical flow extraction model, it is difficult to obtain accurate sky edges. In the present disclosure, by using video optical flow to assist in optimizing the initial sky segmentation results, high-precision sky area annotations can be automatically extracted from the time-lapse video, thereby achieving the automatic extraction of massive, highly diverse, and accurately labeled sky area annotation data from the sky time-lapse video without any human intervention, which is used to train the sky segmentation model and improve the robustness of the sky segmentation algorithm in actual scenes.

[0089] about Figure 4 The method in which the specific methods of step S401, step S402, step S404 and step S405 have been respectively described in the relevant Figure 2 The steps S201, S202, S203 and S204 are described in detail in the above description and will not be elaborated in detail here.

[0090] The above has been combined Figures 1 to 4 The image processing method according to the exemplary embodiment of the present disclosure is described. Figure 5 and Figure 6 An image processing apparatus and its units according to an exemplary embodiment of the present disclosure are described.

[0091] Figure 5 A block diagram of an image processing apparatus according to an exemplary embodiment of the present disclosure is shown.

[0092] Reference Figure 5 The image processing device includes an initial segmentation unit 51, an optical flow acquisition unit 52, a confidence map generation unit 53 and a segmentation optimization unit 54.

[0093] The initial segmentation unit 51 is configured to obtain an average sky segmentation map of all image frames of the sky time-lapse video. Here, the sky time-lapse video is a video obtained by time-lapse photography of the sky.

[0094] In an exemplary embodiment of the present disclosure, the initial segmentation unit 51 can be configured to: perform sky segmentation on each image frame of the sky time-lapse video through a sky segmentation model to obtain a sky segmentation result for each image frame of the sky time-lapse video; perform an averaging operation on the sky segmentation results of all image frames of the sky time-lapse video, and use the averaging operation result as an average segmentation map of all image frames of the sky time-lapse video.

[0095] The optical flow acquisition unit 52 is configured to acquire an average optical flow field of all image frames of the sky time-lapse video.

[0096] In an exemplary embodiment of the present disclosure, the optical flow acquisition unit 52 may be configured to: acquire the average optical flow field of all image frames of the sky time-lapse video through an optical flow extraction model.

[0097] The confidence map generating unit 53 is configured to generate a confidence map of the sky time-lapse video based on the average sky segmentation map and the average optical flow field. Here, the confidence map is a map indicating the confidence that the pixels of the sky time-lapse video are sky pixels.

[0098] In an exemplary embodiment of the present disclosure, the confidence map generation unit 53 may be configured to: determine the confidence of pixels in the sky time-lapse video based on the average sky segmentation map; determine the optical flow amplitude of pixels in the sky time-lapse video based on the average optical flow field; and generate the confidence map of the sky time-lapse video based on the confidence and optical flow amplitude of pixels in the sky time-lapse video.

[0099] In an exemplary embodiment of the present disclosure, the confidence map generation unit 53 may be configured to: classify the pixels in the sky time-lapse video into a plurality of preset categories based on the confidence and optical flow amplitude of the pixels in the sky time-lapse video; assign category confidence to each of the plurality of categories; generate a multivariate map of the sky time-lapse video based on the confidence, category and category confidence of each pixel in the sky time-lapse video, and use the multivariate map as the confidence map of the sky time-lapse video.

[0100] In an exemplary embodiment of the present disclosure, the generated multivariate map of the sky time-lapse video is a ternary map.

[0101] In an exemplary embodiment of the present disclosure, the confidence map generation unit 53 may be configured to: classify pixels in the sky time-lapse video whose confidence is higher than a first threshold as first-category pixels; classify pixels in the sky time-lapse video whose confidence is lower than or equal to the first threshold and whose optical flow amplitude is higher than a second threshold as second-category pixels; and classify pixels in the sky time-lapse video other than first-category pixels and second-category pixels as third-category pixels.

[0102] The segmentation optimization unit 54 is configured to generate a sky segmentation result of the sky time-lapse video based on the sky time-lapse video, the average sky segmentation map and the confidence map.

[0103] In an exemplary embodiment of the present disclosure, the segmentation optimization unit 54 may be configured to obtain a sky segmentation result of the sky time-lapse video based on the sky time-lapse video, the average sky segmentation map and the confidence map using a preset weighted-directed graph filtering optimization algorithm.

[0104] In an exemplary embodiment of the present disclosure, the segmentation optimization unit 54 may be configured to: select pixels whose optical flow amplitude is higher than a third threshold; determine whether the selected pixels are segmented as sky pixels in the average sky segmentation map; when the selected pixels are not segmented as sky pixels in the average sky segmentation map, adjust the selected pixels to sky pixels in the average sky segmentation map.

[0105] Figure 6 A block diagram of an image processing apparatus according to another exemplary embodiment of the present disclosure is shown.

[0106] Reference Figure 6 The image processing device includes an initial segmentation unit 61, an optical flow acquisition unit 62, a segmentation map adjustment unit 63, a confidence map generation unit 64 and a segmentation optimization unit 65.

[0107] The initial segmentation unit 61 is configured to obtain an average sky segmentation map of all image frames of the sky time-lapse video. Here, the sky time-lapse video is a video obtained by time-lapse photography of the sky.

[0108] The optical flow acquisition unit 62 is configured to acquire an average optical flow field of all image frames of the sky time-lapse video.

[0109] The segmentation map adjusting unit 63 is configured to adjust the average sky segmentation map based on the average optical flow field, and use the adjusted average sky segmentation map as the average sky segmentation map.

[0110] The confidence map generating unit 64 is configured to generate a confidence map of the sky time-lapse video based on the average sky segmentation map and the average optical flow field. Here, the confidence map is a map indicating the confidence that the pixels of the sky time-lapse video are sky pixels.

[0111] The segmentation optimization unit 65 is configured to generate a sky segmentation result of the sky time-lapse video based on the sky time-lapse video, the average sky segmentation map and the confidence map.

[0112] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0113] The above has been combined Figure 5 and Figure 6 The image processing apparatus according to the exemplary embodiment of the present disclosure is described. Figure 7 An electronic device according to an exemplary embodiment of the present disclosure is described.

[0114] Figure 7 A block diagram of an electronic device 700 according to an exemplary embodiment of the present disclosure is shown.

[0115] Reference Figure 7 The electronic device 700 includes at least one memory 701 and at least one processor 702, wherein the at least one memory 701 stores a set of computer executable instructions, and when the computer executable instruction set is executed by the at least one processor 702, the image processing method according to the exemplary embodiment of the present disclosure is performed.

[0116] In an exemplary embodiment of the present disclosure, the electronic device 700 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 700 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 700 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0117] In the electronic device 700, the processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0118] The processor 702 may execute instructions or codes stored in the memory 701, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0119] The memory 701 may be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 701 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 701 and the processor 702 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 702 can read files stored in the memory.

[0120] In addition, the electronic device 700 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 700 may be connected to each other via a bus and / or a network.

[0121] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium including instructions is also provided, such as a memory 701 including instructions, and the above instructions can be executed by a processor 702 of an apparatus 700 to complete the above method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0122] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided. The computer program product includes a computer program / instruction. When the computer program / instruction is executed by a processor, the image processing method according to the exemplary embodiment of the present disclosure is implemented.

[0123] The above has been referred to Figures 1 to 7 The image processing method and apparatus according to the exemplary embodiments of the present disclosure are described. However, it should be understood that: Figure 5 and Figure 6 The image processing apparatus and its units shown in the figure may be respectively configured as software, hardware, firmware or any combination of the above items to perform specific functions. Figure 7 The electronic device shown in is not limited to including the components shown above, but some components may be added or deleted as needed, and the above components may also be combined.

[0124] According to the image processing method and device disclosed in the present invention, an average sky segmentation map of all image frames of a sky time-lapse video is obtained, wherein the sky time-lapse video is a video obtained by time-lapse photography of the sky, an average optical flow field of all image frames of the sky time-lapse video is obtained, and a confidence map of the sky time-lapse video is generated based on the average sky segmentation map and the average optical flow field, wherein the confidence map is a map representing the confidence that pixels of the sky time-lapse video are sky pixels, and then a sky segmentation result of the sky time-lapse video is generated based on the sky time-lapse video, the average sky segmentation map and the confidence map, thereby improving the accuracy of sky segmentation.

[0125] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0126] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Obtaining an average sky segmentation map of all image frames of a sky time-lapse video, wherein the sky time-lapse video is a video obtained by time-lapse photography of the sky; Obtaining an average optical flow field of all image frames of the sky time-lapse video; Determining confidences of pixels in the sky time-lapse video based on the average sky segmentation map; determining an optical flow amplitude of a pixel in the sky time-lapse video based on the average optical flow field; Generating a confidence map of the sky time-lapse video based on the confidence of the pixels in the sky time-lapse video and the optical flow amplitude, wherein the confidence map is a map representing the confidence that the pixels of the sky time-lapse video are sky pixels; A sky segmentation result of the sky time-lapse video is generated based on the sky time-lapse video, the average sky segmentation map and the confidence map.

2. The image processing method according to claim 1, characterized in that: The step of obtaining the average sky segmentation map of all image frames of the sky time-lapse video comprises: Performing sky segmentation on each image frame of the sky time-lapse video using a sky segmentation model to obtain a sky segmentation result for each image frame of the sky time-lapse video; An averaging operation is performed on the sky segmentation results of all the image frames of the sky time-lapse video, and the averaging operation result is used as an average segmentation map of all the image frames of the sky time-lapse video.

3. The image processing method according to claim 1, characterized in that: The step of obtaining the average optical flow field of all image frames of the sky time-lapse video comprises: The average optical flow field of all image frames of the sky time-lapse video is obtained through an optical flow extraction model.

4. The image processing method according to claim 1, characterized in that: Before determining the confidence of the pixels in the sky time-lapse video based on the average sky segmentation map, the image processing method further includes: The average sky segmentation map is adjusted based on the average optical flow field, and the adjusted average sky segmentation map is used as the average sky segmentation map.

5. The image processing method according to claim 1, characterized in that: The step of generating the confidence map of the sky time-lapse video based on the confidence and optical flow amplitude of the pixels in the sky time-lapse video comprises: Classifying pixels in the sky time-lapse video into a plurality of preset categories based on confidence and optical flow amplitude of the pixels in the sky time-lapse video; assigning a class confidence to each of the plurality of classes; A multivariate map of the sky time-lapse video is generated based on the confidence, category, and category confidence of each pixel in the sky time-lapse video, and the multivariate map is used as the confidence map of the sky time-lapse video.

6. The image processing method according to claim 5, characterized in that: The step of classifying the pixels in the sky time-lapse video into a plurality of preset categories based on the confidence and the optical flow amplitude of the pixels in the sky time-lapse video comprises: Classifying pixels in the sky time-lapse video whose confidence level is higher than a first threshold as first-category pixels; Classifying pixels in the sky time-lapse video whose confidence level is lower than or equal to a first threshold and whose optical flow amplitude is higher than a second threshold as second-category pixels; Pixels other than the first-category pixels and the second-category pixels in the sky time-lapse video are classified into third-category pixels.

7. The image processing method according to claim 1, characterized in that: The step of generating a sky segmentation result of the sky time-lapse video based on the sky time-lapse video, the average sky segmentation map and the confidence map comprises: Based on the sky time-lapse video, the average sky segmentation map and the confidence map, a preset weight-guided graph filtering optimization algorithm is used to obtain a sky segmentation result of the sky time-lapse video.

8. The image processing method according to claim 4, characterized in that: The step of adjusting the average sky segmentation map based on the average optical flow field comprises: Selecting pixels whose optical flow magnitude is higher than a third threshold; Determining whether the selected pixel is segmented as a sky pixel in the average sky segmentation map; When the selected pixel is not segmented as a sky pixel in the average sky segmentation map, the selected pixel is adjusted to a sky pixel in the average sky segmentation map.

9. The image processing method according to claim 5, characterized in that: The generated multivariate graph of the sky time-lapse video is a ternary graph.

10. An image processing device, characterized in that: include: an initial segmentation unit configured to obtain an average sky segmentation map of all image frames of a sky time-lapse video, wherein the sky time-lapse video is a video obtained by time-lapse photography of the sky; an optical flow acquisition unit, configured to acquire an average optical flow field of all image frames of the sky time-lapse video; a confidence map generating unit, configured to determine the confidence of pixels in the sky time-lapse video based on the average sky segmentation map, determine the optical flow amplitude of the pixels in the sky time-lapse video based on the average optical flow field, and generate a confidence map of the sky time-lapse video based on the confidence and optical flow amplitude of the pixels in the sky time-lapse video, wherein the confidence map is a map indicating the confidence that the pixels of the sky time-lapse video are sky pixels; and A segmentation optimization unit is configured to generate a sky segmentation result of the sky time-lapse video based on the sky time-lapse video, the average sky segmentation map and the confidence map.

11. The image processing device according to claim 10, characterized in that: The initial segmentation unit is configured as follows: Performing sky segmentation on each image frame of the sky time-lapse video using a sky segmentation model to obtain a sky segmentation result for each image frame of the sky time-lapse video; An averaging operation is performed on the sky segmentation results of all the image frames of the sky time-lapse video, and the averaging operation result is used as an average segmentation map of all the image frames of the sky time-lapse video.

12. The image processing device according to claim 10, characterized in that: The optical flow acquisition unit is configured as follows: The average optical flow field of all image frames of the sky time-lapse video is obtained through an optical flow extraction model.

13. The image processing device according to claim 10, characterized in that: The image processing device further comprises: The segmentation map adjustment unit is configured to adjust the average sky segmentation map based on the average optical flow field, and use the adjusted average sky segmentation map as the average sky segmentation map.

14. The image processing device according to claim 10, characterized in that: The confidence map generating unit is configured to: Classifying pixels in the sky time-lapse video into a plurality of preset categories based on confidence and optical flow amplitude of the pixels in the sky time-lapse video; assigning a class confidence to each of the plurality of classes; A multivariate map of the sky time-lapse video is generated based on the confidence, category, and category confidence of each pixel in the sky time-lapse video, and the multivariate map is used as the confidence map of the sky time-lapse video.

15. The image processing device according to claim 14, characterized in that: The confidence map generating unit is configured to: Classifying pixels in the sky time-lapse video whose confidence level is higher than a first threshold as first-category pixels; Classifying pixels in the sky time-lapse video whose confidence level is lower than or equal to a first threshold and whose optical flow amplitude is higher than a second threshold as second-category pixels; Pixels other than the first-category pixels and the second-category pixels in the sky time-lapse video are classified into third-category pixels.

16. The image processing device according to claim 10, characterized in that: The segmentation optimization unit is configured to: Based on the sky time-lapse video, the average sky segmentation map and the confidence map, a preset weight-guided graph filtering optimization algorithm is used to obtain a sky segmentation result of the sky time-lapse video.

17. The image processing device according to claim 13, characterized in that: The segmentation optimization unit is configured to: Selecting pixels whose optical flow magnitude is higher than a third threshold; Determining whether the selected pixel is segmented as a sky pixel in the average sky segmentation map; When the selected pixel is not segmented as a sky pixel in the average sky segmentation map, the selected pixel is adjusted to a sky pixel in the average sky segmentation map.

18. The image processing device according to claim 14, characterized in that: The generated multivariate graph of the sky time-lapse video is a ternary graph.

19. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 9.

20. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor of an electronic device, the electronic device is caused to execute the image processing method according to any one of claims 1 to 9.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 9 is implemented.

Images