Image-based sky area segmentation and special effects processing method, device and equipment

Abstract

The embodiment of the present application provides a method, device and equipment for image-based sky area segmentation and special effects processing, which relates to the field of image processing. The method obtains an original image containing sky content features, inputs the original image into a sky segmentation network for semantic segmentation processing, uses the first convolution unit of the encoder module of the sky segmentation network to extract the first image features with different receptive fields of the original image, inputs the first image features into the second convolution unit of the decoder module of the sky segmentation network for processing, restores the first feature image with the same resolution as the input original image, and obtains a probability image corresponding to the original image, and segments the sky area of ​​the original image according to the probability image, wherein the sky segmentation network is deployed on the client. The technical solution realizes the accurate segmentation of the sky area in the image on the client, with stable performance, fast processing speed, and improved image display effect.

Description

Technical Field

[0001] The present application relates to the field of image processing. Specifically, the present application relates to an image-based sky area segmentation and special effects processing method, device and equipment, and also includes a computer-readable storage medium. Background Art

[0002] With the rapid development of artificial intelligence, especially deep learning, semantic segmentation has become an important research topic with a wide range of application scenarios. Sky segmentation refers to segmenting the sky area in an image, and then realizing weather changes such as sunny days, rainy days, and sunsets through relevant post-processing, and adding special effects such as fireworks, streamers, and auroras.

[0003] In related technologies, semantic segmentation technology based on deep learning is generally used. By inputting images into a deep learning network structure or a convolutional neural network for training, a sky segmentation model is obtained, and feature extraction, segmentation, detection and recognition are performed through the sky segmentation model. However, the current sky segmentation models either occupy a large amount of memory, can only be implemented in the cloud, and cannot be deployed on user terminals to run in real time. Even if some sky segmentation models run on user terminals, the segmentation is inaccurate, resulting in poor segmentation effects and affecting the user experience. Summary of the invention

[0004] The purpose of this application is to solve at least one of the above technical defects, especially the problem of large amount of calculation or low segmentation accuracy.

[0005] In a first aspect, an embodiment of the present application provides an image-based sky region segmentation method, comprising the following steps:

[0006] Obtaining the original image containing sky content features;

[0007] Input an original image into a sky segmentation network, perform semantic segmentation processing on the original image, use a first convolution unit of an encoder module of the sky segmentation network to extract first image features with different receptive fields of the original image, input the first image features into a second convolution unit of a decoder module of the sky segmentation network for processing, restore a first feature image consistent with the resolution of the input original image, and obtain a probability image corresponding to the original image based on the first feature image; wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training a second feature image output by a second convolution unit of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image;

[0008] The sky area of ​​the original image is segmented according to the probability image.

[0009] In one of the embodiments, the encoder module includes multiple layers of first convolutional units;

[0010] The step of extracting first image features of the original image with different receptive fields using the first convolution unit of the encoder module of the sky segmentation network comprises:

[0011] The multi-layer first convolutional units of the encoder module of the sky segmentation network are used to extract first image features with different receptive fields of the original image layer by layer; wherein the resolution of the first image features decreases layer by layer.

[0012] In one embodiment, the decoder module includes multiple layers of second convolution units; the second convolution units correspond one-to-one to the first convolution units;

[0013] The step of inputting the first image feature into the second convolution unit of the decoder module of the sky segmentation network for processing to restore the first feature image having the same resolution as the input original image comprises:

[0014] The first image features output by each first convolution unit of the encoder module are input into the second convolution unit corresponding to the corresponding decoder module by using a feature skipping method, and a network operation is performed layer by layer in combination with an upsampling method to extract the second image features;

[0015] A first feature image having a resolution consistent with the input image to be segmented is restored according to the second image feature.

[0016] In one embodiment, each of the first convolution units includes multiple stacked Conv+BN+ReLU network layers;

[0017] The computing operation of each of the second convolution units includes a Conv+BN+ReLU+UpSample network layer, wherein each layer of the second convolution unit increases the resolution of the output second image features layer by layer through a row bilinear interpolation upsampling operation.

[0018] In one embodiment, the step of training the second feature image output by the second convolutional unit of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image comprises:

[0019] Determine a resolution ratio of a second feature image output by a second convolution unit of at least two layers of the decoder module relative to the original image;

[0020] Determining, according to the resolution ratio, a type and a weight value of a BCE loss function for training each convolutional layer;

[0021] Based on the weight values, the sky segmentation network is constrainedly trained using a BCE loss function of the corresponding type.

[0022] In one embodiment, the step of determining a resolution ratio of a second feature image output by the second convolutional units of at least two layers of the decoder module relative to the original image comprises:

[0023] Adjusting the feature channels of the second convolution units of at least two layers of the decoder module to output channels through 1×1 convolution, and outputting corresponding second feature images;

[0024] Based on the resolution of the second feature image, a resolution ratio of the second feature image to the original image is calculated.

[0025] In one embodiment, the step of obtaining a probability image corresponding to the original image according to the first feature image comprises:

[0026] The first feature image is transformed using a Sigmoid function to obtain a probability image corresponding to the feature image; wherein the first feature image is a second feature image output by the second convolution unit of the last layer of the encoder module.

[0027] In a second aspect, an embodiment of the present application provides an image special effects processing method, including:

[0028] Get the target sky material selected by the user;

[0029] Determine a sky area of ​​an original image to be segmented, wherein the sky area is determined by inputting the original image into a sky segmentation network, performing semantic segmentation processing on the original image, extracting first image features of the original image with different receptive fields using a first convolution unit of an encoder module of the sky segmentation network, inputting the first image features into a second convolution unit of a decoder module of the sky segmentation network for processing, restoring a first feature image consistent with the resolution of the input original image, and obtaining a probability image corresponding to the original image based on the first feature image; wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training a second feature image output by a second convolution unit of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image;

[0030] The target sky material is merged into the sky area to generate a target image.

[0031] In a third aspect, an embodiment of the present application provides an image-based sky area segmentation device, comprising:

[0032] The original image acquisition module is used to acquire the original image containing sky content features;

[0033] A probability image acquisition module, used to input an original image into a sky segmentation network, perform semantic segmentation processing on the original image, use the first convolution unit of the encoder module of the sky segmentation network to extract first image features with different receptive fields of the original image, input the first image features into the second convolution unit of the decoder module of the sky segmentation network for processing, restore and obtain a first feature image consistent with the resolution of the input original image, and obtain a probability image corresponding to the original image based on the first feature image; wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training the second feature images output by the second convolution units of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image;

[0034] The sky area segmentation module is used to segment the sky area of ​​the original image according to the probability image.

[0035] In a fourth aspect, an embodiment of the present application provides an image special effects processing device, including:

[0036] A sky material acquisition module is used to acquire the target sky material selected by the user;

[0037] A sky area determination module is used to determine the sky area of ​​the original image to be segmented, wherein the sky area is determined by inputting the original image into a sky segmentation network, performing semantic segmentation processing on the original image, extracting first image features of the original image with different receptive fields using the first convolution unit of the encoder module of the sky segmentation network, inputting the first image features into the second convolution unit of the decoder module of the sky segmentation network for processing, restoring a first feature image consistent with the resolution of the input original image, and obtaining a probability image corresponding to the original image based on the first feature image; wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training the second feature images output by the second convolution units of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image;

[0038] The target image generation module is used to fuse the target sky material into the sky area to generate a target image.

[0039] In a fifth aspect, an embodiment of the present application provides an electronic device, characterized in that it includes:

[0040] one or more processors;

[0041] Memory;

[0042] One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs are configured to: execute the image-based sky area segmentation method according to the first aspect or the image special effects processing method according to the second aspect.

[0043] In a sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image-based sky area segmentation method described in the first aspect or the image special effects processing method described in the second aspect.

[0044] The image-based sky area segmentation and special effect processing method, device, equipment and computer-readable storage medium provided in the above embodiments obtain an original image containing sky content features, input the original image into a sky segmentation network for semantic segmentation processing, use the first convolution unit of the encoder module of the sky segmentation network to extract the first image features with different receptive fields of the original image, input the first image features into the second convolution unit of the decoder module of the sky segmentation network for processing, restore the first feature image consistent with the resolution of the input original image, and obtain a probability image corresponding to the original image, and segment the sky area of ​​the original image according to the probability image, wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training the second feature images output by the second convolution units of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image, and is deployed on the client, thereby realizing accurate segmentation of the sky area in the image on the client, with stable performance and fast processing speed, improving the image display effect, and meeting user needs.

[0045] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0047] Figure 1 It is a schematic diagram of an application system framework involved in image-based sky area segmentation processing provided in an embodiment of the present application;

[0048] Figure 2 is a flow chart of a method for segmenting a sky area based on an image provided in an embodiment of the present application;

[0049] Figure 3is a working principle diagram of a sky segmentation network provided in an embodiment of the present application;

[0050] Figure 4 is a flow chart of an image special effects processing method provided by an embodiment of the present application;

[0051] Figure 5 This is a schematic diagram of the implementation principle of image special effects processing provided by an embodiment of the present application;

[0052] Figure 6 is a structural schematic diagram of an image-based sky area segmentation device provided by an embodiment;

[0053] Figure 7 It is a structural schematic diagram of an image special effect processing device provided by an embodiment. DETAILED DESCRIPTION

[0054] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0055] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0056] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0057] The application scenarios involved in the embodiments of the present application are described below.

[0058] The embodiments of the present application are applied to scenarios where the sky in an image is transformed in effect. Specifically, it can be applied to images to identify and segment the sky area in the image, and replace the original sky with a target sky style through weather conversion, color change, etc.

[0059] For example, the client is deployed with a neural network that recognizes and segments the sky area of ​​an image. The sky area is determined by the neural network, and the target sky material selected by the user is obtained. The original sky in the image is replaced with the target sky material, and then the image is displayed, thereby improving the image display effect and making the image more attractive.

[0060] Based on the above application scenarios, the neural network needs to be able to execute on the client and accurately segment the sky area to better improve the image display effect. Of course, the technical solution provided in the embodiment of the present application can also be applied to other positioning scenarios, which are not listed here one by one.

[0061] In order to better illustrate the technical solution of the present application, an application environment to which the solution can be applied is shown below. Figure 1 is a schematic diagram of an application system framework involved in image-based sky region segmentation processing provided in an embodiment of the present application, such as Figure 1 As shown, the application system 10 includes a client 101 and a server 102, and a communication connection is established between the client 101 and the server 102 via a wired network or a wireless network.

[0062] The client 101 can be a portable device such as a smart phone, a smart camera, a PDA, a tablet computer, an e-book, and a laptop computer, but is not limited thereto. The client 101 can have functions such as taking pictures and image processing to realize image-based sky area segmentation and special effect processing. Optionally, the client 101 has a touch screen, and the user can perform corresponding operations on the touch screen of the client 101 to realize functions such as sky segmentation, image processing, and special effect synthesis. The client obtains the relevant image, processes the relevant information in the image, and sends it to the server 102, so that the image can be sent to other clients for display through the server 102.

[0063] The server 102 includes an electronic device, such as a backend server that provides backend services to the client 101, which can be implemented by an independent server or a server cluster composed of multiple servers. In one embodiment, the server can be an image sharing platform. After the user takes the image, the image is processed accordingly, such as sky transformation, character beautification, background replacement, etc., and the processed image is uploaded to the server 102, and then the server 102 pushes the image to other clients, so that other users can see the image made by the user.

[0064] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0065] In related image processing technologies, users can set relevant parameters on the client side so that a server with better performance can identify a specified area in the image, such as the sky area. After the style of the sky area is transformed according to the parameters set by the user, the processed image is generated and pushed to each viewer end.

[0066] In this process, the client and server need to be connected to the Internet, and the neural network model is trained with the help of the high performance of the server. Since the neural network model based on deep learning in related technologies has a large amount of computation, it cannot be deployed on the client, and image processing based on the neural network model in an offline state cannot be achieved. In addition, some improved image processing methods can be executed on the client, but the accuracy of other image segmentation is low, resulting in segmentation effect, which affects the image processing effect.

[0067] The image-based sky area segmentation and special effects processing method, device, equipment and computer-readable storage medium provided in this application are intended to solve the above technical problems in the prior art.

[0068] Figure 2 This is a flow chart of an image-based sky region segmentation method provided in an embodiment of the present application, and the image-based sky region segmentation method can be applied to an image-based sky region segmentation device, such as a client. The following description is made by taking a mobile terminal as an example of a client.

[0069] S210: Acquire an original image containing sky content features.

[0070] Generally speaking, users use mobile phone cameras and other devices to capture outdoor images, which include sky areas. Sky content features refer to features that represent the sky, such as clouds, color types, color changes, and color areas.

[0071] In this embodiment, the original image containing the sky content feature may be obtained from a local device such as a mobile phone camera, or from a local storage device or an external storage device. Of course, the original image may not contain the sky content feature. If the original image not containing the sky content feature is obtained, it may be deleted in subsequent processing, or there is no need to perform subsequent sky segmentation or special effects processing.

[0072] Optionally, the image format of the original image may be adjusted according to the sky segmentation scene characteristics.

[0073] In this embodiment, the sky segmentation scene characteristics refer to parameter characteristics that meet the requirements of image processing, such as segmentation accuracy, display resolution, image size, etc. required for image processing, such as photo retouching or short video cover production.

[0074] For example, the image is cropped and the resolution is adjusted, and the height and width of the image are adjusted to (384, 512), where the height of the image is 384 pixels and the width is 512 pixels.

[0075] The preprocessing parameters of the image to be segmented are set according to the characteristics of the sky segmentation scene, and the image format of the original image in the image to be segmented is adjusted according to the preprocessing parameters, such as adjusting the size, resolution and accuracy.

[0076] S220, inputting the original image into a sky segmentation network, performing semantic segmentation processing on the original image, using the first convolution unit of the encoder module of the sky segmentation network to extract the first image features with different receptive fields of the original image, inputting the first image features into the second convolution unit of the decoder module of the sky segmentation network for processing, restoring a first feature image having the same resolution as the input original image, and obtaining a probability image corresponding to the original image based on the first feature image.

[0077] The sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training the second feature image output by the second convolutional units of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image.

[0078] Image semantic segmentation refers to letting the computer segment the image based on its semantics. In the field of images, semantics refers to the content of the image, and the understanding of the meaning of the image. Segmentation means segmenting the different objects in the image from the perspective of pixels, and annotating each pixel in the original image. For example, the white color of the image frame represents the sky area, and the black color represents the non-sky area. Of course, in other embodiments, the white color of the image frame can also represent the sky area, and the black color can represent the non-sky area.

[0079] In this embodiment, the sky segmentation network is a pre-trained convolutional neural network that can adapt to the image-based sky area segmentation scene. It is a lightweight convolutional neural network deployed on the client and can transform the style of the sky area in the image in real time. Compared with the high computational load of the related convolutional neural network, the sky segmentation model of this embodiment performs structural adjustment and parameter adjustment to greatly reduce the computational load on the basis of ensuring the recognition accuracy. Generally speaking, the more channels a neural network has, the greater the data computational load. The sky segmentation network compresses the number of channels and reduces the computational load. For example, the basic number of channels is compressed from 64 to 16. Sky segmentation is based on image processing, and the computational load requirement is lower, so the number of channels is set to 16, which is suitable for reducing the number of parameters. While reducing the computational load, the accuracy and stability of the segmentation effect are also guaranteed. In this embodiment, other parameter adjustments or structural adjustments can also be used to compensate for the segmentation effect caused by the compression of the number of channels to ensure segmentation accuracy.

[0080] In this embodiment, the first convolution unit of the encoder module of the sky segmentation network is used to extract the first image features with different receptive fields of the original image, and the first image features are input into the second convolution unit of the decoder module of the sky segmentation network for processing, so as to restore the first feature image having the same resolution as the input original image, and obtain the probability image corresponding to the original image based on the first feature image.

[0081] The encoder module includes multiple layers of first convolutional units; the decoder module includes multiple layers of second convolutional units, and the second convolutional units correspond to the first convolutional units one by one. In this embodiment, the decoder module includes five layers of second convolutional units, and the decoder module includes five layers of second convolutional units.

[0082] In one embodiment, extracting first image features of the original image with different receptive fields using the first convolution unit of the encoder module of the sky segmentation network in step S220 may include the following steps:

[0083] S2201. Utilize the multi-layer first convolutional units of the encoder module of the sky segmentation network to extract first image features with different receptive fields of the original image layer by layer; wherein the resolution of the first image features decreases layer by layer.

[0084] In a convolutional neural network, the definition of receptive field is the size of the area on the input image where the pixels on the feature map output by each layer of the convolutional neural network are mapped. A more popular explanation is that a point on the feature image corresponds to an area on the input image. In this embodiment, the semantic information and spatial information of the original image are determined by acquiring the first image features with different receptive fields, which is beneficial to improving the accuracy of pixel classification and edge segmentation accuracy of the original image.

[0085] Each first convolution unit includes multiple stacked Conv+BN+ReLU network layers; in this embodiment, the second convolution unit includes two stacked Conv+BN+ReLU network layers.

[0086] In this embodiment, after each processing by a layer of the first convolution unit, the resolution of the output first image features is halved relative to the resolution of the first image features input to the layer.

[0087] In one embodiment, the decoder module includes multiple layers of second convolution units; the second convolution units correspond one to one with the first convolution units. The operation of each second convolution unit includes a Conv+BN+ReLU+UpSample network layer, wherein each layer of the second convolution unit increases the resolution of the output second image features layer by layer through a bilinear interpolation upsampling operation, thereby restoring a feature image with the same resolution as the input original image layer by layer, that is, the first feature image.

[0088] In one embodiment, the step S220 of inputting the first image feature into the second convolution unit of the decoder module of the sky segmentation network for processing to restore the first feature image having the same resolution as the input original image comprises the following steps:

[0089] S2202. The first image features output by each first convolution unit of the encoder module are input into the second convolution unit corresponding to the corresponding decoder module by using feature jumping, and network operations are performed layer by layer in combination with upsampling to extract the second image features.

[0090] Feature skip connect means that the first image feature output by the first convolution unit in the middle layer (not the last layer) of the encoder module is jumped to the corresponding second convolution unit of the corresponding decoder module, and convolution operation is performed with the second image feature output by the second convolution unit of the previous layer corresponding to this layer.

[0091] Figure 3 is a working principle diagram of a sky segmentation network provided in an embodiment of the present application, such as Figure 3As shown, the first image feature output by the in_conv convolution unit of the encoder module is jumped and input to the Decoder_5 convolution unit corresponding to the decoder module, the first image feature output by the Encoder_1 convolution unit of the encoder module is jumped and input to the Decoder_4 convolution unit corresponding to the decoder module, the first image feature output by the Encoder_2 convolution unit of the encoder module is jumped and input to the Decoder_3 convolution unit corresponding to the decoder module, the first image feature output by the Encoder_3 convolution unit of the encoder module is jumped and input to the Decoder_2 convolution unit corresponding to the decoder module, and the first image feature output by the Encoder_4 convolution unit of the encoder module is jumped and input to the Decoder_1 convolution unit corresponding to the decoder module, thereby realizing feature jumping, so as to input the first image features output by each first convolution unit of the encoder module into the second convolution unit corresponding to the corresponding decoder module. At the same time, the Decoder_1 convolution unit corresponding to the decoder module obtains the first image feature output by the Encoder_5 convolution unit, the Decoder_2 corresponding to the decoder module obtains the second image feature output by the Decoder_1 convolution unit, the Decoder_3 convolution unit corresponding to the decoder module obtains the second image feature output by the Decoder_2 convolution unit, the Decoder_4 convolution unit corresponding to the decoder module obtains the second image feature output by the Decoder_3 convolution unit, and the Decoder_5 corresponding to the decoder module obtains the second image feature output by the Decoder_4 convolution layer, thereby realizing that the first image features output by each first convolution unit of the encoder module are input into the second convolution unit corresponding to the corresponding decoder module, and the upsampling method is used to combine the second image features output by each second convolution unit of the decoder module to perform network operations layer by layer to extract the second image features.

[0092] S2203: restore the second image features to obtain a first feature image having a resolution consistent with the input image to be segmented.

[0093] In this embodiment, the decoder module doubles the resolution of the second image feature layer by layer through multiple layers of second convolution units, so that when the resolution of the second image feature output by the second convolution unit of the last layer is consistent with the resolution of the input original image, a first feature image is obtained.

[0094] Furthermore, a probability image corresponding to the original image is obtained according to the first feature image. The probability image refers to an image composed of pixel values ​​of 0 or 1, which is a single-channel image. In the probability image, the value of each pixel is 0 or 1, and the size of the probability image is the same as the size of the input image to be segmented.

[0095] S230, receiving the probability image output by the sky segmentation network, and segmenting the sky area of ​​the original image according to the probability image.

[0096] After the sky segmentation network performs semantic segmentation on the original image, it outputs the corresponding probability image. The value of each pixel in the probability image is 1 or 0. The sky segmentation area is determined according to the pixel value. For example, the pixel value 255 is white, and the white area is the sky area.

[0097] Optionally, the sky region of the image may be segmented according to the probability image output by the sky segmentation network. In this embodiment, the probability image may be subsequently processed, such as guided filtering to optimize the segmentation accuracy of the sky region, or inter-frame mean smoothing to obtain a more accurate probability image, etc., thereby further improving the segmentation effect of the sky region of the image.

[0098] The image-based sky area segmentation method provided in this embodiment obtains an original image containing sky content features; inputs the original image into a sky segmentation network, performs semantic segmentation processing on the original image, and obtains a probability image corresponding to the original image; receives the probability image output by the sky segmentation network, and segments the sky area of ​​the original image according to the probability image, wherein the sky segmentation network is a convolutional neural network with compressed channel number, thereby achieving accurate segmentation of the sky area in the image on the client, with stable performance and fast processing speed, meeting user needs.

[0099] In order to make the present technical solution clearer and easier to understand, the specific implementation process and method of the multiple steps in the present technical solution are described in detail below.

[0100] It should be noted that in the related art, the sky segmentation implementation schemes are all based on image input for processing, which generally have a large amount of calculation. If the neural network in the related art, such as the Unet network, is directly transplanted to the client, the processing speed of sky segmentation is slow or even freezes and cannot work, thereby causing a stuck end, making it impossible for the sky segmentation network in the related art to be deployed on the client for operation. The present application scheme is based on a lightweight sky segmentation network to perform sky segmentation of images, which can be deployed on the client and can accurately segment the sky area in real time, ensuring segmentation accuracy while avoiding stuck.

[0101] Based on this, the image-based sky region segmentation method provided by the present application solution further includes: training and generating a sky segmentation network. In one embodiment, the sky segmentation network can be obtained by:

[0102] S300. Compress the number of basic channels of the Unet network model and increase the number of layers of the Unet network model according to the characteristics of the sky segmentation scene, so that the feature image output by the modified number of basic channels and the number of layers is adapted to the characteristics of the sky segmentation scene, and a lightweight sky segmentation network is trained.

[0103] In this embodiment, the sky segmentation scene characteristics refer to parameter characteristics that meet the requirements of image production platforms, such as segmentation accuracy, display resolution, image size, etc., required for the production of beautiful pictures, short videos, or small videos.

[0104] In this embodiment, the convolutional neural network used in the sky segmentation network is a neural network built on the Unet network architecture. The Unet network architecture first downsamples, performs convolutions of different degrees to learn deep features, and then restores the original image size through upsampling to output a probability image corresponding to the original image.

[0105] In one embodiment, the number of basic channels of the Unet network architecture is compressed from 64 to 16, and at the same time, the 4 convolutional layers of the convolutional neural network are increased to 5 layers to compensate for the loss of segmentation effect caused by the compression of the number of basic channels and enhance the feature extraction capability of the sky segmentation network, so that the resolution of the feature image output by the sky segmentation network with the number of basic channels and the number of convolutional layers is adapted to the characteristics of the sky segmentation scene, thereby obtaining a lightweight sky segmentation network that can be deployed and run on the client.

[0106] In this embodiment, each convolution layer of the sky segmentation network includes several encoder basic convolution modules and decoder basic convolution modules; wherein the encoder basic convolution module includes a twice stacked Conv+BN+ReLU network layer, and the decoder basic convolution module includes a Conv+BN+ReLU+UpSample network layer. Compared with the encoder basic convolution module and the decoder basic convolution module in the related art, the encoder basic convolution module and the decoder basic convolution module provided in this embodiment add BN operations to make the training more stable and the segmentation effect better.

[0107] In this embodiment, the number of encoder basic convolution modules and decoder basic convolution modules included in each convolution layer of the sky segmentation network can be different, so as to achieve different training effects.

[0108] Compared with the original Unet network model with a computing volume of Flops=46.3G, the sky segmentation network provided in this embodiment that can perform sky segmentation in real time has a computing volume of Flops=370M, and the processing speed is increased by more than 160 times. The sky segmentation network can be deployed on mid- and low-end Android smartphones to run in real time and has better segmentation accuracy and stability.

[0109] In one embodiment, obtaining the probability image corresponding to the original image according to the first feature image in step S220 may include the following steps:

[0110] S2204: Convert the first feature image using a Sigmoid function to obtain a probability image corresponding to the first feature image.

[0111] The first feature image is a single-channel image of the same size as the original image. A single-channel image, commonly known as a grayscale image, has only one value per pixel to represent the color. Its pixel value is between 0 and 255, where 0 is black, 255 is white, and the intermediate values ​​are some different levels of gray. The pixel value range of the first feature image is 0 to 255.

[0112] The value range of the Sigmoid function is between 0 and 1, and the Sigmoid function has very good symmetry. In this embodiment, the feature image with a pixel value range of 0 to 255 is converted using the Sigmoid function, and the value range of the probability image corresponding to the output feature image is 0 to 1.

[0113] The Sigmoid function is:

[0114]

[0115] Among them, x is the pixel value of the feature image, and f(x) is the value of the pixel point of the feature image in the probability image.

[0116] In one embodiment, in order to optimize the segmentation effect of the sky segmentation network and reduce the phenomenon of missed detection and false detection, a multi-scale loss constraint method is used to optimize the model training during the training process. Specifically, the second feature image output by the second convolution unit of at least two layers of the decoder module is trained using a loss function corresponding to the resolution of the second feature image, which may include the following steps:

[0117] S3101. Determine a resolution ratio of a second feature image output by second convolution units of at least two layers of a decoder module relative to the original image.

[0118] In this embodiment, the second convolution unit of the previous layer of the decoder module outputs the second feature image as the input of the second convolution unit of the next layer. The resolution of the second feature image output by each layer of the second convolution unit is half the resolution of the second feature image input by the second convolution unit of this layer (i.e., the second feature image output by the previous second convolution unit). For example, if the resolution of the input second feature image is (384,512), the resolution of the output second feature image is (192,256). In this embodiment, the feature channel can be adjusted to the output channel through 1×1 convolution, and the second feature image output by each second convolution unit is output. In the prediction stage, the neural network does not need to calculate these additional 1×1 convolutions, so no additional calculation is added.

[0119] After passing through the second convolution unit of the first layer, the resolution of the output second feature image is half of the resolution of the original image, and the second feature image is input to the second convolution unit of the second layer for processing. After passing through the second convolution unit of the second layer, the resolution of the output second feature image is 1 / 4 of the resolution of the original image (that is, 1 / 2 of the resolution of the second feature image of the previous layer), and so on. The resolution of the second feature image output by at least two second convolution units of the decoder module is obtained, and based on the resolution of the second feature image, the resolution ratio of the second feature image to the resolution of the original image is determined.

[0120] S3102. Determine, according to the resolution ratio, the type and weight value of a BCE (Binary CrossEntropy) loss function for training the second convolutional units of the at least two layers.

[0121] In this embodiment, the type and weight value of the BCE loss function for training the second convolutional units of at least two layers match the resolution of the feature image output by the convolutional layer.

[0122] For example, on the branch outputted by the second feature image with a resolution ratio of 1 / 2, training is performed using the 1 / 2BCE loss function, on the branch outputted by the second feature image with a resolution ratio of 1 / 4, training is performed using the 1 / 4BCE loss function, and on the branch outputted by the second feature image with a resolution ratio of 1 / 8, training is performed using the 1 / 8BCE loss function. Among them, the 1 / 2BCE loss function, the 1 / 4BCE loss function, and the 1 / 8BCE loss function are different types of loss functions.

[0123] Furthermore, the weight of the loss function on each output branch can be set to make the training results more accurate and stable.

[0124] S3103: Based on the weight values, a BCE loss function of a corresponding type is used to perform constrained training on the sky segmentation network.

[0125] In this embodiment, based on the weight of the loss function on each output branch, the final loss Loss of the training phase is calculated as:

[0126]

[0127] in, represents the network prediction value, y represents the true value, and the values ​​of α, β, θ, and λ are 10, 2, 1, and 1. Of course, in other application scenarios, α, β, θ, and λ can be other values, which are not limited here.

[0128] Figure 4 It is a flowchart of an image special effects processing method provided in an embodiment of the present application. The image special effects processing method can be applied to an image special effects processing device, such as a client.

[0129] Specifically, Figure 4 As shown, the image special effects processing method may include the following steps:

[0130] S410: Acquire the target sky material selected by the user.

[0131] In this embodiment, the client has pre-downloaded sky materials locally. The user clicks a related button on the display interface of the client to trigger a pop-up panel of sky materials, and displays different sky materials through the panel. These sky materials can be system pre-configured sky materials or user-defined sky materials.

[0132] The user selects one or more sky materials on the panel as target sky materials for subsequent processing.

[0133] S420: Determine a sky area of ​​the original image to be segmented.

[0134] Among them, the sky area is determined by inputting the original image into a sky segmentation network, performing semantic segmentation processing on the original image, using the first convolution unit of the encoder module of the sky segmentation network to extract the first image features of the original image with different receptive fields, inputting the first image features into the second convolution unit of the decoder module of the sky segmentation network for processing, restoring the first feature image consistent with the resolution of the input original image, and obtaining the probability image corresponding to the original image based on the first feature image; wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training the second feature images output by the second convolution units of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image.

[0135] Get the original image containing sky content features. Generally speaking, users take outdoor images with devices such as mobile phone cameras, and these outdoor images include sky areas. Sky content features refer to features that represent the sky, such as clouds, color types, color changes, and color areas.

[0136] In this embodiment, the original image containing the sky content feature may be obtained from a local device such as a mobile phone camera, or may be obtained from a local storage device or an external storage device.

[0137] The image format of the original image is adjusted according to the sky segmentation scene characteristics. In this embodiment, the sky segmentation scene characteristics refer to parameter characteristics that meet the requirements of image processing, such as segmentation accuracy, display resolution, image size, etc. required for image processing, such as photo retouching or short video cover production.

[0138] For example, the image is cropped and the resolution is adjusted, and the height and width of the image are adjusted to (384, 512), where the height of the image is 384 pixels and the width is 512 pixels.

[0139] The preprocessing parameters of the image to be segmented are set according to the characteristics of the sky segmentation scene, and the image format of the original image in the image to be segmented is adjusted according to the preprocessing parameters, such as adjusting the size, resolution and accuracy.

[0140] The original image after image format adjustment is input into a sky segmentation network, and semantic segmentation processing is performed on the original image to obtain a probability image corresponding to the original image; wherein the sky segmentation network is a convolutional neural network with compressed channel number.

[0141] Image semantic segmentation refers to letting the computer segment the image based on its semantics. In the field of images, semantics refers to the content of the image, and the understanding of the meaning of the image. Segmentation means segmenting the different objects in the image from the perspective of pixels, and annotating each pixel in the original image. For example, the white color of the image frame represents the sky area, and the black color represents the non-sky area. Of course, in other embodiments, the white color of the image frame can also represent the sky area, and the black color can represent the non-sky area.

[0142] In this embodiment, the sky segmentation network is a pre-trained convolutional neural network that can adapt to the image-based sky area segmentation scene. It is a lightweight convolutional neural network deployed on the client and can transform the style of the sky area in the image in real time. Compared with the high computational load of the related convolutional neural network, the sky segmentation model of this embodiment performs structural adjustment and parameter adjustment to greatly reduce the computational load on the basis of ensuring the recognition accuracy. Generally speaking, the more channels a neural network has, the greater the data computational load. The sky segmentation network compresses the number of channels and reduces the computational load. For example, the basic number of channels is compressed from 64 to 16. Sky segmentation is based on image processing, and the computational load requirement is lower, so the number of channels is set to 16, which is suitable for reducing the number of parameters. While reducing the computational load, the accuracy and stability of the segmentation effect are also guaranteed. In this embodiment, other parameter adjustments or structural adjustments can also be used to compensate for the segmentation effect caused by the compression of the number of channels to ensure segmentation accuracy.

[0143] In this embodiment, the first convolution unit of the encoder module of the sky segmentation network is used to extract the first image features with different receptive fields of the original image, and the first image features are input into the second convolution unit of the decoder module of the sky segmentation network for processing, so as to restore the first feature image having the same resolution as the input original image, and obtain the probability image corresponding to the original image based on the first feature image.

[0144] The encoder module includes multiple layers of first convolutional units; the decoder module includes multiple layers of second convolutional units, and the second convolutional units correspond to the first convolutional units one by one. In this embodiment, the decoder module includes five layers of second convolutional units, and the decoder module includes five layers of second convolutional units.

[0145] In one embodiment, the multi-layer first convolutional units of the encoder module of the sky segmentation network are used to extract first image features with different receptive fields of the original image layer by layer; wherein the resolution of the first image features decreases layer by layer.

[0146] In a convolutional neural network, the definition of receptive field is the size of the area on the input image where the pixels on the feature map output by each layer of the convolutional neural network are mapped. A more popular explanation is that a point on the feature image corresponds to an area on the input image. In this embodiment, the semantic information and spatial information of the original image are determined by acquiring the first image features with different receptive fields, which is beneficial to improving the accuracy of pixel classification and edge segmentation accuracy of the original image.

[0147] Each first convolution unit includes multiple stacked Conv+BN+ReLU network layers; in this embodiment, the second convolution unit includes two stacked Conv+BN+ReLU network layers.

[0148] In this embodiment, after each processing by a layer of the first convolution unit, the resolution of the output first image features is halved relative to the resolution of the first image features input to the layer.

[0149] In one embodiment, the decoder module includes multiple layers of second convolution units; the second convolution units correspond one to one with the first convolution units. The operation of each second convolution unit includes a Conv+BN+ReLU+UpSample network layer, wherein each layer of the second convolution unit increases the resolution of the output second image features layer by layer through a bilinear interpolation upsampling operation, thereby restoring a feature image with the same resolution as the input original image layer by layer, that is, the first feature image.

[0150] In one embodiment, the first image features output by each first convolution unit of the encoder module are input into the second convolution unit corresponding to the corresponding decoder module by feature jumping, and the network operation is performed layer by layer in combination with upsampling to extract the second image features.

[0151] Feature skip connect means that the first image feature output by the first convolution unit in the middle layer (not the last layer) of the encoder module is jumped to the corresponding second convolution unit of the corresponding decoder module, and convolution operation is performed with the second image feature output by the second convolution unit of the previous layer corresponding to this layer.

[0152] Figure 3 is a working principle diagram of a sky segmentation network provided in an embodiment of the present application, such as Figure 3As shown, the first image feature output by the in_conv convolution unit of the encoder module is jumped and input to the Decoder_5 convolution unit corresponding to the decoder module, the first image feature output by the Encoder_1 convolution unit of the encoder module is jumped and input to the Decoder_4 convolution unit corresponding to the decoder module, the first image feature output by the Encoder_2 convolution unit of the encoder module is jumped and input to the Decoder_3 convolution unit corresponding to the decoder module, the first image feature output by the Encoder_3 convolution unit of the encoder module is jumped and input to the Decoder_2 convolution unit corresponding to the decoder module, and the first image feature output by the Encoder_4 convolution unit of the encoder module is jumped and input to the Decoder_1 convolution unit corresponding to the decoder module, thereby realizing feature jumping, so as to input the first image features output by each first convolution unit of the encoder module into the second convolution unit corresponding to the corresponding decoder module. At the same time, the Decoder_1 convolution unit corresponding to the decoder module obtains the first image feature output by the Encoder_5 convolution unit, the Decoder_2 corresponding to the decoder module obtains the second image feature output by the Decoder_1 convolution unit, the Decoder_3 convolution unit corresponding to the decoder module obtains the second image feature output by the Decoder_2 convolution unit, the Decoder_4 convolution unit corresponding to the decoder module obtains the second image feature output by the Decoder_3 convolution unit, and the Decoder_5 corresponding to the decoder module obtains the second image feature output by the Decoder_4 convolution layer, thereby realizing that the first image features output by each first convolution unit of the encoder module are input into the second convolution unit corresponding to the corresponding decoder module, and the upsampling method is used to combine the second image features output by each second convolution unit of the decoder module to perform network operations layer by layer to extract the second image features.

[0153] In one embodiment, a first feature image having a resolution consistent with the input image to be segmented is restored according to the second image feature.

[0154] In this embodiment, the decoder module doubles the resolution of the second image feature layer by layer through multiple layers of second convolution units, so that when the resolution of the second image feature output by the second convolution unit of the last layer is consistent with the resolution of the input original image, a first feature image is obtained.

[0155] Furthermore, a probability image corresponding to the original image is obtained according to the first feature image. The probability image refers to an image composed of pixel values ​​of 0 or 1, which is a single-channel image. In the probability image, the value of each pixel is 0 or 1, and the size of the probability image is the same as the size of the input image to be segmented.

[0156] The client receives the probability image output by the sky segmentation network, and segments the sky area of ​​the original image according to the probability image.

[0157] After the sky segmentation network performs semantic segmentation on the original image, it outputs the corresponding probability image. The value of each pixel in the probability image is 1 or 0. The sky segmentation area is determined according to the pixel value. For example, the pixel value 255 is white, and the white area is the sky area.

[0158] Optionally, the sky region of the image may be segmented according to the probability image output by the sky segmentation network. In this embodiment, the probability image may be subsequently processed, such as guided filtering to optimize the segmentation accuracy of the sky region, or inter-frame mean smoothing to obtain a more accurate probability image, etc., thereby further improving the segmentation effect of the sky region of the image.

[0159] S430: Merge the target sky material into the sky area to generate a target image.

[0160] Overlay the target sky material on the sky area of ​​the original image, or delete the original sky style of the original image and replace it with the target sky material selected by the user, and merge the target sky material into the original image to obtain the target image, so that the target image has sky effects, for example, changing a gloomy sky into a clear sunny sky.

[0161] The image special effects processing method provided in this embodiment obtains the target sky material selected by the user; uses a pre-trained lightweight sky segmentation network to perform semantic segmentation processing on the original image to obtain a probability image corresponding to the original image; receives the probability image output by the sky segmentation network, and determines that the probability image corresponds to the sky area of ​​the original image for segmentation, performs special effects processing on the sky area using the target sky material, and outputs a target image fused with the target sky material, thereby realizing accurate segmentation of the sky area of ​​the original image and replacement of the sky material in real time on the client to meet the needs of users.

[0162] In order to explain this scheme more clearly, Figure 5 The implementation process of this solution is illustrated by way of example. Figure 5 This is a schematic diagram of the implementation principle of image special effects processing provided in one embodiment of the present application.

[0163] like Figure 5As shown, the original input image is obtained, and the original image is input into the pre-trained sky segmentation network for semantic segmentation. The encoding module and the convolution unit of the decoding module of the sky segmentation network are used for encoding and decoding. In the encoding process, the convolutional encoding process of the convolutional layers such as in_conv, Encoder_1, Encoder_2, Encoder_3, Encoder_4 and Encoder_5 is performed layer by layer, and then the probability image is obtained after decoding processing such as Decoder_1, Decoder_2, Decoder_3, Decoder_4 and Decoder_5.

[0164] like Figure 5 As shown, for each Decoder_i (i is a positive integer, such as 1, 2, 3, 4 and 5) channel output branch, the output result is trained using the corresponding BCE loss function. For example, if the resolution ratio of the second feature image output by the Decoder_2 channel to the original image is 1 / 8, then the 1 / 8BCE loss function is used for training. Furthermore, the resolution ratio of the second feature image output by the Decoder_3 channel to the original image is 1 / 4, then the 1 / 4BCE loss function is used for training. The resolution ratio of the second feature image output by the Decoder_4 channel to the original image is 1 / 2, then the 1 / 2BCE loss function is used for training. The resolution ratio of the second feature image output by the Decoder_5 channel to the original image is 1 / 1, then the 1 / 1BCE loss function is used for training.

[0165] In this embodiment, different BCE loss function constraints are applied to the output results of different convolutional layer output channels, which is beneficial to model training and learning more refined edge information.

[0166] The above examples are only used to assist in explaining the technical solution of the present disclosure, and the illustrated contents and specific processes involved do not constitute a limitation on the usage scenarios of the technical solution of the present disclosure.

[0167] The following is a detailed description of the relevant embodiments of the image-based sky area segmentation device and the image special effect processing device.

[0168] Figure 6 It is a structural schematic diagram of an image-based sky region segmentation apparatus provided by an embodiment. The image-based sky region segmentation apparatus can be executed in an image-based sky region segmentation device, such as a client.

[0169] Specifically, the image-based sky region segmentation device 200 includes: an original image acquisition module 210 , a probability image acquisition module 220 and a sky region segmentation module 230 .

[0170] Among them, the original image acquisition module 210 is used to acquire the original image containing sky content features; the probability image acquisition module 220 is used to input the original image after the image format is adjusted into the sky segmentation network, perform semantic segmentation processing on the original image, use the first convolution unit of the encoder module of the sky segmentation network to extract the first image features of the original image with different receptive fields, input the first image features into the second convolution unit of the decoder module of the sky segmentation network for processing, restore the first feature image consistent with the resolution of the input original image, and obtain the probability image corresponding to the original image based on the first feature image; wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training the second feature images output by the second convolution units of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image; the sky area segmentation module 230 is used to segment the sky area of ​​the original image based on the probability image.

[0171] The image-based sky area segmentation device provided in this embodiment is deployed on the client and can realize accurate segmentation of the sky area of ​​the image on the client. It has stable performance and fast processing speed, and meets the needs of users.

[0172] In one embodiment, the encoder module comprises a multi-layer first convolution unit;

[0173] The probability image acquisition module 220 includes: a first image feature extraction unit, which is used to extract the first image features with different receptive fields of the original image layer by layer using the multi-layer first convolution units of the encoder module of the sky segmentation network; wherein the resolution of the first image features decreases layer by layer.

[0174] In one embodiment, the decoder module includes multiple layers of second convolution units; the second convolution units correspond one-to-one to the first convolution units;

[0175] The probability image obtaining module 220 includes: a second image feature extraction unit and a first feature image obtaining unit;

[0176] The second image feature extraction unit is used to input the first image features output by each first convolution unit of the encoder module into the second convolution unit corresponding to the corresponding decoder module by means of feature jumping, and perform network operations layer by layer in combination with upsampling to extract the second image features; the first feature image acquisition unit is used to restore the first feature image consistent with the resolution of the input image to be segmented according to the second image features.

[0177] In one embodiment, each of the first convolution units includes multiple stacked Conv+BN+ReLU network layers; the operation of each of the second convolution units includes a Conv+BN+ReLU+UpSample network layer, wherein each layer of the second convolution unit increases the resolution of the output second image features layer by layer through a bilinear interpolation upsampling operation.

[0178] In one embodiment, the image-based sky area segmentation device 200 also includes: a segmentation network training module, which is used to train the second feature image output by the second convolutional units of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image.

[0179] In one embodiment, the segmentation network training module includes: a resolution ratio unit, a weight value determination unit and a constraint training unit; wherein the resolution ratio unit is used to determine the resolution ratio of the second feature image output by the second convolution units of at least two layers of the decoder module relative to the original image; the weight value determination unit is used to determine the type and weight value of the BCE loss function used to train the second convolution units of the at least two layers according to the resolution ratio; the constraint training unit is used to perform constraint training on the sky segmentation network using the corresponding type of BCE loss function based on the weight value.

[0180] In one embodiment, the resolution ratio unit includes: an output channel adjustment subunit and a resolution ratio determination unit;

[0181] Among them, the output channel adjustment subunit is used to adjust the feature channels of the second convolution units of at least two layers of the decoder module into output channels through 1×1 convolution, and output the corresponding second feature image; the resolution ratio determination unit is used to calculate the resolution ratio of the second feature image relative to the original image based on the resolution of the second feature image.

[0182] In one embodiment, the probability image obtaining module 220 includes: a probability image obtaining unit, configured to transform the first feature image using a Sigmoid function to obtain a probability image corresponding to the feature image.

[0183] The image-based sky area segmentation device of the embodiment of the present disclosure can execute the image-based sky area segmentation method provided by the embodiment of the present disclosure. Its implementation principle is similar and has the same beneficial effects as the image special effects processing method, which will not be repeated here.

[0184] Figure 7 It is a structural schematic diagram of an image special effect processing apparatus provided by an embodiment. The image special effect processing apparatus can be executed on an image special effect processing device, such as a client.

[0185] Specifically, Figure 7 As shown, the image special effect processing device 400 includes: a sky material acquisition module 410 , a sky area determination module 420 and a target image generation module 430 .

[0186] The sky material acquisition module 410 is used to acquire the target sky material selected by the user;

[0187] A sky area determination module 420 is used to determine the sky area of ​​the original image to be segmented, wherein the sky area is determined by inputting the original image into a sky segmentation network, performing semantic segmentation processing on the original image, extracting first image features of the original image with different receptive fields using the first convolution unit of the encoder module of the sky segmentation network, inputting the first image features into the second convolution unit of the decoder module of the sky segmentation network for processing, restoring a first feature image consistent with the resolution of the input original image, and obtaining a probability image corresponding to the original image based on the first feature image; wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training the second feature images output by the second convolution units of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image;

[0188] The target image generation module 430 is used to merge the target sky material into the sky area to generate a target image.

[0189] The image special effects processing device provided in this embodiment is deployed on the client, and can realize accurate segmentation of the sky area of ​​the image and synthesis processing of the sky special effects on the client, with stable performance and fast processing speed, meeting the needs of users.

[0190] The image special effects processing device of the embodiment of the present disclosure can execute the image special effects processing method provided by the embodiment of the present disclosure. Its implementation principle is similar and has the same beneficial effects as the image special effects processing method, which will not be repeated here.

[0191] An embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an image-based sky area segmentation method or an image special effects processing method as in any of the above embodiments is implemented.

[0192] When the computer device provided above executes the image-based sky area segmentation method or image special effects processing method provided by any of the above embodiments, it has corresponding functions and beneficial effects.

[0193] An embodiment of the present invention provides a storage medium containing computer executable instructions, wherein the computer executable instructions are used to perform an image-based sky region segmentation method when executed by a computer processor, including:

[0194] Obtaining the original image containing sky content features;

[0195] Input an original image into a sky segmentation network, perform semantic segmentation processing on the original image, use a first convolution unit of an encoder module of the sky segmentation network to extract first image features with different receptive fields of the original image, input the first image features into a second convolution unit of a decoder module of the sky segmentation network for processing, restore a first feature image consistent with the resolution of the input original image, and obtain a probability image corresponding to the original image based on the first feature image; wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training a second feature image output by a second convolution unit of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image;

[0196] The probability image output by the sky segmentation network is received, and the sky area of ​​the original image is segmented according to the probability image.

[0197] When the computer executable instructions are executed by a computer processor, they are also used to execute an image special effect processing method, including:

[0198] Get the target sky material selected by the user;

[0199] Determine a sky area to be segmented of the original image, wherein the sky area is determined by inputting the original image into a sky segmentation network, performing semantic segmentation processing on the original image, extracting first image features of the original image with different receptive fields using a first convolution unit of an encoder module of the sky segmentation network, inputting the first image features into a second convolution unit of a decoder module of the sky segmentation network for processing, restoring a first feature image consistent with the resolution of the input original image, and obtaining a probability image corresponding to the original image based on the first feature image; wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training a second feature image output by a second convolution unit of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image;

[0200] The target sky material is merged into the sky area to generate a target image.

[0201] Of course, the storage medium containing computer executable instructions provided by the embodiment of the present invention, whose computer executable instructions are not limited to the operations of the image-based sky area segmentation method or image special effects processing method as described above, has corresponding functions and beneficial effects.

[0202] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform the image-based sky area segmentation method or image special effects processing method described in any embodiment of the present invention.

[0203] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a portion of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps. The above is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the scope of protection of the present application.

[0204] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for segmenting sky area based on an image, characterized in that: The following steps are involved: Obtaining the original image containing sky content features; The original image is input into a sky segmentation network, and semantic segmentation processing is performed on the original image. The first convolution unit of the encoder module of the sky segmentation network is used to extract the first image features with different receptive fields of the original image, and the first image features are input into the second convolution unit of the decoder module of the sky segmentation network for processing, so as to restore a first feature image consistent with the resolution of the input original image, and obtain a probability image corresponding to the original image according to the first feature image; wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training the second feature images output by the second convolution units of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image; Segmenting the sky area of ​​the original image according to the probability image; The sky segmentation network is obtained based on the following method: Calling a Unet network model, compressing the number of basic channels of the Unet network model and increasing the number of layers of the Unet network model according to the characteristics of the sky segmentation scene, so that the feature image output by the modified number of basic channels and the number of layers is adapted to the characteristics of the sky segmentation scene, thereby obtaining the sky segmentation network; The convolution unit of the Unet network model is a separable convolution unit; The step of training the second feature image output by the second convolutional unit of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image comprises: Determine a resolution ratio of a second feature image output by the second convolutional units of at least two layers of the decoder module relative to the original image; determine the type and weight value of a BCE loss function for training the second convolutional units of the at least two layers according to the resolution ratio; for each layer of the second convolutional units, based on the corresponding weight value, and using the corresponding type of BCE loss function to constrain the sky segmentation network training; the types of the BCE loss function include 1 / 2BCE loss function, 1 / 4BCE loss function and 1 / 8BCE loss function; The step of determining a resolution ratio of a second feature image output by the second convolutional units of at least two layers of the decoder module relative to the original image comprises: Adjusting the feature channels of the second convolution units of at least two layers of the decoder module to output channels through 1×1 convolution, and outputting the corresponding second feature images; Based on the resolution of the second feature image, a resolution ratio of the second feature image to the original image is calculated.

2. The image-based sky area segmentation method according to claim 1, characterized in that: The encoder module includes multiple layers of first convolutional units; The step of extracting first image features of the original image with different receptive fields using the first convolution unit of the encoder module of the sky segmentation network comprises: The multi-layer first convolutional units of the encoder module of the sky segmentation network are used to extract first image features with different receptive fields of the original image layer by layer; wherein the resolution of the first image features decreases layer by layer.

3. The image-based sky area segmentation method according to claim 2, characterized in that: The decoder module includes multiple layers of second convolution units; the second convolution units correspond one-to-one to the first convolution units; The step of inputting the first image feature into the second convolution unit of the decoder module of the sky segmentation network for processing to restore the first feature image having the same resolution as the input original image comprises: The first image features output by each first convolution unit of the encoder module are input into the second convolution unit corresponding to the corresponding decoder module by using a feature skipping method, and a network operation is performed layer by layer in combination with an upsampling method to extract the second image features; A first feature image having a resolution consistent with the input image to be segmented is restored according to the second image feature.

4. The image-based sky area segmentation method according to claim 3, characterized in that: Each of the first convolutional units includes multiple stacked Conv+BN+ReLU network layers; The computing operation of each of the second convolution units includes a Conv+BN+ReLU+UpSample network layer, wherein each layer of the second convolution unit increases the resolution of the output second image features layer by layer through a bilinear interpolation upsampling operation.

5. The image-based sky area segmentation method according to claim 1, characterized in that: The step of obtaining a probability image corresponding to the original image according to the first feature image comprises: The first feature image is transformed using a Sigmoid function to obtain a probability image corresponding to the feature image; wherein the first feature image is a second feature image output by the second convolution unit of the last layer of the encoder module.

6. A method for processing image special effects, characterized in that: include: Get the target sky material selected by the user; Determine a sky area of ​​an original image to be segmented, wherein the sky area is determined by inputting the original image into a sky segmentation network, performing semantic segmentation processing on the original image, extracting first image features of the original image with different receptive fields using a first convolution unit of an encoder module of the sky segmentation network, inputting the first image features into a second convolution unit of a decoder module of the sky segmentation network for processing, restoring a first feature image consistent with the resolution of the input original image, and obtaining a probability image corresponding to the original image based on the first feature image; wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training a second feature image output by a second convolution unit of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image; Merging the target sky material into the sky area to generate a target image; The sky segmentation network is obtained based on the following method: Calling a Unet network model, compressing the number of basic channels of the Unet network model and increasing the number of layers of the Unet network model according to the characteristics of the sky segmentation scene, so that the feature image output by the modified number of basic channels and the number of layers is adapted to the characteristics of the sky segmentation scene, thereby obtaining the sky segmentation network; The convolution unit of the Unet network model is a separable convolution unit; The step of training the second feature image output by the second convolutional unit of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image comprises: Determine a resolution ratio of a second feature image output by the second convolutional units of at least two layers of the decoder module relative to the original image; determine the type and weight value of a BCE loss function for training the second convolutional units of the at least two layers according to the resolution ratio; for each layer of the second convolutional units, based on the corresponding weight value, and using the corresponding type of BCE loss function to constrain the sky segmentation network training; the types of the BCE loss function include 1 / 2BCE loss function, 1 / 4BCE loss function and 1 / 8BCE loss function; The step of determining a resolution ratio of a second feature image output by the second convolutional units of at least two layers of the decoder module relative to the original image comprises: Adjusting the feature channels of the second convolution units of at least two layers of the decoder module to output channels through 1×1 convolution, and outputting the corresponding second feature images; Based on the resolution of the second feature image, a resolution ratio of the second feature image to the original image is calculated.

7. An image-based sky area segmentation device, characterized in that: include: The original image acquisition module is used to acquire the original image containing sky content features; A probability image acquisition module, used to input the original image into a sky segmentation network, perform semantic segmentation processing on the original image, use the first convolution unit of the encoder module of the sky segmentation network to extract the first image features of the original image with different receptive fields, input the first image features into the second convolution unit of the decoder module of the sky segmentation network for processing, restore and obtain a first feature image consistent with the resolution of the input original image, and obtain a probability image corresponding to the original image according to the first feature image; wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training the second feature images output by the second convolution units of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image; A sky region segmentation module, used for segmenting the sky region of the original image according to the probability image; The sky segmentation network is obtained based on the following method: Calling a Unet network model, compressing the number of basic channels of the Unet network model and increasing the number of layers of the Unet network model according to the characteristics of the sky segmentation scene, so that the feature image output by the modified number of basic channels and the number of layers is adapted to the characteristics of the sky segmentation scene, thereby obtaining the sky segmentation network; The convolution unit of the Unet network model is a separable convolution unit; The probability image obtaining module trains the second feature images output by the second convolution units of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image, including: Determine a resolution ratio of a second feature image output by the second convolutional units of at least two layers of the decoder module relative to the original image; determine the type and weight value of a BCE loss function for training the second convolutional units of the at least two layers according to the resolution ratio; for each layer of the second convolutional units, based on the corresponding weight value, and using the corresponding type of BCE loss function to constrain the sky segmentation network training; the types of the BCE loss function include 1 / 2BCE loss function, 1 / 4BCE loss function and 1 / 8BCE loss function; The probability image obtaining module determines a resolution ratio of a second feature image output by the second convolutional units of at least two layers of the decoder module relative to the original image, including: Adjusting the feature channels of the second convolution units of at least two layers of the decoder module to output channels through 1×1 convolution, and outputting corresponding second feature images; Based on the resolution of the second feature image, a resolution ratio of the second feature image to the original image is calculated.

8. An image special effects processing device, characterized in that: include: A sky material acquisition module is used to acquire the target sky material selected by the user; A sky area determination module, used for determining the sky area of ​​an original image to be segmented, wherein the sky area is determined by inputting the original image into a sky segmentation network, performing semantic segmentation processing on the original image, extracting first image features of the original image with different receptive fields by using a first convolution unit of an encoder module of the sky segmentation network, inputting the first image features into a second convolution unit of a decoder module of the sky segmentation network for processing, restoring a first feature image consistent with the resolution of the input original image, and obtaining a probability image corresponding to the original image based on the first feature image; wherein the sky segmentation network is a lightweight deep convolutional neural network, and is obtained by training a second feature image output by a second convolution unit of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature image; A target image generation module, used for fusing the target sky material into the sky area to generate a target image; The sky segmentation network is obtained based on the following method: Calling a Unet network model, compressing the number of basic channels of the Unet network model and increasing the number of layers of the Unet network model according to the characteristics of the sky segmentation scene, so that the feature image output by the modified number of basic channels and the number of layers is adapted to the characteristics of the sky segmentation scene, thereby obtaining the sky segmentation network; The convolution unit of the Unet network model is a separable convolution unit; The sky area determination module trains the second feature images output by the second convolution units of at least two layers of the decoder module using a loss function corresponding to the resolution of the second feature images, including: Determine a resolution ratio of a second feature image output by the second convolutional units of at least two layers of the decoder module relative to the original image; determine the type and weight value of a BCE loss function for training the second convolutional units of the at least two layers according to the resolution ratio; for each layer of the second convolutional units, based on the corresponding weight value, and using the corresponding type of BCE loss function to constrain the sky segmentation network training; the types of the BCE loss function include 1 / 2BCE loss function, 1 / 4BCE loss function and 1 / 8BCE loss function; The sky area determination module determines a resolution ratio of a second feature image output by the second convolution units of at least two layers of the decoder module relative to the original image, including: Adjusting the feature channels of the second convolution units of at least two layers of the decoder module to output channels through 1×1 convolution, and outputting corresponding second feature images; Based on the resolution of the second feature image, a resolution ratio of the second feature image to the original image is calculated.

9. An electronic device, characterized in that: It includes: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the image-based sky area segmentation method according to any one of claims 1-5 or the image special effects processing method according to claim 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image-based sky area segmentation method described in any one of claims 1 to 5 or the image special effects processing method described in claim 6 is implemented.

Images