Image segmentation method, device, electronic device and storage medium

Abstract

The present disclosure relates to an image segmentation method, device, electronic device and storage medium. The method includes: obtaining a target video to be processed; when n is an integer from 0 to m, executing the following process: inputting the n*k+1th frame image of the target video into a first image segmentation model to obtain a segmented image of the n*k+1th frame image; inputting each frame image from the n*k+2th to (n+1)*kth frame image, together with the transmission information of the previous frame image adjacent to it, into a second image segmentation model to obtain a segmented image of each frame image from the n*k+2th to (n+1)*kth frame image, wherein the transmission information of a frame image includes at least part of the image features extracted from the segmented image of the frame image. The technical solution provided by the embodiment of the present disclosure can reduce the amount of calculation of real-time video segmentation and shorten the processing time, while also ensuring the accuracy of the video segmentation algorithm to a certain extent.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to an image segmentation method, device, electronic device and storage medium. Background Art

[0002] Object segmentation is a key technology in the field of computer vision and the basis of many other advanced algorithms. With the acceleration of mobile networks, the popularization of mobile devices and the popularity of short video apps in recent years, segmentation algorithms have been increasingly used in commercial applications, such as human body segmentation, clothing segmentation, hair segmentation, skin segmentation, etc. Due to the limitations of the computing power of mobile devices, higher requirements are placed on the running speed and accuracy of segmentation algorithms.

[0003] The deep learning segmentation network (Unet) algorithm has a network structure that looks like a U shape, so it is called Unet. Figure 1 As shown in the figure, this type of algorithm obtains a wider range of semantic information (the left half of the U-shape) by reducing the resolution of the feature image layer by layer, and then obtains more accurate edge information by increasing the resolution of the feature image (the right half of the U-shape) and using skip connections (indicated by the dotted arrows), and finally achieves a more refined semantic segmentation result.

[0004] However, the Unet algorithm is more suitable for semantic segmentation of single-frame images. When the application scenario is video, Unet needs to run the video frame by frame, which generates a large amount of calculation, causing lag when running on low-end electronic devices, thus failing to achieve the purpose of real-time operation. Summary of the invention

[0005] In order to solve the technical problems described in the background technology, the embodiments of the present disclosure provide an image segmentation method, device, electronic device and storage medium. The technical solutions of the present disclosure are as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an image segmentation method, comprising:

[0007] Get the target video to be processed;

[0008] When n is an integer from 0 to m, the following process is performed:

[0009] Inputting the n*k+1th frame image of the target video into the first image segmentation model to obtain a segmented image of the n*k+1th frame image; wherein m is a positive integer, k is a predetermined constant, and k is an integer greater than 1;

[0010] Input each frame image from the n*k+2th to (n+1)*kth frame image and the transfer information of the previous frame image adjacent to it into the second image segmentation model to obtain a segmented image of each frame image from the n*k+2th to (n+1)*kth frame image, wherein the transfer information of a frame image includes at least part of the image features extracted from the segmented image of the frame image;

[0011] The first image segmentation model is an image segmentation network with a number of layers greater than a first preset value, the second image segmentation model is an image segmentation network with a number of layers less than a second preset value, and the first preset value is greater than or equal to the second preset value.

[0012] Optionally, when n is any integer from 1 to m, inputting the n*k+1th frame image of the target video into the first image segmentation model includes:

[0013] The n*k+1th frame image and the transmission information of the adjacent previous frame image are input into the first image segmentation model.

[0014] Optionally, after obtaining the segmented image of the n*k+1th frame image, the method further includes:

[0015] According to the segmented image of the n*k+1th frame image, the transmission information of the n*k+1th frame image is obtained;

[0016] The step of obtaining the transmission information of the n*k+1th frame image according to the segmented image of the n*k+1th frame image includes:

[0017] Input the pixel value of each pixel in the segmented image of the n*k+1th frame image into the gating function to obtain the target parameter of each pixel in the segmented image of the n*k+1th frame image, wherein the target parameter is used to characterize the probability that the pixel belongs to the main object in the image;

[0018] According to the target parameter, at least part of the image features in the segmented image of the n*k+1th frame image are extracted and determined as the transfer information of the n*k+1th frame image.

[0019] Optionally, the output value of the gating function has a value range of [0, 1], and extracting at least part of the image features in the segmented image of the n*k+1th frame image according to the target parameter and determining it as the transfer information of the n*k+1th frame image includes:

[0020] The pixel value of each pixel point in the segmented image of the n*k+1th frame image is calculated, and the product of the pixel value and the target parameter of the pixel point is calculated to obtain the transmission information of the n*k+1th frame image.

[0021] Optionally, the gating function is g=1 / (1+exp(-x));

[0022] Wherein, x represents the pixel value of the pixel point in the segmented image, and g represents the target parameter.

[0023] Optionally, each of the n*k+2th to (n+1)*kth image frames and the transmission information of the previous adjacent image frame are input into the second image segmentation model, including:

[0024] The images of multiple channels included in each frame of the image from the n*k+2th to the (n+1)*kth frame are input into the second image segmentation model together with the transmission information of the adjacent previous frame of the image.

[0025] Optionally, the image segmentation network is a deep learning segmentation Unet network.

[0026] Optionally, the parameters of the second image segmentation model used to perform image segmentation on each frame image from the n*k+2th frame image to the (n+1)*kth frame image are the same.

[0027] According to a second aspect of an embodiment of the present disclosure, there is provided an image segmentation apparatus, comprising:

[0028] A video acquisition module is configured to acquire a target video to be processed;

[0029] The first processing module is configured to execute the following process when n is an integer from 0 to m:

[0030] Inputting the n*k+1th frame image of the target video into the first image segmentation model to obtain a segmented image of the n*k+1th frame image; wherein m is a positive integer, k is a predetermined constant, and k is an integer greater than 1;

[0031] Input each frame image from the n*k+2th to (n+1)*kth frame image and the transfer information of the previous frame image adjacent to it into the second image segmentation model to obtain a segmented image of each frame image from the n*k+2th to (n+1)*kth frame image, wherein the transfer information of a frame image includes at least part of the image features extracted from the segmented image of the frame image;

[0032] The first image segmentation model is an image segmentation network with a number of layers greater than a first preset value, the second image segmentation model is an image segmentation network with a number of layers less than a second preset value, and the first preset value is greater than or equal to the second preset value.

[0033] Optionally, when n is any integer from 1 to m, when the first processing module inputs the n*k+1th frame image of the target video into the first image segmentation model, the first processing module is specifically configured as follows:

[0034] The n*k+1th frame image and the transmission information of the adjacent previous frame image are input into the first image segmentation model.

[0035] Optionally, also include:

[0036] A transfer information acquisition module is configured to acquire transfer information of the n*k+1th frame image according to the segmented image of the n*k+1th frame image;

[0037] Wherein, the transmission information acquisition module includes:

[0038] The parameter determination submodule is configured to input the pixel value of each pixel point in the segmented image of the n*k+1th frame image into the gating function to obtain the target parameter of each pixel point in the segmented image of the n*k+1th frame image, wherein the target parameter is used to characterize the probability that the pixel point belongs to the main object in the image;

[0039] The information extraction submodule is configured to extract at least part of the image features in the segmented image of the n*k+1th frame image according to the target parameter, and determine them as the transmission information of the n*k+1th frame image.

[0040] Optionally, the output value of the gating function has a value range of [0, 1], and the information extraction submodule is specifically configured as follows:

[0041] The pixel value of each pixel point in the segmented image of the n*k+1th frame image is calculated, and the product of the pixel value and the target parameter of the pixel point is calculated to obtain the transmission information of the n*k+1th frame image.

[0042] Optionally, the gating function is g=1 / (1+exp(-x));

[0043] Wherein, x represents the pixel value of the pixel point in the segmented image, and g represents the target parameter.

[0044] Optionally, when the first processing module inputs each frame image from the n*k+2th to (n+1)*kth frame image and the transmission information of the adjacent previous frame image into the second image segmentation model, the first processing module is specifically configured as follows:

[0045] The images of multiple channels included in each frame of the image from the n*k+2th to the (n+1)*kth frame are input into the second image segmentation model together with the transmission information of the adjacent previous frame of the image.

[0046] Optionally, the image segmentation network is a deep learning segmentation Unet network.

[0047] Optionally, the parameters of the second image segmentation model used to perform image segmentation on each frame image from the n*k+2th frame image to the (n+1)*kth frame image are the same.

[0048] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0049] processor;

[0050] a memory for storing instructions executable by the processor;

[0051] Wherein, the processor is configured to execute the instructions to implement the above-mentioned image segmentation method.

[0052] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned image segmentation method.

[0053] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program product comprising instructions, which, when executed on a computer, enables the computer to implement the above-mentioned image segmentation method.

[0054] The technical solution provided by the embodiment of the present disclosure obtains the target video to be processed, and then performs the following process when n is an integer from 0 to m: the n*k+1th frame image of the target video is input into the first image segmentation model to obtain the segmented image of the n*k+1th frame image; each frame image from the n*k+2th to (n+1)*kth frame image is input into the second image segmentation model together with the transmission information of the previous frame image adjacent to it, to obtain the segmented image of each frame image from the n*k+2th to (n+1)*kth frame image. Wherein, the transmission information of a frame image includes at least part of the image features extracted from the segmented image of the frame image, m is a positive integer, k is a predetermined constant, and k is an integer greater than 1; the first image segmentation model is an image segmentation network with a number of layers greater than a first preset value, and the second image segmentation model is an image segmentation network with a number of layers less than a second preset value, and the first preset value is greater than or equal to the second preset value.

[0055] It can be seen that in the technical solution provided by the embodiment of the present disclosure, an image segmentation network with a larger number of layers is used as the first image segmentation model, and an image segmentation network with a smaller number of layers is used as the second image segmentation model. The first image segmentation model is used to perform image segmentation every preset number (i.e., k) of frame images, and the second image segmentation model is used to perform image segmentation on other frame images between the frame images processed by the first image segmentation model. At the same time, when the second image segmentation model is used for processing, the image to be processed and the transmission information of its previous frame image are input into the second image segmentation model for image segmentation.

[0056] Among them, if the first image segmentation model has more layers, then using the first image segmentation model for image segmentation can obtain more image information, so that the image segmentation result is more accurate. If the second image segmentation model has fewer layers, then using the second image segmentation model for image segmentation can generate less calculation, and the first image segmentation model is used for processing once every certain number of frame images, and the other frame images are processed by the second image segmentation model, so that the process of real-time image segmentation of a video is faster, that is, it takes less time. At the same time, when the second image segmentation model is used for processing, the image to be processed and the transmission information of the previous frame image are input into the second model for image segmentation, which can prevent large deviations from being generated when the second image segmentation model is used for image segmentation.

[0057] Therefore, the technical solution provided by the embodiments of the present disclosure can reduce the amount of real-time video segmentation calculations and shorten the processing time, while also ensuring the accuracy of the video segmentation algorithm to a certain extent, so that the image segmentation results will not produce large deviations, and will not cause lag when performing image segmentation on videos in mid- and low-end electronic devices, thereby achieving the purpose of real-time image segmentation.

[0058] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a result schematic diagram of the Unet network structure in the prior art;

[0060] Figure 2 is a flow chart of an image segmentation method according to an exemplary embodiment;

[0061] Figure 3 is a schematic diagram showing the principle of performing image segmentation on each frame image of a target video according to an exemplary embodiment;

[0062] Figure 4 is a flow chart of another image segmentation method according to an exemplary embodiment;

[0063] Figure 5 is a block diagram of an image segmentation device according to an exemplary embodiment;

[0064] Figure 6 is a block diagram of an electronic device according to an exemplary embodiment;

[0065] Figure 7 is a block diagram of another electronic device according to an exemplary embodiment;

[0066] Figure 8 This is one of the comparison results after using GConv and Unet to segment each frame of the same video;

[0067] Fig. 9 This is the second comparison of the results of using GConv and Unet to segment each frame of the same video;

[0068] Fig.10 This is the third comparison chart of the results after using GConv and Unet to segment each frame of the same video. DETAILED DESCRIPTION

[0069] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0070] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0071] In order to solve the technical problem in the related art that when using the Unet network structure to perform real-time image segmentation on videos, there is a large amount of calculation, which causes lag when running on low-end and mid-end electronic devices and cannot achieve real-time operation, the embodiments of the present disclosure provide an image segmentation method, device, electronic device and storage medium.

[0072] According to a first aspect of an embodiment of the present disclosure, there is provided an image segmentation method, such as Figure 2 As shown, the image segmentation method may include the following steps:

[0073] Step 201: Obtain a target video to be processed.

[0074] The target video may be a video clip or a complete video file. The target video may be a video clip or a complete video file stored locally on an electronic device, or a video clip or a complete video file downloaded in real time from a network by the electronic device. The image segmentation method provided in the embodiment of the present disclosure can perform image segmentation in the process of the electronic device downloading the target video in real time, that is, the electronic device downloads a part of the video clip, and then uses the image segmentation method provided in the embodiment of the present disclosure to process a part of the video clip in real time.

[0075] In addition, after acquiring the target video, when n is an integer from 0 to m, the following steps 202 to 203 are performed.

[0076] Step 202: input the n*k+1th frame image of the target video into a first image segmentation model to obtain a segmented image of the n*k+1th frame image.

[0077] Wherein, m is a positive integer, k is a predetermined constant, and k is an integer greater than 1.

[0078] That is, in the image segmentation method provided by the embodiment of the present disclosure, when n is 0, the first frame image is processed by the first image segmentation model; when n is 1, the k+1th frame image is processed by the first image segmentation model; when n is 2, the 2k+1th frame image is processed by the first image segmentation model… when n is m, the m*k+1th frame image is processed by the first image segmentation model, wherein m*k+1 is less than the total number of video frames included in the target video.

[0079] It can be seen that in the image segmentation method provided by the embodiment of the present disclosure, the first image segmentation model is used to perform image segmentation on the corresponding image every k frames of images.

[0080] In addition, k is an integer greater than 1, indicating that the number of frames of images between two adjacent images segmented by the first image segmentation model is greater than 1. That is, in the image segmentation method provided by the embodiment of the present disclosure, there is no situation where every frame of the target video is segmented by the first image segmentation model.

[0081] Step 203: Input each frame image from the n*k+2th to (n+1)*kth frame image and the transmission information of its adjacent previous frame image into the second image segmentation model to obtain a segmented image of each frame image from the n*k+2th to (n+1)*kth frame image.

[0082] The information transmitted by a frame of image includes at least part of the image features extracted from the segmented image of the frame of image. The image features may refer to pixel values. The pixel value refers to the value assigned by a computer when an image is digitized, which represents the average brightness information of a small square of the original image, or the average reflection (transmission) density information of the small square.

[0083] It can be seen from step 203 that in the image segmentation method provided by the embodiment of the present disclosure, the process of performing image segmentation on the n*k+2th to (n+1)*kth frame images is as follows:

[0084] Extracting at least part of the image features in the segmented image of the n*k+1th frame image, and determining the features as the transfer information of the n*k+1th frame image;

[0085] Input the transmission information of the n*k+1th frame image and the n*k+2th frame image into the second image segmentation model to obtain the segmented image of the n*k+2th frame image;

[0086] Extracting at least part of the image features in the segmented image of the n*k+2th frame image, and determining the features as the transfer information of the n*k+2th frame image;

[0087] The transmission information of the n*k+2th frame image and the n*k+3th frame image are input into the second image segmentation model together to obtain the segmented image of the n*k+3th frame image;

[0088] Referring to the above process, and in the order of increasing frames of the target video, image segmentation is performed on each frame after the n*k+3th frame until the image segmentation of the (n+1)*kth frame is completed.

[0089] As can be seen from the above, the n*k+1th frame image and the n*k+2th frame image are two adjacent frames of images. In the image segmentation method provided by the embodiment of the present disclosure, at least part of the image features in the segmented image of the n*k+1th frame image will be used as the transmission information and input into the second image segmentation model together with the n*k+2th frame image to perform image segmentation. Therefore, the segmented image of the n*k+2th frame image is obtained after performing image segmentation on the n*k+2th frame image based on the partial features of the segmented image of the n*k+1th frame image as a reference. Among them, the partial features of the segmented image of the n*k+1th frame image are used as a reference for image segmentation of the n*k+2th frame image, so as to avoid large deviations when performing image segmentation on the n*k+2th frame image, that is, to a certain extent, the accuracy of image segmentation is guaranteed.

[0090] In addition, in the image segmentation method provided by the embodiment of the present disclosure, after obtaining the segmented image of the n*k+2th frame image, at least part of the image features in the segmented image of the n*k+2th frame image will be extracted and determined as the transfer information of the n*k+2th frame image; then the transfer information of the n*k+2th frame image and the n*k+3th frame image are input into the corresponding model for processing to obtain the segmented image of the n*k+3th frame image.

[0091] It can be seen that when the second image segmentation model is used for processing, the image to be processed and the transmission information of its previous frame image are input into the second model for image segmentation, which can prevent large deviations from occurring when the second image segmentation model is used for image segmentation.

[0092] In addition, in the image segmentation method provided by the embodiment of the present disclosure, the first image segmentation model is an image segmentation network with a number of layers greater than a first preset value, and the second image segmentation model is an image segmentation network with a number of layers less than a second preset value, and the first preset value is greater than or equal to the second preset value.

[0093] From the above, it can be seen that in the image segmentation method provided by the embodiment of the present disclosure, an image segmentation network with a larger number of layers is used as the first image segmentation model, and an image segmentation network with a smaller number of layers is used as the second image segmentation model. The first image segmentation model is used to perform image segmentation every preset number (i.e., k) of frame images, and the second image segmentation model is used to perform image segmentation on other frame images between the frame images processed by the first image segmentation model. At the same time, when the second image segmentation model is used for processing, the image to be processed and the transmission information of its previous frame image are input into the second model for image segmentation.

[0094] Among them, if the first image segmentation model has more layers, then using the first image segmentation model for image segmentation can obtain more image information, thereby making the image segmentation result more accurate. If the second image segmentation model has fewer layers, then using the second image segmentation model for image segmentation can generate less calculation amount. The first image segmentation model is only used once every certain number of frame images, and the other frame images are processed using the second image segmentation model, which makes the process of real-time image segmentation of a video faster, that is, it takes less time.

[0095] That is, when performing image segmentation on the target video, the first image segmentation model is run once every a certain number of frame images, and the second image segmentation model is run at other times, so the processing time of the entire video mainly depends on the size of the second image segmentation model, wherein the computational complexity of the second image segmentation model is usually only 1 / h of the first image segmentation model (h can be changed by setting the number of layers of the second image segmentation model). Therefore, the image segmentation method provided in the embodiment of the present disclosure uses the above-mentioned method to run the first image segmentation model and the second image segmentation model, so that the process of real-time image segmentation of a video is faster, that is, it takes less time.

[0096] At the same time, when the second image segmentation model is used for processing, the image to be processed and the transmission information of its previous frame image are input into the second image segmentation model for image segmentation, which can prevent large deviations from occurring when the second image segmentation model is used for image segmentation.

[0097] Therefore, the image segmentation method provided by the embodiment of the present disclosure can reduce the amount of real-time video segmentation calculations and shorten the processing time, while also ensuring the accuracy of the video segmentation algorithm to a certain extent, so that the image segmentation results will not produce large deviations, and mid- and low-end electronic devices will not produce lags when performing image segmentation on videos, thereby achieving the purpose of real-time image segmentation.

[0098] According to a second aspect of the embodiment of the present disclosure, another image segmentation method is provided, such as Figure 4 As shown, the image segmentation method may include the following steps:

[0099] Step 401: Obtain a target video to be processed.

[0100] The target video may be a video clip or a complete video file. The target video may be a video clip or a complete video file stored locally on an electronic device, or a video clip or a complete video file downloaded in real time from a network side by the electronic device. The image segmentation method provided by the embodiment of the present disclosure can perform image segmentation in the process of the electronic device downloading the target video in real time, that is, a portion of the video clip is downloaded electronically, and the image segmentation method provided by the embodiment of the present disclosure is used to process a portion of the video clip in real time.

[0101] In addition, after acquiring the target video, when n is an integer from 0 to m, the process described in steps 402 to 403 is performed.

[0102] Step 402: When n is 0, the n*k+1th frame image is input into the first image segmentation model. When n is an integer from 1 to m, the n*k+1th frame image and the transmission information of its adjacent previous frame image are input into the first image segmentation model together to obtain a segmented image of the n*k+1th frame image.

[0103] Wherein, m is a positive integer, k is a predetermined constant, and k is an integer greater than 1.

[0104] That is, in the image segmentation method provided by the embodiment of the present disclosure, when n is 0, the first frame image is processed by the first image segmentation model; when n is 1, the k+1th frame image is processed by the first image segmentation model; when n is 2, the 2k+1th frame image is processed by the first image segmentation model… when n is m, the m*k+1th frame image is processed by the first image segmentation model, wherein m*k+1 is less than the total number of image frames of the target video (that is, the total number of video frames included in the target video).

[0105] It can be seen that in the image segmentation method provided by the embodiment of the present disclosure, the first image segmentation model is used to perform image segmentation on the corresponding image every k frames of images.

[0106] In addition, k being an integer greater than 1 indicates that two images are segmented using the first image segmentation model, and the number of frames between adjacent images is greater than 1. That is, in the image segmentation method provided in the embodiment of the present disclosure, there is no situation where each frame of the target video is segmented using the first image segmentation model.

[0107] In addition, in the image segmentation method provided by the embodiment of the present disclosure, for an image processed using the first image segmentation model, it can also be input into the first image segmentation model together with its adjacent previous frame image for processing, which can further improve the accuracy of image segmentation using the first image segmentation model.

[0108] Step 403: Input each frame image from the n*k+2th to (n+1)*kth frame image and the transmission information of its adjacent previous frame image into the second image segmentation model to obtain a segmented image of each frame image from the n*k+2th to (n+1)*kth frame image.

[0109] The information transmitted by a frame of image includes at least part of the image features extracted from the segmented image of the frame of image. The image features may refer to pixel values. The pixel value refers to the value assigned by a computer when an image is digitized, which represents the average brightness information of a small square of the original image, or the average reflection (transmission) density information of the small square.

[0110] It can be seen from step 403 that in the image segmentation method provided by the embodiment of the present disclosure, the process of performing image segmentation on the n*k+2th to (n+1)*kth frame images is as follows:

[0111] Extracting at least part of the image features in the segmented image of the n*k+1th frame image, and determining the features as the transfer information of the n*k+1th frame image;

[0112] Input the transmission information of the n*k+1th frame image and the n*k+2th frame image into the second image segmentation model to obtain the segmented image of the n*k+2th frame image;

[0113] Extracting at least part of the image features in the segmented image of the n*k+2th frame image, and determining the features as the transfer information of the n*k+2th frame image;

[0114] The transmission information of the n*k+2th frame image and the n*k+3th frame image are input into the second image segmentation model together to obtain the segmented image of the n*k+3th frame image;

[0115] Referring to the above process, and in the order of increasing frames of the target video, image segmentation is performed on each frame after the n*k+3th frame until the image segmentation of the (n+1)*kth frame is completed.

[0116] It can be seen that when the second image segmentation model is used for processing, the image to be processed and the transmission information of its previous frame image are input into the second model for image segmentation, which can prevent large deviations from occurring when the second image segmentation model is used for image segmentation.

[0117] In addition, in the image segmentation method provided by the embodiment of the present disclosure, the first image segmentation model is an image segmentation network with a number of layers greater than a first preset value, and the second image segmentation model is an image segmentation network with a number of layers less than a second preset value, and the first preset value is greater than or equal to the second preset value.

[0118] From the above, it can be seen that in the image segmentation method provided by the embodiment of the present disclosure, an image segmentation network with a larger number of layers is used as the first image segmentation model, and an image segmentation network with a smaller number of layers is used as the second image segmentation model. The first image segmentation model is used to perform image segmentation every preset number (i.e., k) of frame images, and the second image segmentation model is used to perform image segmentation on other frame images between the frame images processed by the first image segmentation model. At the same time, when the second image segmentation model is used for processing, the image to be processed and the transmission information of its previous frame image are input into the second model for image segmentation.

[0119] Among them, if the first image segmentation model has more layers, then using the first image segmentation model for image segmentation can obtain more image information, thereby making the image segmentation result more accurate. If the second image segmentation model has fewer layers, then using the second image segmentation model for image segmentation can generate less calculation amount. The first image segmentation model is only used once every certain number of frame images, and the other frame images are processed using the second image segmentation model, which makes the process of real-time image segmentation of a video faster, that is, it takes less time.

[0120] At the same time, when the second image segmentation model is used for processing, the image to be processed and the transmission information of its previous frame image are input into the second image segmentation model for image segmentation, which can prevent large deviations from occurring when the second image segmentation model is used for image segmentation.

[0121] Therefore, the image segmentation method provided by the embodiment of the present disclosure can reduce the amount of real-time video segmentation calculations and shorten the processing time, while also ensuring the accuracy of the video segmentation algorithm to a certain extent, so that the image segmentation results will not produce large deviations, and mid- and low-end electronic devices will not produce lags when performing image segmentation on videos, thereby achieving the purpose of real-time image segmentation.

[0122] In addition, when the value of n is 0, the first frame image is processed by the first image segmentation model to obtain a segmented image of the first frame image, so that at least part of the image features in the segmented image of the first frame image are input into the second image segmentation model together with the second frame image to obtain a segmented image of the second frame image;

[0123] When the value of n is 1, the k+1th frame image is processed by the first image segmentation model to obtain a segmented image of the k+1th frame image, so that at least part of the image features in the segmented image of the k+1th frame image are input into the second image segmentation model together with the k+2th frame image to obtain a segmented image of the k+2th frame image;

[0124] When the value of n is 2, the 2k+1th frame image is processed by the first image segmentation model to obtain the segmented image of the 2k+1th frame image, thereby inputting at least part of the image features in the segmented image of the 2k+1th frame image together with the 2k+2th frame image into the second image segmentation model to obtain the segmented image of the 2k+2th frame image…When the value of n is m, the m*k+1th frame image is processed by the first image segmentation model to obtain the segmented image of the m*k+1th frame image, thereby inputting at least part of the image features in the segmented image of the m*k+1th frame image together with the m*k+2th frame image into the second image segmentation model to obtain the segmented image of the m*k+2th frame image.

[0125] It can be seen that in the image segmentation method provided by the embodiment of the present disclosure, after performing image segmentation on a certain frame image using the first image segmentation model, at least part of the image features in the obtained segmented image can be transferred to the next frame image, that is, used as a reference for performing image segmentation on the next frame image using the second image segmentation model, thereby further improving the accuracy of image segmentation.

[0126] In summary, if the value of k is 24, the first image segmentation model is used to segment the image of the corresponding frame of the target video every 24 frames, and the second image segmentation model is used to segment the images of other frames. Specifically, when the value of k is 24, the image segmentation method provided by the embodiment of the present disclosure is used to segment the target video. The specific process can be as follows: Figure 3 shown.

[0127] That is, the first frame image is input into the first image segmentation model 301 for image segmentation to obtain a segmented image 31; then, at least part of the image features in the segmented image 31 are extracted as transfer information, and the image is input into the second image segmentation model 302 together with the second frame image for image segmentation to obtain a segmented image 32;

[0128] Then, at least part of the image features in the segmented image 32 are extracted as the transmission information, and are inputted into the second image segmentation model 302 together with the third frame image for image segmentation. In this way, the second image segmentation model 302 is used to perform image segmentation on the fourth to twenty-fourth frames of image in sequence, and a segmented image of the twenty-fourth frame of image is obtained;

[0129] Secondly, extract at least part of the image features in the segmented image of the 24th frame image as the transfer information, and input the 25th frame image into the first image segmentation model 301 for image segmentation to obtain the segmented image 325;

[0130] Thirdly, at least part of the image features in the segmented image 325 are extracted as the transfer information, and the 26th frame image is input into the second image segmentation model 302 for image segmentation to obtain a segmented image 326;

[0131] Thereafter, each subsequent frame image of the target video may be processed in sequence in the same manner as above.

[0132] Optionally, after obtaining the segmented image of the n*k+1th frame image, the method further includes:

[0133] According to the segmented image of the n*k+1th frame image, the transmission information of the n*k+1th frame image is obtained;

[0134] The step of obtaining the transmission information of the n*k+1th frame image according to the segmented image of the n*k+1th frame image includes:

[0135] Input the pixel value of each pixel in the segmented image of the n*k+1th frame image into the gating function to obtain the target parameter of each pixel in the segmented image of the n*k+1th frame image, wherein the target parameter is used to characterize the probability that the pixel belongs to the main object in the image;

[0136] According to the target parameter, at least part of the image features in the segmented image of the n*k+1th frame image are extracted and determined as the transfer information of the n*k+1th frame image.

[0137] For example, if the resolution of the segmented image of the n*k+1th frame image is 572*572, then the segmented image includes 572*572 pixels, and each pixel has a pixel value. In the image segmentation method provided by the embodiment of the present disclosure, the pixel value of each pixel in the segmented image needs to be input into the gating function to obtain the target parameter corresponding to each pixel, and then the pixel values ​​of which pixels in the segmented image can be extracted as the transmission information can be determined according to the target parameter corresponding to each pixel.

[0138] Optionally, the output value of the gating function has a value range of [0, 1], and according to the target parameter, at least part of the image features in the segmented image of the n*k+1th frame image are extracted and determined as the transfer information of the n*k+1th frame image, including:

[0139] The pixel value of each pixel point in the segmented image of the n*k+1th frame image is calculated, and the product of the pixel value and the target parameter of the pixel point is calculated to obtain the transmission information of the n*k+1th frame image.

[0140] It can be understood that the process of obtaining the transmission information of each frame image in the target video except the n*k+1th frame image is the same as the process of obtaining the transmission information of the n*k+1th frame image mentioned above, and will not be repeated here.

[0141] It can be seen from this that the image segmentation method provided by the embodiment of the present disclosure uses the product of the pixel value of the pixel point and the target parameter as the transmission information of the segmented image.

[0142] Among them, the target parameter is equal to 1, which means that the pixel point corresponding to the target parameter is the pixel point of the main object in the segmented image where it is located. Then the pixel value of the pixel point corresponding to the target parameter is the necessary information of the segmented image and needs to be passed to the next frame image.

[0143] If the target parameter is greater than 0 and less than 1, it means that the pixel corresponding to the target parameter has a high probability of belonging to the pixel of the main object in the segmented image. In this case, some features of the pixel are also necessary information for the segmented image and need to be passed to the next frame image.

[0144] The target parameter is equal to 0, which means that the pixel point corresponding to the target parameter is not the pixel point of the main object in the segmented image where it is located. The pixel value of the pixel point corresponding to the target parameter is not necessary information for segmenting the image and does not need to be passed to the next frame image.

[0145] Therefore, through the gating function, the necessary information in the segmented image of one frame can be passed to the next frame, while the unnecessary information is not passed to the next frame, thereby reducing a certain amount of calculation while ensuring the accuracy of image segmentation for the next frame to a certain extent.

[0146] Optionally, the gating function is g=1 / (1+exp(-x)); wherein x represents the pixel value of a pixel point in the segmented image, and g represents the target parameter.

[0147] Among them, the value range of g is [0, 1]. When g is 0, it means that the pixel value of the corresponding pixel point is completely useless and will not be passed to the next frame image; when g is 1, it means that the pixel value of the corresponding pixel point is very useful and will be passed to the next frame image; when g takes an intermediate value between 0 and 1, it means that part of the features of the pixel value of the corresponding pixel point are passed to the next frame image.

[0148] In addition, taking human body segmentation as an example, an ideal gating unit is to pass human body related features to the next frame and shut out background related features.

[0149] Optionally, each of the n*k+2th to (n+1)*kth image frames and the transmission information of the previous adjacent image frame are input into the second image segmentation model, including:

[0150] The images of multiple channels included in each frame of the image from the n*k+2th to the (n+1)*kth frame are input into the second image segmentation model together with the transmission information of the adjacent previous frame of the image.

[0151] For example, the transmission information of the n*k+1th frame image and the n*k+2th frame image are input together into the second image segmentation model for processing, including: the transmission information of the n*k+1th frame image and the images of multiple channels included in the n*k+2th frame image are input together into the second image segmentation model for processing.

[0152] Specifically, for example, the n*k+2th frame image includes an image of the R channel, an image of the G channel, and an image of the B channel. When the n*k+2th frame image is segmented, the images of these three channels and the transmission information of the n*k+1th frame image are input into the second image segmentation model for processing, so that when the n*k+2th frame image is segmented, the transmission information of the n*k+1th frame image can be referred to, thereby improving the accuracy of image segmentation for the n*k+2th frame image.

[0153] Optionally, the image segmentation network is a deep learning segmentation Unet network. That is, the first image segmentation model (also referred to as the teacher model) may be a Unet network with a number of layers greater than a first preset value, and the second image segmentation model (also referred to as the student model) may be a Unet network with a number of layers less than a second preset value.

[0154] It can be seen that the teacher model and the student model constitute the image segmentation model used in the image segmentation method provided in the embodiment of the present disclosure, and the model can be called a gated convolutional network (GConv). The structure of the teacher model and the student model is basically similar to the Unet structure. The specific structure can be referred to Figure 1 The structural diagram of Unet in , their input is a single frame image, and the output is a segmentation mask; the difference is that the teacher model has more parameters and a larger amount of calculation, while the student model contains much smaller parameters, the number of network layers is much smaller, and the amount of calculation is also much smaller. Therefore, the student model is similar to a mini version of the Unet network.

[0155] in, Figure 1 The rectangle enclosed by a dotted ellipse in the figure constitutes a CNN (Convolutional Neural Network), that is, one of the rectangles in a dotted ellipse represents a layer of a CNN network. Therefore, a Unet includes multiple CNN networks. In the image segmentation method provided in the embodiment of the present disclosure, both the teacher model and the student model are Unet, but the specific number of layers and parameters included are different, so both the teacher model and the student model include multiple CNN networks. Therefore, the GConv used in the image segmentation method provided in the embodiment of the present disclosure is based on the CNN unit.

[0156] Optionally, the parameters of the second image segmentation model used to perform image segmentation on each frame image from the n*k+2th frame image to the (n+1)*kth frame image are the same.

[0157] Among them, the image segmentation method provided by the embodiment of the present disclosure adopts the first image segmentation model (i.e., teacher model) to perform image segmentation every preset number (i.e., k) of frame images, and adopts the second image segmentation model (i.e., student model) to perform image segmentation for other frame images between the frame images processed by the first image segmentation model. That is, for all the frame images included in the target video, each adjacent k+1 frame image can be divided into a group, the first frame in each group adopts the teacher model to perform image segmentation, and the other frame images in the group adopt the student model to perform image processing, wherein, when each frame image in a group is processed by the student model for image segmentation, the parameters of the student model are the same (i.e., the parameters are shared between the student models applied when performing image segmentation in a group), thereby reducing the number of parameters.

[0158] Therefore, a teacher model and k student models can be grouped together, and then this group of models can be used cyclically when performing image segmentation on the target video. Therefore, the GConv used in the image segmentation method provided in the embodiment of the present disclosure belongs to RNN (Recurrent Neural Network, cyclic network structure).

[0159] In summary, the GConv used in the image segmentation method provided in the embodiment of the present disclosure is a gated RNN model based on the CNN unit. It can be seen that GConv is a recurrent network structure based on convolution, which is a variant of the RNN network in the field of images, so it has the advantages of both CNN and RNN, and can make full use of the continuity information of the image in time. Since GConv can extract more information, it only needs less calculation to achieve a segmentation effect similar to Unet, which is especially suitable for video applications on mid- and low-end electronic devices.

[0160] The results of image segmentation of each frame of the same video using the above GConv and Unet are compared, as shown in the figure. Figures 8 to 10 As shown, compared with Unet, GConv has a higher segmentation accuracy when used to segment each frame of the video.

[0161] In addition, it should be noted that the image segmentation method provided in the embodiments of the present disclosure can be applied to video projects, such as various magic expressions based on segmentation, barrage based on human body segmentation, face changing algorithms based on face segmentation, etc.

[0162] According to a third aspect of the embodiments of the present disclosure, there is provided an image segmentation device, such as Figure 5 As shown, the image segmentation device 50 includes:

[0163] The video acquisition module 501 is configured to acquire a target video to be processed;

[0164] The first processing module 502 is configured to execute the following process when n is an integer from 0 to m:

[0165] Inputting the n*k+1th frame image of the target video into the first image segmentation model to obtain a segmented image of the n*k+1th frame image; wherein m is a positive integer, k is a predetermined constant, and k is an integer greater than 1;

[0166] Input each frame image from the n*k+2th to (n+1)*kth frame image and the transfer information of the previous frame image adjacent to it into the second image segmentation model to obtain a segmented image of each frame image from the n*k+2th to (n+1)*kth frame image, wherein the transfer information of a frame image includes at least part of the image features extracted from the segmented image of the frame image;

[0167] The first image segmentation model is an image segmentation network with a number of layers greater than a first preset value, the second image segmentation model is an image segmentation network with a number of layers less than a second preset value, and the first preset value is greater than or equal to the second preset value.

[0168] Optionally, the device further comprises:

[0169] The transfer information acquisition module 503 is configured to acquire the transfer information of the n*k+1th frame image according to the segmented image of the n*k+1th frame image;

[0170] Wherein, the transmission information acquisition module 503 includes:

[0171] The parameter determination submodule 5031 is configured to input the pixel value of each pixel point in the segmented image of the n*k+1th frame image into the gating function to obtain the target parameter of each pixel point in the segmented image of the n*k+1th frame image, wherein the target parameter is used to represent the probability that the pixel point belongs to the main object in the image;

[0172] The information extraction submodule 5032 is configured to extract at least part of the image features in the segmented image of the n*k+1th frame image according to the target parameter, and determine them as the transfer information of the n*k+1th frame image.

[0173] Optionally, the output value of the gating function has a value range of [0, 1], and the information extraction submodule 5032 is specifically configured as follows:

[0174] The pixel value of each pixel point in the segmented image of the n*k+1th frame image is calculated, and the product of the pixel value and the target parameter of the pixel point is calculated to obtain the transmission information of the n*k+1th frame image.

[0175] Optionally, the gating function is g=1 / (1+exp(-x));

[0176] Wherein, x represents the pixel value of the pixel point in the segmented image, and g represents the target parameter.

[0177] Optionally, when the first processing module 502 inputs each frame image from the n*k+2th to the (n+1)*kth frame image and the transmission information of the previous frame image adjacent thereto into the second image segmentation model, the first processing module 502 is specifically configured as follows:

[0178] The images of multiple channels included in each frame of the image from the n*k+2th to the (n+1)*kth frame are input into the second image segmentation model together with the transmission information of the adjacent previous frame of the image.

[0179] Optionally, the image segmentation network is a deep learning segmentation Unet network.

[0180] Optionally, the parameters of the second image segmentation model used to perform image segmentation on each frame image from the n*k+2th frame image to the (n+1)*kth frame image are the same.

[0181] It can be seen that the image segmentation device provided by the embodiment of the present disclosure adopts an image segmentation network with a larger number of layers as the first image segmentation model, and adopts an image segmentation network with a smaller number of layers as the second image segmentation model, and adopts the first image segmentation model to perform image segmentation every preset number (i.e., k) of frame images, while the other frame images between the frame images processed by the first image segmentation model are processed by the second image segmentation model. At the same time, when the second image segmentation model is used for processing, the image to be processed and the transmission information of its previous frame image are input into the second image segmentation model for image segmentation.

[0182] Among them, if the first image segmentation model has more layers, then using the first image segmentation model for image segmentation can obtain more image information, so that the image segmentation result is more accurate. If the second image segmentation model has fewer layers, then using the second image segmentation model for image segmentation can generate less calculation, and the first image segmentation model is used for processing once every certain number of frame images, and the other frame images are processed by the second image segmentation model, so that the process of real-time image segmentation of a video is faster, that is, it takes less time. At the same time, when the second image segmentation model is used for processing, the image to be processed and the transmission information of the previous frame image are input into the second model for image segmentation, which can prevent large deviations from being generated when the second image segmentation model is used for image segmentation.

[0183] Therefore, the image segmentation device provided by the embodiment of the present disclosure can reduce the amount of real-time video segmentation calculations and shorten the processing time, while also ensuring the accuracy of the video segmentation algorithm to a certain extent, so that the image segmentation results will not produce large deviations, and will not cause lag when low- and medium-end electronic devices perform image segmentation on the video, thereby achieving the purpose of real-time image segmentation.

[0184] Figure 6 is a block diagram of an electronic device according to an exemplary embodiment. Figure 6 , the terminal comprises:

[0185] Processor 610;

[0186] A memory 60 for storing instructions executable by the processor;

[0187] Wherein, the processor is configured to execute the instructions to implement the above-mentioned image segmentation method.

[0188] Figure 7 is a block diagram of another electronic device 700 according to an exemplary embodiment. For example, the device 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0189] Reference Figure 7 , the device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0190] The processing component 702 generally controls the overall operation of the device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0191] The memory 704 is configured to store various types of data to support operations on the electronic device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0192] The power supply component 706 provides power to the various components of the device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 700.

[0193] The multimedia component 708 includes a screen that provides an output interface between the device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0194] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the device 700 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0195] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0196] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the device 700. For example, the sensor assembly 714 can detect the open / closed state of the device 700, the relative positioning of components, such as the display and keypad of the device 700, and the sensor assembly 714 can also detect the position change of the device 700 or a component of the device 700, the presence or absence of user contact with the device 700, the orientation or acceleration / deceleration of the device 700, and the temperature change of the device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0197] The communication component 716 is configured to facilitate wired or wireless communication between the device 700 and other devices. The device 700 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 7G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0198] In an exemplary embodiment, the apparatus 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the image segmentation method described above.

[0199] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 707 including instructions, and the instructions can be executed by the processor 720 of the device 700 to complete the above method. Optionally, for example, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0200] In another aspect of the implementation of the present disclosure, an embodiment of the present disclosure further provides a storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above-mentioned image segmentation method.

[0201] According to another aspect of the embodiments of the present disclosure, a computer program product including instructions is provided, and when the computer program product is executed on a computer, the computer implements the above-mentioned image segmentation method.

[0202] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0203] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image segmentation method, characterized in that: include: Get the target video to be processed; When n is an integer from 0 to m, the following process is performed: Input the n*k+1th frame image of the target video into the first image segmentation model to obtain a segmented image of the n*k+1th frame image, wherein m is a positive integer, k is a predetermined constant, and k is an integer greater than 1; input each frame image from the n*k+2th to (n+1)*kth frame image, together with the transfer information of the previous frame image adjacent to it, into the second image segmentation model to obtain a segmented image of each frame image from the n*k+2th to (n+1)*kth frame image, wherein the transfer information of a frame image includes at least part of the image features extracted from the segmented image of the frame image; According to the segmented image of the n*k+1th frame image, the transmission information of the n*k+1th frame image is obtained; The step of obtaining the transmission information of the n*k+1th frame image according to the segmented image of the n*k+1th frame image includes: Input the pixel value of each pixel in the segmented image of the n*k+1th frame image into the gating function to obtain the target parameter of each pixel in the segmented image of the n*k+1th frame image, wherein the target parameter is used to characterize the probability that the pixel belongs to the main object in the image; Extracting at least part of the image features in the segmented image of the n*k+1th frame image according to the target parameter, and determining the features as the transfer information of the n*k+1th frame image; The first image segmentation model is an image segmentation network with a number of layers greater than a first preset value, the second image segmentation model is an image segmentation network with a number of layers less than a second preset value, and the first preset value is greater than or equal to the second preset value.

2. The image segmentation method according to claim 1, characterized in that: When n is any integer from 1 to m, the step of inputting the n*k+1th frame image of the target video into the first image segmentation model includes: The n*k+1th frame image and the transmission information of the adjacent previous frame image are input into the first image segmentation model.

3. The image segmentation method according to claim 1, characterized in that: The output value of the gating function has a value range of [0, 1], and extracting at least part of the image features in the segmented image of the n*k+1th frame image according to the target parameter and determining it as the transfer information of the n*k+1th frame image includes: The pixel value of each pixel point in the segmented image of the n*k+1th frame image is calculated, and the product of the pixel value and the target parameter of the pixel point is calculated to obtain the transmission information of the n*k+1th frame image.

4. The image segmentation method according to claim 1, characterized in that: The gating function is g=1 / (1+exp(-x)); Wherein, x represents the pixel value of the pixel point in the segmented image, and g represents the target parameter.

5. The image segmentation method according to claim 1, characterized in that: The step of inputting each image frame from the n*k+2th to the (n+1)*kth image frame, together with the transmission information of the previous image frame adjacent to the image frame, into the second image segmentation model comprises: The images of multiple channels included in each frame of the image from the n*k+2th to the (n+1)*kth frame are input into the second image segmentation model together with the transmission information of the adjacent previous frame of the image.

6. The image segmentation method according to claim 1, characterized in that: The image segmentation network is a deep learning segmentation Unet network.

7. The image segmentation method according to claim 1, characterized in that: The parameters of the second image segmentation model used to perform image segmentation on each frame image from the n*k+2th frame image to the (n+1)*kth frame image are the same.

8. An image segmentation device, characterized in that: include: A video acquisition module is configured to acquire a target video to be processed; The first processing module is configured to execute the following process when n is an integer from 0 to m: Inputting the n*k+1th frame image of the target video into the first image segmentation model to obtain a segmented image of the n*k+1th frame image; wherein m is a positive integer, k is a predetermined constant, and k is an integer greater than 1; Input each frame image from the n*k+2th to (n+1)*kth frame image and the transfer information of the previous frame image adjacent to it into the second image segmentation model to obtain a segmented image of each frame image from the n*k+2th to (n+1)*kth frame image, wherein the transfer information of a frame image includes at least part of the image features extracted from the segmented image of the frame image; A transfer information acquisition module is configured to acquire transfer information of the n*k+1th frame image according to the segmented image of the n*k+1th frame image; Wherein, the transmission information acquisition module includes: The parameter determination submodule is configured to input the pixel value of each pixel point in the segmented image of the n*k+1th frame image into the gating function to obtain the target parameter of each pixel point in the segmented image of the n*k+1th frame image, wherein the target parameter is used to characterize the probability that the pixel point belongs to the main object in the image; an information extraction submodule, configured to extract at least part of the image features in the segmented image of the n*k+1th frame image according to the target parameter, and determine the features as the transfer information of the n*k+1th frame image; The first image segmentation model is an image segmentation network with a number of layers greater than a first preset value, the second image segmentation model is an image segmentation network with a number of layers less than a second preset value, and the first preset value is greater than or equal to the second preset value.

9. The image segmentation device according to claim 8, characterized in that: When n is any integer from 1 to m, when the first processing module inputs the n*k+1th frame image of the target video into the first image segmentation model, the first processing module is specifically configured as follows: The n*k+1th frame image and the transmission information of the adjacent previous frame image are input into the first image segmentation model.

10. The image segmentation device according to claim 8, characterized in that: The output value of the gating function has a value range of [0, 1], and the information extraction submodule is specifically configured as follows: The pixel value of each pixel point in the segmented image of the n*k+1th frame image is calculated, and the product of the pixel value and the target parameter of the pixel point is calculated to obtain the transmission information of the n*k+1th frame image.

11. The image segmentation device according to claim 8, characterized in that: The gating function is g=1 / (1+exp(-x)); Wherein, x represents the pixel value of the pixel point in the segmented image, and g represents the target parameter.

12. The image segmentation device according to claim 8, characterized in that: When the first processing module inputs each frame image from the n*k+2th to (n+1)*kth frame image and the transmission information of the adjacent previous frame image into the second image segmentation model, the first processing module is specifically configured as follows: The images of multiple channels included in each frame of the image from the n*k+2th to the (n+1)*kth frame are input into the second image segmentation model together with the transmission information of the adjacent previous frame of the image.

13. The image segmentation device according to claim 8, characterized in that: The image segmentation network is a deep learning segmentation Unet network.

14. The image segmentation device according to claim 8, characterized in that: The parameters of the second image segmentation model used to perform image segmentation on each frame image from the n*k+2th frame image to the (n+1)*kth frame image are the same.

15. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image segmentation method according to any one of claims 1 to 7.

16. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image segmentation method as claimed in any one of claims 1 to 7.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the image segmentation method according to any one of claims 1 to 7 is implemented.

Images