Image segmentation method and apparatus, and computing device cluster

Abstract

An image segmentation method, comprising: acquiring a video stream, and acquiring a target image frame from the video stream; obtaining at least one mask on the basis of image information of the target image frame and optical flow information of multiple image frames; and on the basis of the at least one mask, segmenting, from the target image frame, pixel points corresponding to pixel values in the at least one mask. In the present application, according to the method, when image segmentation is performed an image frame in a video, a person and an object that have the same or similar optical flow information can be classified into the same category on the basis of the image and optical flow information of multiple image frames previous to and subsequent to the image. According to the method, when a person is segmented from an image, an object of the same category to which the person belongs is also segmented, so that the person and the object associated with the person are segmented.

Description

Image segmentation method, device and computing device cluster

[0001] The present application claims priority to the Chinese patent application No. 202411307090.9, filed on September 18, 2024, and entitled "Image segmentation method, device and computing device cluster", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of image segmentation, and in particular to an image segmentation method, device and computing device cluster. BACKGROUND

[0003] Virtual background generally refers to a background effect used in video applications such as video conferencing, live streaming, video calls, etc. The virtual background can be a digitally produced image or a real-time generated virtual background, which is used to replace the real environment. When the virtual background is replaced, the original image needs to be segmented to fuse the segmented human image with the virtual background to obtain the image after background replacement. The image segmentation technology in the related art is not good enough, resulting in poor effect after virtual background replacement. SUMMARY

[0004] To solve the above problems, an image segmentation method is provided in the embodiments of the present application, which can segment the same kind of object to which the human belongs when segmenting the human from the image. In addition, the present application also provides an image segmentation device and a computing device cluster corresponding to the image segmentation method.

[0005] To this end, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an image segmentation method is provided in the embodiments of the present application, comprising: obtaining a video stream, and obtaining a target frame image from the video stream; obtaining at least one mask according to image information of the target frame image and optical flow information of a plurality of frame images; the plurality of frame images include at least one frame image before the target frame image, the target frame image, and at least one frame image after the target frame image in the video stream; the optical flow information of each frame image is calculated by optical flow field of two consecutive frame images; the mask includes pixel values of each pixel point of at least one object and pixel values of each pixel point of a human; and segmenting, from the target frame image, pixel points corresponding to the pixel values in the at least one mask according to the at least one mask.

[0007] In the embodiment, when the method performs image segmentation on a frame of the video, the same or similar persons and objects can be classified into the same class according to the optical flow information of the frame and the frames before and after the frame. When the method segments a person from the frame, the person and the objects of the same class to which the person belongs are segmented together, so that the person and the objects associated with the person are segmented.

[0008] In an embodiment, the obtaining at least one mask according to the image information of the target frame and the optical flow information of the multiple frames specifically comprises: performing feature extraction on the image information of the target frame and the optical flow information of the multiple frames to obtain image features of the target frame and optical flow features of the multiple frames; performing feature fusion on the image features of the target frame and the optical flow features of the multiple frames to obtain fused features of the target frame; obtaining multiple masks according to the fused features of the target frame and multiple sub-images; the multiple sub-images are partial images segmented from the target frame; each sub-image includes pixel points of a person or pixel points of an object; and the at least one mask is selected from the multiple masks according to a rule input by a user.

[0009] In the embodiment, the image information and the optical flow information are extracted to capture and represent more complex and abstract information and patterns in the data, so that the features can be considered in time and space dimensions when processing the continuous frames, and the spatio-temporal relationship can be extracted more effectively.

[0010] In an embodiment, the obtaining multiple masks according to the fused features of the target frame and multiple sub-images specifically comprises: performing image segmentation on the target frame to obtain the multiple sub-images; identifying feature vectors of the multiple sub-images from the fused features of the target frame; calculating similarity between a feature vector of any sub-image including pixel points of a person and a feature vector of a sub-image including pixel points of an object; and forming a mask with pixel values of the sub-image including pixel points of the person and pixel values of the sub-image including pixel points of the object corresponding to the similarity greater than a first set value, to obtain the multiple masks.

[0011] In an embodiment, the method further comprises: calculating similarity between feature vectors of any two masks; and forming a mask with the two masks corresponding to the similarity greater than a second set value, to obtain the multiple masks.

[0012] In an embodiment, the method further comprises: obtaining depth information of the multiple frames of images; obtaining at least one mask according to the image information of the target frame of image, the optical flow information of the multiple frames of images, and the depth information of the multiple frames of images; and segmenting, from the target frame of image, pixel points corresponding to pixel values in the at least one mask according to the at least one mask.

[0013] In this embodiment, if there are multiple persons in the image, the method can additionally introduce depth information. The method can divide persons and objects with the same or similar optical flow information and the same or similar depth information into the same category according to the image and the optical flow information and the depth information of multiple frames of images before and after the image, and can avoid the image segmentation system from dividing multiple persons and objects associated with the multiple persons in the same motion state into the same category.

[0014] In an embodiment, the method further comprises: fusing the pixel points segmented from the target frame of image and a virtual background to obtain a background-replaced image.

[0015] In this embodiment, after the method segments, from the target frame of image, pixel points corresponding to a target person and objects associated with the target person, the method fuses the segmented pixel points and a virtual background, thereby obtaining a background-replaced image with the target person and the objects associated with the target person.

[0016] In a second aspect, an image segmentation device is provided in the embodiments of the present application, which comprises: a first processing unit configured to obtain a video stream and a target frame of image from the video stream; a second processing unit configured to obtain at least one mask according to image information of the target frame of image and optical flow information of the multiple frames of images; the multiple frames of images comprising at least one frame of image before the target frame of image, the target frame of image, and at least one frame of image after the target frame of image in the video stream; the optical flow information of each frame of image being optical flow fields calculated through two consecutive frames of image; the mask comprising pixel values of each pixel point of at least one object and pixel values of each pixel point of one person; and a third processing unit configured to segment, from the target frame of image, pixel points corresponding to pixel values in the at least one mask according to the at least one mask.

[0017] In an embodiment, the second processing unit is specifically configured to perform feature extraction on the image information of the target frame image and the optical flow information of the plurality of frame images respectively to obtain image features of the target frame image and optical flow features of the plurality of frame images; perform feature fusion on the image features of the target frame image and the optical flow features of the plurality of frame images to obtain fused features of the target frame image; and obtain a plurality of masks according to the plurality of sub-images and the fused features of the target frame image, wherein the plurality of sub-images are partial images segmented from the target frame image, and each sub-image includes pixel points of a person or pixel points of an object; and select the at least one mask from the plurality of masks according to a rule input by a user.

[0018] In an embodiment, the second processing unit is specifically configured to perform image segmentation on the target frame image to obtain the plurality of sub-images; identify feature vectors of the plurality of sub-images from the fused features of the target frame image; calculate a similarity between a feature vector of any one sub-image including pixel points of a person and a feature vector of a sub-image including pixel points of an object; and form a mask by using pixel values of the sub-image including pixel points of the person and pixel values of the sub-image including pixel points of the object corresponding to a similarity greater than a first set value to obtain the plurality of masks.

[0019] In an embodiment, the second processing unit is further configured to calculate a similarity between feature vectors of any two masks; and form a mask by using the two masks corresponding to a similarity greater than a second set value to obtain the plurality of masks.

[0020] In an embodiment, the first processing unit is further configured to obtain depth information of the plurality of frame images; the second processing unit is further configured to obtain at least one mask according to the image information of the target frame image, the optical flow information of the plurality of frame images, and the depth information of the plurality of frame images; and the third processing unit is further configured to segment, from the target frame image, pixel points corresponding to pixel values in the at least one mask.

[0021] In an embodiment, the third processing unit is further configured to fuse the pixel points segmented from the target frame image and a virtual background to obtain a background-replaced image.

[0022] In a third aspect, an embodiment of the present application provides a computing device, including: at least one memory; and at least one processor configured to execute instructions stored in the memory to cause the computing device to perform the embodiments of the first aspect.

[0023] In a fourth aspect, a computer-readable storage medium is provided in the embodiments of the present application, and includes computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the embodiments of each possible implementation of the first aspect.

[0024] In a fifth aspect, a computer program product containing instructions is provided in the embodiments of the present application. The computer program product stores instructions. When the instructions are executed by a computing device, the computing device implements the embodiments of each possible implementation of the first aspect.

[0025] In a sixth aspect, a computing device cluster is provided in the embodiments of the present application. The computing device cluster includes at least one computing device. Each computing device includes a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the embodiments of each possible implementation of the first aspect.

[0026] In a seventh aspect, a computer-readable storage medium is provided in the embodiments of the present application, and includes computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the embodiments of each possible implementation of the first aspect.

[0027] In an eighth aspect, a computer program product containing instructions is provided in the embodiments of the present application. The computer program product stores instructions. When the instructions are executed by a computing device cluster, the computing device cluster implements the embodiments of each possible implementation of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0028] The drawings required for use in the embodiments or prior art description are briefly introduced as follows.

[0029] FIG. 1 is an example diagram of a video live scene provided in the embodiments of the present application;

[0030] FIG. 2 is a process schematic diagram of object segmentation performed by a background device provided in the embodiments of the present application;

[0031] FIG. 3 is a structural schematic diagram of an image segmentation system provided in the embodiments of the present application;

[0032] FIG. 4 is a scene schematic diagram of a user using an image segmentation system provided in the embodiments of the present application;

[0033] FIG. 5 is a flowchart of an image segmentation method provided in the embodiments of the present application;

[0034] FIG. 6 is a structural schematic diagram of an image segmentation apparatus provided in the embodiments of the present application;

[0035] FIG. 7 is a structural schematic diagram of a computing device according to an embodiment of the present application;

[0036] FIG. 8 is an architectural schematic diagram of a computing device cluster according to an embodiment of the present application;

[0037] FIG. 9 is an architectural schematic diagram of another computing device cluster according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0039] The term "and / or" in the present document is used to describe an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The symbol " / " in the present document means an or relationship of associated objects, for example, A / B means A or B.

[0040] The terms "first" and "second" and the like in the description and claims of the present document are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first response message and the second response message are used to distinguish different response messages, rather than to describe a specific order of the response messages.

[0041] In the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design presented as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Rather, the use of the words "exemplary" or "for example" is intended to present concepts in a concrete manner.

[0042] In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units, and the like; a plurality of elements means two or more elements, and the like.

[0043] Before introducing the technical solutions protected by the present application, several professional terms involved in the technical solutions protected by the present application are explained in advance, which are as follows:

[0044] Optical flow refers to the phenomenon of changes in pixel brightness in image sequences due to object movement or camera movement, which expresses the changes in images. When extracting the optical flow, the motion direction and speed of a pixel point can be inferred according to the changes in the gray scale or color value of the pixel point in two consecutive images in a video, so as to obtain the optical flow of the pixel point.

[0045] An optical flow field refers to a field of instantaneous velocity used to describe the motion of pixels in a video sequence in video processing and computer vision. The optical flow field is a two-dimensional vector field reflecting the gray level change trend of each pixel point in an image. Generally, when calculating the optical flow field, the optical flow of each pixel point in the entire image can be calculated to obtain the optical flow field of the image.

[0046] A mask is transparency information of an image. The transparency information includes a 0-1 floating point number corresponding to each pixel point. 0 represents complete transparency, and 1 represents complete opacity. The long side of the mask is the same as the long side of the image, and the wide side of the mask is the same as the wide side of the image. The mask is generally composed of multiple pixel values, and each pixel value represents the transparency information of a pixel point in the image.

[0047] Next, the technical solutions provided by the present application are introduced.

[0048] Generally, in a video scene, when a host or speaker explains a certain product, the segmentation model in the industry generally only segments the human region, and cannot segment the interactive product that the human wants to show. If the product needs to be segmented, the segmentation model can separately segment the object. However, the segmented object lacks relevance between the human or the object, so that all the segmented objects are displayed, or all the segmented objects of the same type are displayed, resulting in a poor effect after virtual background replacement.

[0049] Therefore, the image segmentation system provided by the embodiments of the present application can divide the human and the object with the same or similar optical flow information into the same category according to the optical flow information of the image and multiple images before and after the image when the image segmentation system performs image segmentation on a frame of image in a video. When the image segmentation system segments the human from the image, the object of the same category to which the human belongs is also segmented, so that the human and the object associated with the human are segmented.

[0050] If there are multiple humans in the image, the image segmentation system can additionally introduce depth information. The image segmentation system can divide the human and the object with the same or similar optical flow information and the same or similar depth information into the same category according to the optical flow information and the depth information of the image and multiple images before and after the image, which can avoid the image segmentation system from dividing multiple humans with the same motion state and the objects associated with the multiple humans into the same category.

[0051] Exemplarily, the image segmentation system is applied to a live video streaming scenario. As shown in FIG. 1, the live video streaming scenario includes two anchors, a male anchor and a female anchor. Both of the two anchors are seated at the same table. Two microphones are placed on the table and in front of the two anchors respectively. At least one camera is arranged in the scenario to capture the scenario and transmit the scenario to a backstage device in the form of a video stream.

[0052] FIG. 2 is a schematic diagram of a process of object segmentation performed by the backstage device according to an embodiment of the present application. As shown in FIG. 2, after receiving the video stream transmitted by the camera, the backstage device can collect image information and depth information of each frame of the video stream, and optical flow information between two consecutive frames. The backstage device can input the image information and depth information of each frame, and the optical flow information between two consecutive frames to the image segmentation system. The image segmentation system can segment the two anchors, the table and the two microphones from the video stream based on the image information, the depth information and the optical flow information. The image segmentation system can generate four masks according to the correlation among the anchors, the table and the microphones, which are mask 1, mask 2, mask 3 and mask 4. The mask 1 includes pixels of the male anchor and pixels of the microphone on the table and close to the male anchor. The mask 2 includes pixels of the female anchor and pixels of the microphone on the table and close to the female anchor. The mask 3 includes pixels of the male anchor, pixels of the table and pixels of the microphone on the table and close to the male anchor. The mask 4 includes pixels of the female anchor, pixels of the table and pixels of the microphone on the table and close to the female anchor.

[0053] The image segmentation system selects a mask with the highest probability from the four masks according to the probability distribution of the four masks, and outputs the mask. The backstage device can segment the image according to the mask to obtain a person in the mask and each object associated with the person. The backstage device can fuse the pixels of the person and the objects segmented by the backstage device with a virtual background to obtain an image after background replacement, and then transmit the image to a cloud server in the form of a video, and the cloud server can distribute the image to devices corresponding to each viewer. Thus, the viewer sees a picture in which the background is replaced, there are anchors and each object associated with the anchors.

[0054] FIG. 3 is a schematic diagram of a structure of an image segmentation system according to an embodiment of the present application. As shown in FIG. 3, the image segmentation system 300 can be divided into a pre-processing unit 310, an image processing unit 320, a depth processing unit 330, an optical flow processing unit 340 and a feature fusion unit 350 according to the functions performed by the image segmentation system 300.

[0055] The preprocessing unit 310, the image processing unit 320, the depth processing unit 330, the optical flow processing unit 340, and the feature fusion unit 350 can be implemented by software or by hardware. For example, the implementation of the preprocessing unit 310 is described below. The implementation of the image processing unit 320, the depth processing unit 330, the optical flow processing unit 340, and the feature fusion unit 350 can be similar to the implementation of the preprocessing unit 310.

[0056] As an example of a software functional unit, the preprocessing unit 310 can include code running on a computing instance. The computing instance can include at least one of a physical host (computing device), a virtual machine, and a container. Further, the computing instance can be one or more. For example, the preprocessing unit 310 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs. Each AZ includes one data center or multiple data centers in a similar geographical location. Typically, one region includes multiple AZs.

[0057] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same virtual private cloud (VPC), or in multiple VPCs. Typically, one VPC is set in one region, and communication between two VPCs in the same region or between VPCs in different regions needs to be set through a communication gateway in each VPC to realize the interconnection between the VPCs.

[0058] As an example of a hardware functional unit, the preprocessing unit 310 can include at least one computing device, such as a server or the like. Alternatively, the preprocessing unit 310 can also be a device implemented with an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the like. The PLD can be implemented with a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0059] The plurality of computing devices included in the preprocessing unit 310 can be distributed in the same region or in different regions. The plurality of computing devices included in the preprocessing unit 310 can be distributed in the same AZ or in different AZs. Similarly, the plurality of computing devices included in the preprocessing unit 310 can be distributed in the same VPC or in multiple VPCs. The plurality of computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0060] It should be noted that in other embodiments, the preprocessing unit 310 can be configured to perform any step of the image segmentation method, the image processing unit 320 can be configured to perform any step of the image segmentation method, the depth processing unit 330 can be configured to perform any step of the image segmentation method, the optical flow processing unit 340 can be configured to perform any step of the image segmentation method, and the feature fusion unit 350 can be configured to perform any step of the image segmentation method. The steps to be performed by the preprocessing unit 310, the image processing unit 320, the depth processing unit 330, the optical flow processing unit 340, and the feature fusion unit 350 can be specified as needed, and the overall function of the image segmentation system can be achieved by the preprocessing unit 310, the image processing unit 320, the depth processing unit 330, the optical flow processing unit 340, and the feature fusion unit 350 performing different steps of the image segmentation method.

[0061] When the image segmentation system 300 performs image segmentation on the target frame image, not only the image information, depth information and optical flow information of the target frame image need to be obtained from the video, but also the image information, depth information and optical flow information of the first set number of images before the target frame image and the image information, depth information and optical flow information of the second set number of images after the target frame image need to be obtained. When the image segmentation system 300 performs image segmentation on the target frame image, the image information, depth information and optical flow information of the multiple frames of images before and after the target frame image can be combined to improve the accuracy of image segmentation on the target frame image.

[0062] It should be noted that since the optical flow information is obtained through two consecutive frames of images, the optical flow information of each frame of image is obtained through the frame of image and the previous frame of image in this application.

[0063] The preprocessing unit 310 is configured to perform image segmentation on the target frame image to obtain a plurality of sub-images after receiving the image information of the target frame image.

[0064] Exemplarily, the preprocessing unit 310 can select an algorithm based on a traditional image processing method, a machine learning method, a deep learning method, an increasing method based on an image segmentation model, etc., to perform image segmentation on the person and the object in the target frame image to obtain a plurality of sub-images including one person or one object.

[0065] Taking the segmenting object by locations (SOLO) algorithm in the deep learning-based method as an example. After receiving the target frame image, the SOLO algorithm can perform standardization processing on the target frame image, such as adjusting the image size, normalizing the pixel value, etc., to ensure the consistency of the input image.

[0066] The SOLO algorithm can use a convolutional neural network (CNN) or other deep learning model to extract features in the target frame image to capture important information in the target frame image, such as edges, textures, etc. The SOLO algorithm can predict the center position and the corresponding region of each target in the target frame image through the network. The SOLO algorithm can generate a segmentation mask based on the center position and the corresponding region of each target. The mask represents the accurate distribution of each target in the target frame image. The position of each target is associated with a specific segmentation region, thereby forming a segmentation contour of the target in the image. The SOLO algorithm identifies the segmented targets to identify which targets belong to persons and which targets belong to objects. The SOLO algorithm can perform post-processing on the sub-image corresponding to each target, such as refining the boundary, removing noise, etc., to improve the accuracy and consistency of segmentation.

[0067] The preprocessing unit 310 can identify and classify the plurality of sub-images, to facilitate subsequent similarity calculation between the sub-images including the person and the sub-images including the object. In an embodiment, taking the second frame image in FIG. 2 as an example. The second frame image includes five sub-images, i.e., a male anchor (hereinafter referred to as “person A”), a female anchor (hereinafter referred to as “person B”), a microphone (hereinafter referred to as “object C”) on the desktop and close to the male anchor, a microphone (hereinafter referred to as “object D”) on the desktop and close to the female anchor, and a table (hereinafter referred to as “object E”).

[0068] The preprocessing unit 310 can identify and classify each sub-image to identify and classify the type of each sub-image. For example, the preprocessing unit 310 identifies that “object A” is “person 1”, “object B” is “person 2”, “object C” is “object 1”, “object D” is “object 2”, and “object E” is “object 3”.

[0069] The image processing unit 320 is configured to extract features from the image information to obtain image feature information of each frame image.

[0070] For example, after the image processing unit 320 obtains the image information of each frame image, a fixed batch size (batch_size) can be selected, such as 5 consecutive frame images as a sequence. At this time, the first set number is 2 frames. The second set number is 2 frames. The target frame image is the third frame image in the 5 consecutive frame images.

[0071] The image processing unit 320 can concatenate the images with dimensions of HxWxC (height, weight, and color channel) in the time dimension to obtain a tensor with a shape of (batch_size, 5, H, W, C). The image processing unit 320 can create a 3D convolutional neural network (CNN), and the 3D CNN can receive the tensor with the shape of (batch_size, 5, H, W, C). The image processing unit 320 can input the tensor with the shape of (batch_size, 5, H, W, C) into the 3D CNN, so that the convolutional layer of the 3D CNN can process the information in the space (height and width) and the time (sequence depth) at the same time. The 3D CNN can fuse the information in the space and the time by stacking multiple 3D convolutional layers, pooling layers, etc., to obtain image space-time features of each frame image. The image space-time features are a tensor containing higher-level space-time features.

[0072] The image processing unit 320 outputs a tensor containing higher-level spatio-temporal features, which can capture and represent more complex and abstract information and patterns in the data. When processing images of consecutive frames, the image processing unit 320 uses 3D convolutional layers that can consider features in both temporal and spatial dimensions simultaneously, thus more effectively extracting spatio-temporal relationships.

[0073] The depth processing unit 330 is configured to obtain depth information of each frame of image, and then extract features from the depth information to obtain depth feature information of each frame of image.

[0074] For example, after obtaining the depth information of each frame of image, the depth processing unit 330 can select a fixed batch size, such as 5 consecutive images including depth information, as a sequence. The depth processing unit 330 can concatenate the images of dimension HxWxC in the temporal dimension to obtain a tensor of shape (batch_size, 5, H, W, C).

[0075] The depth processing unit 330 can create a 3D CNN that can receive the tensor of shape (batch_size, 5, H, W, C). The depth processing unit 330 can input the tensor of shape (batch_size, 5, H, W, C) into the 3D CNN, so that the convolutional layers of the 3D CNN can process information in both spatial (height and width) and temporal (sequence depth) dimensions simultaneously. The 3D CNN can fuse information in spatial and temporal dimensions by stacking multiple 3D convolutional layers, pooling layers, etc., to obtain depth spatio-temporal features of each frame of image. The depth spatio-temporal features are a tensor containing higher-level spatio-temporal features.

[0076] The optical flow processing unit 340 is configured to obtain optical flow information of each frame of image, and then extract features from the optical flow information to obtain optical flow feature information of each frame of image.

[0077] For example, after obtaining the depth information of each frame of image, the depth processing unit 330 can select a fixed batch size, such as 5 consecutive images including depth information, as a sequence. The depth processing unit 330 can concatenate the images of dimension HxWxC in the temporal dimension to obtain a tensor of shape (batch_size, 5, H, W, C).

[0078] The optical flow processing unit 340 can create a 3D CNN that can receive a tensor with a shape of (batch_size, 5, H, W, C). The optical flow processing unit 340 can input the tensor with a shape of (batch_size, 5, H, W, C) into the 3D CNN, so that the convolutional layers of the 3D CNN can process information in space (height and width) and time (sequence depth) simultaneously. The 3D CNN can fuse information in space and time by stacking multiple 3D convolutional layers, pooling layers, etc., to obtain optical flow spatio-temporal features of each frame of image. The optical flow spatio-temporal features are a tensor containing higher-level spatio-temporal features.

[0079] The feature fusion unit 350 is configured to fuse the image feature information and the optical flow feature information of each frame of image to obtain fusion features carrying the image and the optical flow, or to fuse the image feature information, the optical flow feature information and the depth feature information of each frame of image to obtain fusion features carrying the image, the optical flow and the depth.

[0080] For example, after receiving the image feature information and the optical flow feature information of each frame of image, the feature fusion unit 350 can use an attention mechanism to fuse the image feature information and the optical flow feature information to obtain image-optical flow fusion features of the target frame of image.

[0081] If the feature fusion unit 350 also receives the depth feature information of each frame of image, the image-optical flow fusion features can be fused with the depth feature information to obtain image-optical flow-depth fusion features of the target frame of image. Hereinafter, the “image-optical flow fusion features” and the “image-optical flow-depth fusion features” are collectively referred to as “fusion features”.

[0082] The feature fusion unit 350 is also configured to obtain a plurality of masks according to the plurality of sub-images and the fusion features. Each mask includes pixel values of at least one object and pixel values of a person. In each mask, the pixel point area of the person is connected to the pixel point area of the object, and the optical flow information on the pixel points of the person is the same as the optical flow information on the pixel points of the object or the difference between them is less than a set value.

[0083] Exemplarily, the feature fusion unit 350 can design an attention mechanism, which can help the model to focus on important regions or features in the image and learn the correlation between pixels from the feature tensor. The attention mechanism can be a self-attention, a spatial attention, etc. The feature fusion unit 350 can input the fused features of each frame of image into the attention mechanism to calculate the attention weight of each position or channel of the fused features. After the feature fusion unit 350 calculates the attention mechanism of each position or channel by using the attention mechanism, the feature fusion unit 350 can reweight the feature tensor, so that the attention mechanism pays more attention to important features or differences in subsequent classification or segmentation tasks, thereby improving the accuracy. The feature tensor obtained by the feature fusion unit 350 after the attention mechanism processing is a feature tensor with a shape of (batch_size, time_steps, H, W, C), which is a feature tensor that pays more attention to important features.

[0084] The feature fusion unit 350 can perform pooling, convolution, etc. on the feature tensor processed by the attention mechanism, and convert the feature tensor with a shape of (batch_size, time_steps, H, W, C) into a feature vector with a shape of (batch_size, H, W, D). The feature fusion unit 350 extracts a feature vector with a shape of (batch_size, D, N) from the feature vector with a shape of (batch_size, H, W, D). N represents the number of masks that can be converted from a frame of image. The feature fusion unit 350 can obtain the feature vectors of N masks, and then compare the similarity of the feature vectors of the N masks two by two.

[0085] In one embodiment, still taking the second frame of image in FIG. 2 as an example. The feature fusion unit 350 can obtain five sub-images, i.e., the pixel region of object A in the second frame of image, the pixel region of object B in the second frame of image, the pixel region of object C in the second frame of image, the pixel region of object D in the second frame of image, and the pixel region of object E in the second frame of image. The feature fusion unit 350 can extract the feature vector corresponding to the pixel region of each sub-image according to the pixel region of each sub-image.

[0086] The feature fusion unit 350 can calculate the similarity between the feature vector of any one sub-image including a person and the feature vector of any one sub-image including an object according to the feature vectors corresponding to the pixel points of the sub-images. The feature fusion unit 350 can detect whether each similarity is greater than a first set value, and group two sub-images corresponding to the similarity greater than the first set value into one mask. For example, when the feature fusion unit 350 determines that the similarity between the feature vector of the mask of “person A” and the feature vector of the mask of “object C” is greater than the first set value, the pixel values of “person A” and “object C” can be grouped into one mask (i.e., “mask 1” in FIG. 2).

[0087] After obtaining a plurality of masks, the feature fusion unit 350 can calculate the similarity between the feature vectors corresponding to any two masks according to the feature vectors corresponding to the pixel points of the sub-images in each mask. The feature fusion unit 350 can detect whether each similarity is greater than a second set value, and re-group two masks corresponding to the similarity greater than the second set value into one mask. For example, when the feature fusion unit 350 determines that the similarity between the feature vector of the mask including the pixel values of “person A” and “object C” and the feature vector of the mask including the pixel values of “person C” and “object E” is greater than the second set value, the pixel values of “person A”, “person C” and “object E” can be grouped into one mask (i.e., “mask 3” in FIG. 2). Finally, the feature fusion unit 350 can output two sub-images corresponding to the similarity greater than the first set value into one mask, and re-group two masks corresponding to the similarity greater than the second set value into one mask.

[0088] After obtaining a plurality of masks, the feature fusion unit 350 can select one or more masks from the plurality of masks according to a rule actively input by a user. The rule actively input by the user can be a requirement selected by the user, such as a rule of “selecting to display only “person A” and not displaying “person B” in the video stream”. After receiving the rule, the feature fusion unit 350 can select a mask including the pixel values of “person A” from the plurality of masks.

[0089] Optionally, after obtaining the mask, the feature fusion unit 350 can screen the pixel values in the mask, and screen out pixel values less than a set threshold to obtain a screened mask. The feature fusion unit 350 can screen out pixel values in the sub-image that are not in the sub-image and are on the edge of the sub-image to improve the accuracy of image segmentation.

[0090] After obtaining the mask, the feature fusion unit 350 can multiply the pixel value (i.e., the transparency) corresponding to each pixel point in the mask with the image information corresponding to each pixel point in the target frame image, so as to segment out the pixel points corresponding to each sub-image in the mask from the target frame image, and obtain the target person and each object associated with the target person.

[0091] In the embodiment of the present application, when the image segmentation system 300 needs to segment the target frame image in the video, the image information and the optical flow information of the target frame image, the continuous multiple frames before the target frame and the continuous multiple frames after the target frame can be obtained. The image segmentation system 300 can segment each person and each object in the target frame image according to the image information of the target frame image. The image segmentation system 300 can divide the persons and objects with the same or similar optical flow information into the same class according to the optical flow information of the multiple frames. When the image segmentation system 300 segments the person from the target frame image, the objects belonging to the same class as the target frame image are also segmented, so as to realize the segmentation of the person and the objects associated with the person.

[0092] If there are multiple persons in the target frame image, the image segmentation system 300 can additionally introduce depth information. The image segmentation system 300 can divide the persons and objects with the same or similar optical flow information and the same or similar depth information into the same class according to the optical flow information and the depth information of the target frame image and the multiple frames before and after the target frame image, so as to avoid the image segmentation system 300 dividing multiple persons with the same motion state and the objects associated with the multiple persons into the same class.

[0093] After obtaining the pixels corresponding to the target person and the objects associated with the target person, the background device can fuse the pixels corresponding to the target person and the objects associated with the target person with the virtual background to obtain the image after background replacement, and then transmit the image to the cloud server, which is then distributed to the devices corresponding to each audience.

[0094] It should be understood that the functional modules and functional devices involved in the above-mentioned image segmentation system 300 can also be realized by software or hardware, and the specific implementation can be determined according to the actual situation, which is not limited here. In addition, the functional modules and functional devices involved in the above-mentioned image segmentation system 300 can be arranged separately or integrated, which is not limited here.

[0095] The above is the introduction of the image segmentation system 300 provided by the embodiment of the present application. It can be understood that the image segmentation system 300 described above can be configured on a cloud computing platform, for example, deployed on at least one virtual machine or container instance, so that the cloud computing platform can provide image segmentation services. Of course, the image segmentation system 300 can also be configured on a node other than the cloud computing platform, for example, can be deployed in at least one data center, or deployed on at least one server, which can be determined according to actual conditions, and is not limited here. Among them, the cloud computing platform can provide a page related to the public cloud service for users to remotely access the public cloud service. In the embodiment, the user can purchase the image segmentation service that the image segmentation system 300 can provide in advance on the cloud computing platform. For ease of understanding, the interaction mode between the user and the cloud computing platform is described below.

[0096] As shown in FIG. 4, the interaction between the user and the cloud computing platform mainly includes: the user logs in the cloud computing platform 400 through the client (notebook) webpage, selects and purchases the cloud service (i.e. image segmentation service) related to the image segmentation system 300 in the cloud computing platform 400, and after the purchase, the user can generate the image segmentation system 300 on the cloud computing platform 400 based on the functions provided by the image segmentation service. Among them, the cloud computing platform 400 is mainly used to manage the infrastructure for running the image segmentation service. For example, the infrastructure of the image segmentation service can include multiple data centers set in different regions, and each data center includes multiple servers. The data center can provide basic resources such as computing resources and storage resources for the image segmentation service. Therefore, when the user purchases and uses the image segmentation service, the user mainly pays for the resources used. When the user uses the image segmentation service, the user can input the demand for the image segmentation service through the configuration interface, the application program interface (API) or the interface for interacting with the user provided by the cloud computing platform 400, and the cloud computing platform 400 can generate the image segmentation service matched with the user's demand according to the demand input by the user (or other software / hardware, etc.).

[0097] In addition, the modules in the image segmentation system 300 can also be partially configured on the cloud side and partially configured on the terminal side, so as to realize the image segmentation service in the mode of terminal-cloud cooperation. In addition, the image segmentation system 300 can also be entirely configured on the terminal side, which can be determined according to actual conditions, and is not limited here.

[0098] The above is the introduction of the image segmentation system provided by the embodiment of the present application. Next, based on the above content, the simulation method provided by the embodiment of the present application is introduced.

[0099] Exemplarily, FIG. 5 shows a flowchart of an image segmentation method provided by an embodiment of the present application. It can be understood that the image segmentation method can be executed by the image segmentation system 300 described above, and the implementation process is as follows:

[0100] In step S501, a video stream is obtained, and a target frame image is obtained from the video stream.

[0101] When the image segmentation system 300 performs image segmentation on the target frame image, not only the image information, the depth information and the optical flow information of the target frame image need to be obtained from the video, but also the image information, the depth information and the optical flow information of the first set number of images before the target frame image and the image information, the depth information and the optical flow information of the second set number of images after the target frame image need to be obtained. When the image segmentation system 300 performs image segmentation on the target frame image, the image information, the depth information and the optical flow information of the multiple frames of images before and after the target frame image can be combined to improve the accuracy of image segmentation on the target frame image.

[0102] After the image segmentation system 300 receives the image information of the target frame image, the image segmentation system 300 performs image segmentation on the target frame image to obtain multiple sub-images. The image segmentation system 300 can identify and classify the multiple sub-images, which facilitates subsequent similarity calculation of the sub-images including the person and the sub-images including the object. The image segmentation system 300 can identify and classify each sub-image to identify and classify the type of each sub-image.

[0103] In step S502, at least one mask is obtained according to the image information of the target frame image and the optical flow information of the multiple frames of images.

[0104] After the image segmentation system 300 obtains the image information of each frame of image, a fixed batch size can be selected, and the continuous multiple frames of images are taken as a sequence. The image segmentation system 300 can splice the images with the dimension of HxWxC in the time dimension to obtain a tensor with the shape of (batch_size, 5, H, W, C). The image segmentation system 300 can input the tensor with the shape of (batch_size, 5, H, W, C) into the 3D CNN, so that the convolutional layer of the 3D CNN can process the information in space and time at the same time. The image segmentation system 300 can make the 3D CNN fuse the information in space and time by stacking multiple 3D convolutional layers, pooling layers and the like, so as to obtain the image spatio-temporal features of each frame of image.

[0105] After the image segmentation system 300 obtains the depth information of each frame of image, a fixed batch size can be selected, and the continuous multiple frames of images are taken as a sequence. The image segmentation system 300 can splice the images with the dimension of HxWxC in the time dimension to obtain a tensor with the shape of (batch_size, 5, H, W, C). The image segmentation system 300 can input the tensor with the shape of (batch_size, 5, H, W, C) into the 3D CNN, so that the convolutional layer of the 3D CNN can process the information in space and time at the same time. The image segmentation system 300 allows the 3D CNN to fuse the information in space and time by stacking multiple 3D convolutional layers, pooling layers, etc., so as to obtain the depth spatio-temporal features of each frame of image.

[0106] After the image segmentation system 300 obtains the depth information of each frame of image, a fixed batch size can be selected, and the continuous multiple frames of images are taken as a sequence. The image segmentation system 300 can splice the images with the dimension of HxWxC in the time dimension to obtain a tensor with the shape of (batch_size, 5, H, W, C). The image segmentation system 300 can input the tensor with the shape of (batch_size, 5, H, W, C) into the 3D CNN, so that the convolutional layer of the 3D CNN can process the information in space and time at the same time. The image segmentation system 300 allows the 3D CNN to fuse the information in space and time by stacking multiple 3D convolutional layers, pooling layers, etc., so as to obtain the optical flow spatio-temporal features of each frame of image.

[0107] After the image segmentation system 300 receives the image feature information and the optical flow feature information of each frame of image, the image segmentation system 300 can adopt an attention mechanism to fuse the image feature information and the optical flow feature information, so as to obtain the image-optical flow fusion feature of the target frame of image. If the image segmentation system 300 also receives the depth feature information of each frame of image, the image-optical flow fusion feature can be fused with the depth feature information to obtain the image-optical flow-depth fusion feature of the target frame of image. Hereinafter, the “image-optical flow fusion feature” and the “image-optical flow-depth fusion feature” are collectively referred to as “fusion feature”.

[0108] The image segmentation system 300 can input the fused features of each frame of image to an attention mechanism to calculate the attention weight of each position or channel of the fused features. After the image segmentation system 300 calculates the attention mechanism of each position or channel by using the attention mechanism, the image segmentation system 300 can reweight the feature tensor so that the attention mechanism pays more attention to important features or differences in subsequent classification or segmentation tasks, thereby improving the accuracy. The feature tensor obtained by the image segmentation system 300 after the attention mechanism processing is a feature tensor with a shape of (batch_size, time_steps, H, W, C), and the feature tensor is more focused on important features.

[0109] The image segmentation system 300 can perform pooling, convolution, etc. on the feature tensor processed by the attention mechanism to convert the feature tensor with a shape of (batch_size, time_steps, H, W, C) into a feature vector with a shape of (batch_size, H, W, D). The image segmentation system 300 extracts a feature vector with a shape of (batch_size, D, N) from the feature vector with a shape of (batch_size, H, W, D). N represents the number of masks that can be converted from a frame of image. The image segmentation system 300 can obtain the feature vectors of N masks, and then compare the similarity between the feature vectors of two masks.

[0110] The image segmentation system 300 can calculate the similarity between the feature vector of any one sub-image including a person and the feature vector of any one sub-image including an object according to the feature vectors corresponding to the pixel points of each sub-image. The image segmentation system 300 can detect whether each similarity is greater than a first set value, and group two sub-images corresponding to the similarity greater than the first set value into a mask. After obtaining a plurality of masks, the image segmentation system 300 can calculate the similarity between the feature vectors corresponding to any two masks according to the feature vectors corresponding to the pixel points of each sub-image in each mask. The image segmentation system 300 can detect whether each similarity is greater than a second set value, and re-group two masks corresponding to the similarity greater than the second set value into a mask.

[0111] In step S503, at least one mask is segmented from the target frame of image according to the pixel value corresponding to the pixel point in the at least one mask.

[0112] After obtaining the plurality of masks, the image segmentation system 300 can select one or more masks from the plurality of masks according to rules input by a user. After obtaining the mask, the image segmentation system 300 can multiply the pixel value (i.e., transparency) of each pixel point in the mask with the image information of each pixel point in the target frame image, so as to segment the pixel points corresponding to each sub-image in the mask from the target frame image, and obtain the target person and each object associated with the target person.

[0113] After obtaining the pixels corresponding to the target person and each object associated with the target person, the image segmentation system 300 can fuse the pixels corresponding to the target person and each object associated with the target person with the virtual background, obtain the image after background replacement, and then transmit the image to the cloud server for distribution to the devices corresponding to each viewer.

[0114] In the embodiment of the present application, when the image segmentation system 300 needs to segment the target frame image in the video, the image segmentation system 300 can obtain the image information and the optical flow information of the target frame image, the plurality of continuous frames before the target frame, and the plurality of continuous frames after the target frame. The image segmentation system 300 can segment each person and each object in the target frame image according to the image information of the target frame image. The image segmentation system 300 can divide the persons and objects with the same or similar optical flow information into the same class according to the optical flow information of the plurality of frames. When the image segmentation system 300 segments the person from the target frame image, the image segmentation system 300 also segments the objects belonging to the same class as the target frame image, so as to segment the person and the objects associated with the person.

[0115] If there are multiple persons in the target frame image, the image segmentation system 300 can additionally introduce depth information. The image segmentation system 300 can divide the persons and objects with the same or similar optical flow information and the same or similar depth information into the same class according to the optical flow information and the depth information of the target frame image and the plurality of frames before and after the target frame, so as to avoid the image segmentation system 300 dividing multiple persons with the same motion state and the objects associated with the multiple persons into the same class.

[0116] FIG. 6 is a structural schematic diagram of an image segmentation device provided in an embodiment of the present application. As shown in FIG. 6, the image segmentation device 600 can be divided into a first processing unit 610, a second processing unit 620, and a third processing unit 630 according to the functions performed.

[0117] The first processing unit 610, the second processing unit 620, and the third processing unit 630 can be implemented by software or by hardware. For example, the implementation of the first processing unit 610 is described below. Similarly, the implementation of the second processing unit 620 and the third processing unit 630 can refer to the implementation of the first processing unit 610.

[0118] As an example of a software functional unit, the first processing unit 610 can include code running on a compute instance. The compute instance can be at least one of a physical host (computing device), a virtual machine, a container, etc. Further, the computing device can be one or more. For example, the first processing unit 610 can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers for running the application can be distributed in the same region, or in different regions. The multiple hosts / virtual machines / containers for running the code can be distributed in the same AZ, or in different AZs. Each AZ includes one data center or multiple data centers in close geographical proximity. Typically, one region can include multiple AZs.

[0119] Similarly, the multiple hosts / virtual machines / containers for running the code can be distributed in the same VPC, or in multiple VPCs. Typically, one VPC is set up within one region. Communication between two VPCs in the same region, and cross-region communication between VPCs in different regions, requires a communication gateway to be set up in each VPC to achieve interconnection between VPCs.

[0120] As an example of a hardware functional unit, the first processing unit 610 can include at least one computing device, such as a server, etc. Alternatively, the first processing unit 610 can also be a device implemented by an ASIC, or a PLD, etc. The PLD can be a CPLD, an FPGA, a GAL, or any combination thereof.

[0121] The multiple computing devices included in the first processing unit 610 can be distributed in the same region, or in different regions. The multiple computing devices included in the first processing unit 610 can be distributed in the same AZ, or in different AZs. Similarly, the multiple computing devices included in the first processing unit 610 can be distributed in the same VPC, or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs, etc.

[0122] The image segmentation apparatus 600 implements the process as follows:

[0123] The first processing unit 610 is configured to acquire a video stream and acquire a target frame image from the video stream. The second processing unit 620 is configured to obtain at least one mask according to image information of the target frame image and optical flow information of a plurality of frame images. The plurality of frame images include at least one frame image before the target frame image, the target frame image, and at least one frame image after the target frame image in the video stream. The optical flow information of each frame image is an optical flow field calculated from two continuous frame images. The mask includes pixel values of each pixel point of at least one object and pixel values of each pixel point of a person. The third processing unit 630 is configured to segment, from the target frame image, pixel points corresponding to the pixel values in the at least one mask according to the at least one mask.

[0124] In an embodiment, the second processing unit 620 is specifically configured to perform feature extraction on the image information of the target frame image and the optical flow information of the plurality of frame images to obtain image features of the target frame image and optical flow features of the plurality of frame images. The second processing unit 620 is specifically configured to perform feature fusion on the image features of the target frame image and the optical flow features of the plurality of frame images to obtain fused features of the target frame image. A plurality of masks are obtained according to the fused features of the plurality of sub-images and the target frame image. The plurality of sub-images are partial images segmented from the target frame image. Each sub-image includes pixel points of a person or pixel points of an object. The second processing unit 620 is specifically configured to select at least one mask from the plurality of masks according to a rule input by a user.

[0125] In an embodiment, the second processing unit 620 is specifically configured to perform image segmentation on the target frame image to obtain a plurality of sub-images. The second processing unit 620 is specifically configured to identify feature vectors of the plurality of sub-images from the fused features of the target frame image. The second processing unit 620 is specifically configured to calculate a similarity between a feature vector of any one sub-image including pixel points of a person and a feature vector of a sub-image including pixel points of an object. The second processing unit 620 is specifically configured to form a mask including pixel values of the sub-image including pixel points of the person and pixel values of the sub-image including pixel points of the object corresponding to a similarity greater than a first set value, to obtain a plurality of masks.

[0126] In an embodiment, the second processing unit 620 is further configured to calculate a similarity between feature vectors of any two masks. The second processing unit 620 is further configured to form a mask including the two masks corresponding to a similarity greater than a second set value, to obtain a plurality of masks.

[0127] In an embodiment, the first processing unit 610 is further configured to acquire depth information of the plurality of frames of images. The second processing unit 620 is further configured to obtain at least one mask according to the image information of the target frame of image, the optical flow information of the plurality of frames of images, and the depth information of the plurality of frames of images. The third processing unit 630 is further configured to segment, from the target frame of image, at least one pixel point corresponding to a pixel value in the at least one mask.

[0128] In an embodiment, the third processing unit 630 is further configured to fuse the pixel point segmented from the target frame of image and the virtual background to obtain an image after background replacement.

[0129] FIG. 7 is a structural schematic diagram of a computing device provided in an embodiment of the present application. As shown in FIG. 7, the computing device 700 includes a bus 710, a processor 720, a memory 730, and a communication interface 740. The processor 720, the memory 730, and the communication interface 740 communicate with each other through the bus 710. The computing device 700 can be a server, a computer, a portable notebook, a cabinet, etc. It should be understood that the number of processors and memories in the computing device 700 is not limited in the present application.

[0130] The bus 710 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one line is used in FIG. 7, but it does not mean that there is only one bus or only one type of bus. The bus 710 can include a path for transmitting information between various components (e.g., the processor 720, the memory 730, the communication interface 740) of the computing device 700.

[0131] The processor 720 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0132] The memory 730 can include volatile memory (such as random access memory (RAM)), and / or non-volatile memory (such as read-only memory (ROM), floppy disks, or hard disks).

[0133] The memory 730 stores executable program code, which is executed by the processor 720 to implement the functions of the aforementioned modules, such as the first processing unit 610, the second processing unit 620, and the third processing unit 630, to implement the planning method. That is, the memory 730 stores instructions for implementing the planning method.

[0134] Alternatively, the memory 730 stores executable program code, which is executed by the processor 720 to implement the functions of the aforementioned modules, to implement the planning method. That is, the memory 730 stores instructions for implementing the planning method.

[0135] The communication interface 740 uses a transceiver module such as, but not limited to, a network interface card or a transceiver, to implement communication between the computing device 700 and other devices or communication networks.

[0136] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a notebook computer, or a smartphone.

[0137] As shown in FIG. 8, the computing device cluster includes at least one computing device 700. The memory 730 in one or more computing devices 700 in the computing device cluster can store the same instructions for implementing the planning method.

[0138] In some possible implementations, the memory 730 of one or more computing devices 700 in the computing device cluster can also respectively store partial instructions for implementing the planning method. In other words, the combination of one or more computing devices 100 can collectively execute the instructions for implementing the planning method.

[0139] It should be noted that the memory 730 in different computing devices 700 in the computing device cluster can store different instructions for performing part of the functions of the first processing unit 610, the second processing unit 620 and the third processing unit 630 respectively. That is, the instructions stored in the memory 730 in different computing devices 700 can implement the functions of one or more modules in the first processing unit 610, the second processing unit 620 and the third processing unit 630.

[0140] In some possible implementation manners, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. Figure 9 shows a possible implementation manner. As shown in Figure 9, two computing devices are computing device 700A and computing device 700B respectively, which are connected through a network. Specifically, the communication interface in each computing device is connected to the network. In this kind of possible implementation manner, the memory 730 in the computing device 700A stores instructions for performing the functions of part of the modules in the first processing unit 610, the second processing unit 620 and the third processing unit 630. Meanwhile, the memory 730 in the computing device 700B stores instructions for performing the functions of another part of the modules in the first processing unit 610, the second processing unit 620 and the third processing unit 630.

[0141] The connection manner between the computing device cluster shown in Figure 9 can be that a large amount of data needs to be stored in consideration of the planning method provided in the present application, and therefore it is considered to assign the functions implemented by another part of the modules in the first processing unit 610, the second processing unit 620 and the third processing unit 630 to the computing device 700B to perform.

[0142] It should be understood that the functions of the computing device 700A shown in Figure 9 can also be completed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be completed by multiple computing devices 700.

[0143] The embodiments of the present application also provide another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to the connection manners of the computing device clusters described with reference to Figures 7 and 8. The difference is that the memory 730 in one or more computing devices 700 in the computing device cluster can store the same instructions for performing the planning method.

[0144] In some possible implementation manners, the memory 730 in one or more computing devices 700 in the computing device cluster can also respectively store part of the instructions for performing the planning method. In other words, the combination of one or more computing devices 700 can collectively execute the instructions for performing the planning method.

[0145] It should be noted that the memories 730 in different computing devices 700 in the computing device cluster can store different instructions for performing part of the functions of the computing device 700. That is, the instructions stored in the memories 730 in different computing devices 700 can implement the functions of one or more modules in the first processing unit 610, the second processing unit 620, and the third processing unit 630 described above.

[0146] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform the hidden watermark embedding method.

[0147] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium contains instructions, which instruct the computing device to perform the planning method.

[0148] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An image segmentation method characterized by, The method comprises the following steps: acquiring a video stream and a target frame image from the video stream; obtaining at least one mask according to image information of the target frame image and optical flow information of a plurality of frame images; the plurality of frame images comprise at least one frame image before the target frame image, the target frame image and at least one frame image after the target frame image in the video stream; the optical flow information of each frame image is an optical flow field calculated through two continuous frame images; the mask comprises pixel values of each pixel point of at least one object and pixel values of each pixel point of a person; segmenting, from the target frame image, pixel points corresponding to the pixel values in the at least one mask according to the at least one mask.

2. The method of claim 1, wherein, The step of obtaining the at least one mask according to the image information of the target frame image and the optical flow information of the plurality of frame images specifically comprises the following steps: performing feature extraction on the image information of the target frame image and the optical flow information of the plurality of frame images to obtain image features of the target frame image and optical flow features of the plurality of frame images; performing feature fusion on the image features of the target frame image and the optical flow features of the plurality of frame images to obtain fusion features of the target frame image; obtaining a plurality of masks according to the fusion features of the target frame image and a plurality of sub-images; the plurality of sub-images are partial images segmented from the target frame image; each sub-image comprises pixel points of a person or pixel points of an object; selecting the at least one mask from the plurality of masks according to a rule input by a user.

3. The method of claim 2, wherein, The step of obtaining the plurality of masks according to the fusion features of the target frame image and the plurality of sub-images specifically comprises the following steps: performing image segmentation on the target frame image to obtain the plurality of sub-images; identifying feature vectors of the plurality of sub-images from the fusion features of the target frame image; calculating a similarity between a feature vector of any one sub-image comprising pixel points of a person and a feature vector of a sub-image comprising pixel points of an object; composing, into one mask, pixel values of a sub-image comprising pixel points of a person and pixel values of a sub-image comprising pixel points of an object corresponding to a similarity greater than a first set value to obtain the plurality of masks.

4. The method of claim 3, wherein, The method further comprises the following steps: calculating a similarity between feature vectors of any two masks; composing, into one mask, two masks corresponding to a similarity greater than a second set value to obtain the plurality of masks.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises the following steps: acquiring depth information of the plurality of frame images; obtaining at least one mask according to the image information of the target frame image, the optical flow information of the plurality of frame images and the depth information of the plurality of frame images; segmenting, from the target frame image, pixel points corresponding to the pixel values in the at least one mask according to the at least one mask.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises the following steps: fusing the pixel points segmented from the target frame image and a virtual background to obtain an image after background replacement.

7. An image segmentation apparatus characterized by comprising: The device comprises the following units: a first processing unit, configured to acquire a video stream and a target frame image from the video stream; a second processing unit, configured to obtain at least one mask according to image information of the target frame image and optical flow information of the multiple frame images; the multiple frame images include at least one frame image before the target frame image, the target frame image, and at least one frame image after the target frame image in the video stream; the optical flow information of each frame image is an optical flow field calculated through two continuous frame images; the mask includes pixel values of each pixel point of at least one object and pixel values of each pixel point of a person; a third processing unit, configured to segment, from the target frame image, pixel points corresponding to pixel values in the at least one mask according to the at least one mask.

8. The apparatus of claim 7, wherein the second processing unit is specifically configured to perform feature extraction on the image information of the target frame image and the optical flow information of the multiple frame images to obtain image features of the target frame image and optical flow features of the multiple frame images; perform feature fusion on the image features of the target frame image and the optical flow features of the multiple frame images to obtain fused features of the target frame image; obtain multiple masks according to the fused features of the target frame image and multiple sub-images; the multiple sub-images are partial images segmented from the target frame image; each sub-image includes pixel points of a person or pixel points of an object; select the at least one mask from the multiple masks according to a rule input by a user.

9. The apparatus of claim 8, wherein the second processing unit is specifically configured to perform image segmentation on the target frame image to obtain the multiple sub-images; identify feature vectors of the multiple sub-images from the fused features of the target frame image; calculate a similarity between a feature vector of any one sub-image including pixel points of a person and a feature vector of any one sub-image including pixel points of an object; compose pixel values of a sub-image including pixel points of a person and pixel values of a sub-image including pixel points of an object into a mask to obtain the multiple masks, when the similarity is greater than a first set value.

10. The apparatus of claim 9, wherein the second processing unit is further configured to calculate a similarity between feature vectors of any two masks; compose two masks into a mask to obtain the multiple masks, when the similarity is greater than a second set value.

11. The apparatus of any one of claims 7-10, wherein the first processing unit is further configured to acquire depth information of the multiple frame images; the second processing unit is further configured to obtain at least one mask according to the image information of the target frame image, the optical flow information of the multiple frame images, and the depth information of the multiple frame images; the third processing unit is further configured to segment, from the target frame image, pixel points corresponding to pixel values in the at least one mask according to the at least one mask.

12. The apparatus of any one of claims 7-11, wherein The third processing unit is further configured to fuse the pixel points segmented from the target frame image and the virtual background to obtain a background-replaced image.

13. A cluster of computing devices comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory to cause the cluster of computing devices to perform the method of any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, A computer program product comprising computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method of any one of claims 1-6.

15. A computer program product comprising instructions, characterized in that, The computer program product stores instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to implement the method of any one of claims 1-6.

Images