Video image segmentation method and device, electronic equipment and computer storage medium

Abstract

The embodiment of the invention provides a video image segmentation method and device, electronic equipment and a computer storage medium. The method comprises the following steps: acquiring a to-be-processed video frame and N1 cyclic memory parameters; the to-be-processed video frame is a frame of image in the to-be-processed video, the N1 cyclic memory parameters are in one-to-one correspondence with the N1 scales, and the cyclic memory parameters carry time sequence feature information of the scales corresponding to the previous frame or multiple frames of the to-be-processed video; performing multi-scale feature extraction on the to-be-processed video frame to obtain N1 pieces of initial feature information corresponding to N1 scales; performing multi-scale down-sampling on the to-be-processed video frame to obtain N2 pieces of down-sampling data; according to the N1 cyclic memory parameters, the N1 pieces of initial feature information and the N2 pieces of down-sampling data, performing multi-level convolution processing to obtain a target object mask pattern; the target object mask picture comprises a target object region and a background region in the to-be-processed video frame. According to the embodiment of the invention, the image segmentation accuracy can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a video image segmentation method, device, electronic device and computer storage medium. Background Art

[0002] Portrait segmentation refers to separating the portrait and background in an image into different areas and distinguishing them with different labels. Specifically, the part of the input image containing the portrait can be subjected to portrait segmentation detection, and the segmentation result is presented as the distinction between the portrait and the background. With the development of artificial intelligence, more and more artificial intelligence technologies are applied to portrait segmentation. Existing portrait segmentation based on deep learning, especially for the portrait segmentation of continuous video frames in a video, still needs to be improved in terms of segmentation effect and efficiency. Summary of the invention

[0003] The embodiments of the present disclosure provide a video image segmentation method, device, electronic device and computer storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a video image segmentation method, comprising:

[0005] Obtain a video frame to be processed and N1 loop memory parameters; the video frame to be processed is an image frame in the video to be processed, the N1 loop memory parameters correspond to the N1 scales one by one, and the loop memory parameters carry temporal feature information of the scales corresponding to the previous frame or multiple frames of the video to be processed;

[0006] Performing multi-scale feature extraction on the video frame to be processed to obtain N1 initial feature information corresponding to the N1 scales;

[0007] Perform multi-scale downsampling on the video frame to be processed to obtain N2 downsampled data;

[0008] Performing multi-level convolution processing according to the N1 loop memory parameters, the N1 initial feature information, and the N2 down-sampled data to obtain a target object mask map; the target object mask map includes a target object area and a background area in the video frame to be processed;

[0009] Wherein, N1 and N2 are both positive integers.

[0010] In a second aspect, an embodiment of the present disclosure provides an image processing device, including:

[0011] A feature encoding module is used to perform multi-scale feature extraction on the video frame to be processed to obtain N1 initial feature information corresponding to N1 scales; perform multi-scale downsampling on the video frame to be processed to obtain N2 downsampling data; wherein the video frame to be processed is an image frame in the video to be processed;

[0012] an iterative decoding module, configured to perform multi-stage convolution processing according to N1 loop memory parameters, the N1 initial feature information, and the N2 down-sampled data to obtain a target object mask image; the N1 loop memory parameters correspond one-to-one to the N1 scales, the loop memory parameters carry temporal feature information of the previous frame or multiple frames of the video to be processed corresponding to the scale, and the target object mask image includes a target object area and a background area in the video frame to be processed;

[0013] Among them, N1 and N2 are both positive integers.

[0014] In a third aspect, an embodiment of the present disclosure provides an electronic device, the electronic device comprising a memory and a processor; wherein:

[0015] The memory is used to store a computer program that can be run on the processor;

[0016] The processor is used to execute the method as described in the first aspect when running the computer program.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by at least one processor, the method described in the first aspect is implemented.

[0018] The embodiments of the present disclosure provide a method, apparatus, electronic device and computer storage medium for video image segmentation. When performing image segmentation, a cyclic memory parameter having temporal feature information of a previous frame or multiple frames is combined so that the temporal correlation between video frames is fully considered during the image segmentation process, and information transmission between continuous video frames can be performed, so that rich background information can be learned as the target object moves, thereby improving the stability and robustness of the segmentation of the video target object and achieving high-precision segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of a flow chart of a video image segmentation method provided in an embodiment of the present disclosure;

[0020] Figure 2 A detailed flowchart of a video image segmentation method provided by an embodiment of the present disclosure;

[0021] Figure 3 A schematic diagram of the composition structure of a video portrait segmentation network provided in an embodiment of the present disclosure;

[0022] Figure 4 A schematic diagram of the structure of a bottleneck module provided in an embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of the structure of an upsampling module provided in an embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of the structure of an output convolution module provided in an embodiment of the present disclosure;

[0025] Figure 7 A schematic diagram of a portrait mask provided by an embodiment of the present disclosure;

[0026] Figure 8 A schematic diagram of the structure of a video image segmentation device provided in an embodiment of the present disclosure;

[0027] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. It is understood that the specific embodiments described herein are only used to explain the relevant disclosure, rather than to limit the disclosure. It should also be noted that, for the convenience of description, only the parts related to the relevant disclosure are shown in the drawings.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure belongs. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0030] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present disclosure are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present disclosure described here can be implemented in an order other than that illustrated or described here.

[0032] In recent years, image and video segmentation methods based on deep learning have developed rapidly, but most of these methods are applied to processing single images. Even when processing video data, each frame of the video is processed as an independent image to obtain the image segmentation result of this frame. These methods ignore the timing characteristics between consecutive frames in the video, while the feature information between the timing can improve the segmentation effect and increase the robustness of video segmentation. In addition, even though the current image or video portrait segmentation methods have greatly improved the segmentation accuracy, most of them need to use some auxiliary information, such as a background image or initial character target information, and in terms of inference speed, because the network requires a lot of calculations to maintain the segmentation effect, it lacks real-time performance and is often not well deployed on edge devices.

[0033] Based on this, the embodiment of the present disclosure provides an image segmentation method. When performing image segmentation, a cyclic memory parameter with temporal feature information is combined, so that the temporal correlation between video frames is fully considered in the image segmentation process, and information transmission between continuous video frames can be performed, so that rich background information can be learned as the target object moves, thereby improving the stability and robustness of video target object segmentation and achieving high-precision segmentation.

[0034] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0035] In one embodiment of the present disclosure, see Figure 1 , which shows a flow chart of an image segmentation method provided by an embodiment of the present disclosure. Figure 1 As shown, the method may include:

[0036] S101: Obtain a video frame to be processed and N1 loop memory parameters.

[0037] It should be noted that the video image segmentation method provided in the embodiments of the present disclosure can be applied to a video image segmentation device, or an electronic device integrated with the device. Here, the electronic device can be, for example, a smart phone, a tablet computer, a server, a personal computer, etc., without specific limitation.

[0038] It should also be noted that the video frame to be processed is an image frame in the video to be processed, which can also be recorded as a video sequence frame. The N1 cyclic memory parameters correspond to the N1 scales one by one. The cyclic memory parameters carry the temporal feature information of the corresponding scale, and N1 is a positive integer. Here, the cyclic memory parameters can carry the temporal feature information of the previous video frame of the corresponding scale, or can also carry the temporal feature information of the previous video frames of the corresponding scale. Since the cyclic memory parameters are used to record the temporal feature information at different scales, the relevant feature information of the frames before the current frame can be retained, which can play a role in predicting the feature information of the current frame.

[0039] It should also be noted that temporal feature information can involve but is not limited to the continuity of actions in the video and changes in the temporal sequence, such as time information, spatial information, etc. Among them, temporal information (also known as temporal complexity) measures the temporal change of the video sequence, which can be reflected by the time difference between video frames. Video sequences with a higher degree of motion usually have higher temporal information values, that is, temporal information reflects the dynamic changes of actions in the video, which is crucial for tasks such as action recognition and temporal action positioning. Spatial information (also known as spatial complexity) can characterize the amount of spatial details of the image, reflecting the spatial structure and content of the image. Spatial information can be measured by calculating the standard deviation of the image. The more complex the scene is, the higher the corresponding spatial information value. Spatial information also plays an important role in video understanding, especially in applications that need to analyze spatial features such as the position, size and shape of objects in the image.

[0040] It can be understood that for the first frame of the video to be processed, there is no previous frame. At this time, the loop memory parameter is a default value. The default value can be set based on experience.

[0041] Here, the video to be processed may be, for example, a real-time surveillance video, a video call video, a played film or TV picture, and the like.

[0042] It should also be noted that the purpose of the embodiment of the present disclosure is to separate the target object from the background in the video frame to be processed. The target object may be a portrait, or the target object may be an animal, such as a cat, a dog, etc., or the target object may be other objects, such as an airplane, a car, etc. In the video to be processed, the target object may be fixed or moving. In the embodiment of the present disclosure, due to the use of cyclic memory parameters with temporal feature information, the temporal correlation between video frames is fully considered in the image segmentation process, so for moving target objects, it has a good segmentation effect.

[0043] S102: Perform multi-scale feature extraction on the video frame to be processed to obtain N1 feature information corresponding to N1 scales.

[0044] S103: Perform multi-scale downsampling on the video frame to be processed to obtain N2 down-sampled data.

[0045] It should be noted that step S102 and step S103 may be executed simultaneously or either one may be executed first, and the embodiment of the present disclosure does not specifically limit the execution order thereof, wherein N2 is a positive integer.

[0046] It should also be noted that, since the video frame to be processed is usually large in size and the format may not meet the requirements, before executing step S102 and step S103, the method may further include:

[0047] Pre-process the video frame to be processed; wherein the pre-processing includes at least one of the following processing methods: color space conversion, size scaling, and normalization operation.

[0048] It should be noted that color space conversion can be to convert the color space of the video frame to be processed into a red, green, blue (Red, Green, Blue, RGB) color space, resizing can be to reduce the resolution of the video frame to be processed, and normalization operation can be to process the pixel value of the video frame to be processed into a floating point decimal between 0 and 1. After pre-processing, multi-scale feature extraction and multi-scale downsampling are performed on the video frame to be processed, which is conducive to avoiding excessive calculation and improving processing speed, but it is not specifically limited. Pre-processing is an optional but not mandatory processing measure combined with actual conditions.

[0049] It should also be noted that when performing multi-scale feature extraction on the video frame to be processed, assuming that N1 is 3, feature extraction is required to obtain feature information of three different scales. For example, the three different scales may be 1 / 2, 1 / 4, 1 / 8, or 1 / 4, 1 / 8, 1 / 16, or 1 / 2, 1 / 8, 1 / 16, etc., which is not specifically limited in the embodiments of the present disclosure.

[0050] Here, multi-scale feature extraction plays a vital role in the field of computer vision and image processing. It helps improve the accuracy and robustness of segmentation results by extracting feature information of images at different scales. For example: the target objects in the image may have different scales. Multi-scale features can analyze the image at different scales and better capture the scale changes of the target objects, thereby avoiding the problem of smaller target objects being ignored or larger target objects being inaccurately segmented; multi-scale features can integrate contextual information at different scales to provide a more comprehensive feature expression, which helps to infer the position and boundaries of the target objects, as well as capture the details and textures inside the target objects; due to the presence of factors such as noise and occlusion, image segmentation tasks often have a certain degree of difficulty, and multi-scale features can reduce the impact of these noises and interferences by using features of different scales, enhance robustness, and further improve the accuracy of segmentation.

[0051] It should also be noted that when multi-scale downsampling is performed on the processed video frame, N2 is usually less than or equal to N1. Since feature fusion and convolution processing are required later, features with the same scale can be spliced ​​and fused. In practice, the correspondence between the scale of multi-scale downsampling and the scale of multi-scale feature extraction may not be limited, and the scale consistency may be ensured by upsampling and / or downsampling in subsequent steps; alternatively, when performing multi-scale downsampling, the scale corresponding to the multi-scale feature extraction may be adopted to simplify subsequent calculation steps and save calculation resources.

[0052] For example, N1 and N2 are the same, and the scales of multi-scale feature extraction and multi-scale downsampling are also the same (such as 1 / 4, 1 / 8, 1 / 16). For another example, assuming that N1 is 3 and N2 is 2, the three scales corresponding to multi-scale feature extraction are a1, a2, and a3 from large to small, then the scales of the two multi-scale downsampling data are a1 and a2, that is, the scales of multi-scale downsampling correspond to the larger scales of multi-scale feature extraction, so as to facilitate the scale correspondence during the subsequent multi-level convolution processing.

[0053] In some embodiments, multi-scale downsampling is performed on the video frame to be processed to obtain N2 downsampled data, including:

[0054] j is executed from 1 to N2 in sequence: down-sample the j-th input feature map to obtain the j-th down-sampled data; wherein, the 1st input feature map is the video frame to be processed, the 2nd to N2th input feature maps are the j-1th down-sampled data, and j is an integer greater than 1 and less than or equal to N2.

[0055] It should be noted that downsampling can be performed in the pooling layer, and the pooling window and sampling step size of the j+1th pooling layer are smaller than the pooling window and sampling step size of the jth pooling layer. Assuming N2 is 2, the corresponding two downsampling scales are 1 / 4 and 1 / 8 respectively, and the number of corresponding pooling layers is also 2. The first pooling layer downsamples the first input feature map (i.e., the video frame to be processed) to obtain the first downsampled data with a scale of 1 / 4; the second pooling layer further downsamples the first downsampled data to obtain downsampled data with a scale of 1 / 8.

[0056] S104: Perform multi-level convolution processing according to N1 cyclic memory parameters, N1 initial feature information, and N2 down-sampled data to obtain a target object mask map.

[0057] It should be noted that the target object mask includes the target object region where the segmented target object is located in the video frame to be processed and the background region excluding the target object. After multi-level convolution processing, the target object region and the background region can be segmented in the video frame to be processed.

[0058] It should also be noted that when performing multi-level convolution processing, it can be iterated in order from small to large scale to restore the scale of the image step by step. For example, N1 and N2 are both 3, and the three scales are 1 / 4, 1 / 8, and 1 / 16, respectively. In some implementations, the process of multi-level convolution processing can be as follows: the initial feature information and downsampled data with a scale of 1 / 16 are fused (for example, spliced), and the time series feature processing (such as time series convolution processing) is combined with the cyclic memory parameter with a scale of 1 / 16 to obtain fused feature information with a scale of 1 / 16 and time series characteristics; then the fused feature information with a scale of 1 / 16 is upsampled to obtain feature information with a scale of 1 / 8, and then it is fused (for example, spliced) with the initial feature information and downsampled data with a scale of 1 / 8, and then the time series feature processing (such as time series convolution processing) is combined with the cyclic memory parameter with a scale of 1 / 8 to obtain a scale of 1 / 8 with time series characteristics. The fused feature information with the scale of 1 / 8 is then upsampled to obtain the feature information with the scale of 1 / 4, and then fused (for example, spliced) with the initial feature information and downsampled data with the scale of 1 / 4, and then combined with the cyclic memory parameter with the scale of 1 / 4 to process the temporal features, and obtain the fused feature information with the scale of 1 / 4 and the temporal characteristics; finally, the fused feature information with the scale of 1 / 4 is upsampled again to obtain the fused feature information with the same scale as the video frame to be processed (depending on the specific implementation method, if pre-processing is performed, it is the video frame to be processed after pre-processing), and fused with the video frame to be processed to obtain the required target object mask. In each temporal convolution process, the cyclic memory parameter is also updated.

[0059] In this way, due to the involvement of the cyclic memory parameter in the multi-level convolution processing process, the cyclic memory parameter carries the temporal feature information of the previous frame or the previous few frames of the corresponding scale at this scale, so that for the information transmission between video frames, it is possible to learn rich background information as the target object moves, improve the stability and robustness of video image segmentation, and achieve high-precision segmentation.

[0060] In addition, when performing image segmentation, the embodiments of the present disclosure may also consider higher-level semantic features to further improve the segmentation effect. Therefore, in other implementations, N1 is 1 greater than N2, and except for the first scale, the initial feature information of the remaining scales corresponds to the scale of each downsampled data one by one; the process of multi-level convolution processing may be as follows:

[0061] The method further includes: performing semantic segmentation on initial feature information of a first scale to generate semantic feature information; the first scale is the smallest one among N1 scales;

[0062] Accordingly, a multi-level convolution process is performed based on N1 cyclic memory parameters, N1 initial feature information, and N2 down-sampled data to obtain a target object mask map, including:

[0063] Performing temporal convolution processing on the semantic feature information and the recurrent memory parameters of the first scale to obtain fused feature information of the first scale;

[0064] According to the order of scale from small to large, feature fusion and temporal convolution processing are performed according to the initial feature information of the current scale, the downsampled data, the cyclic memory parameters and the fused feature information of the previous scale to obtain the fused feature information of the current scale; until the fused feature information of the second scale is obtained; the second scale is the largest one among the N1 scales;

[0065] The fused feature information of the second scale and the video frame to be processed are subjected to feature fusion and convolution processing to obtain a mask map of the target object.

[0066] It should be noted that, here, N2 is smaller than N1. According to the order of scale from large to small, the N1 initial feature information are respectively the 1st feature information to the N1th initial feature information, and the N1 loop memory parameters are respectively the 1st loop memory parameter to the N1th loop memory parameter. For example: N1=3, the 3 scales are 1 / 4, 1 / 8, 1 / 16, respectively, and the scales of the 1st initial feature information to the 3rd initial feature information are respectively: 1 / 4, 1 / 8, 1 / 16. The scales corresponding to the 1st loop memory parameter to the 3rd loop memory parameter are respectively: 1 / 4, 1 / 8, 1 / 16.

[0067] The N2 downsampled data are respectively the 1st downsampled data to the N2th downsampled data. For example, N2=2, the scale of the 1st downsampled data is 1 / 4, and the scale of the 2nd downsampled data is 1 / 8. The subsequent descriptions are all based on N1=3 (3 feature extraction scales are 1 / 4, 1 / 8, 1 / 16) and N2=2 (2 downsampled scales are 1 / 4, 1 / 8) as examples.

[0068] Based on this example, semantic segmentation is performed on the initial feature information of the first scale, that is, semantic segmentation is performed on the third initial feature information (scale 1 / 16), to obtain semantic features. Image semantic features are an important concept in the field of computer vision, which involves associating each pixel in an image with a specific class label, thereby inferring high-level semantic information from the image to help computers understand the scene.

[0069] On this basis, the semantic feature information and the recurrent memory parameters of the first scale are subjected to temporal convolution processing to obtain the fused feature information of the first scale, that is, the semantic feature information and the recurrent memory parameters of the 1 / 16 scale are subjected to temporal convolution processing to obtain the fused feature information of the 1 / 16 scale and the updated recurrent memory parameters of the 1 / 16 scale.

[0070] If the current scale is recorded as the i-th scale, in the order of scales from small to large, feature fusion and temporal convolution processing are performed according to the initial feature information, downsampled data, cyclic memory parameters and fused feature information of the previous scale of the current scale to obtain the fused feature information of the current scale, that is: according to i from N1-1 to 1, j from N2 to 1, it is executed in sequence: feature fusion and temporal convolution processing are performed according to the j-th downsampled data, the i-th initial feature information, the i+1-th fused feature information, and the i-th cyclic memory parameter to obtain the i-th fused feature information and the updated i-th cyclic memory parameter; until the first fused feature information is obtained; finally, the first fused feature information and the video frame to be processed are subjected to feature fusion and convolution processing to obtain the mask map of the target object.

[0071] It should be noted that the process of feature fusion may include steps such as upsampling and splicing. The purpose of upsampling is to make the scales of the downsampled data, the initial feature information and the fused feature information consistent, so that corresponding splicing and convolution operations can be performed to obtain the fused feature information of the scale.

[0072] Temporal convolution processing can effectively extract the timing information in the signal and capture the dependency between the previous and next video frames, thereby improving the accuracy of image segmentation. Temporal convolution processing can be performed by a temporal convolution module (or also referred to as a temporal convolution model, etc.), which has built-in timing information to retain the feature information of the current video frame and the previous video frame. Here, the temporal convolution module can also be pre-trained so that the temporal convolution module can accurately extract the timing information of the video frame and update the loop memory parameters.

[0073] It should also be noted that in the temporal convolution module, the temporal convolution processing is performed in combination with the cyclic memory parameters, so that the previous frame or the previous few video frames affect the processing results of the current video frame, so that the image segmentation effect has the temporal correlation between continuous video frames, especially for moving objects, which can significantly improve the image segmentation effect. At the same time, the temporal convolution module will also update the cyclic memory parameters in combination with the various features of the current video frame to be processed, so that the cyclic memory parameters carry the temporal feature information of the current video frame to be processed, and use the updated cyclic memory parameters to perform temporal convolution processing on the next frame of the video frame to be processed, effectively transferring the temporal feature information of the current video frame to be processed to the next frame of the video frame to be processed. It can be understood that the cyclic memory parameters corresponding to the current video frame to be processed are updated according to the previous video frame to be processed.

[0074] Therefore, in some embodiments, during the multi-stage convolution processing, the method further includes:

[0075] The loop memory parameters of each scale are updated so that the loop memory parameters carry the temporal feature information of the current video frame to be processed at the corresponding scale; the updated loop memory parameters are used for multi-level convolution processing of the next video frame to be processed in the video to be processed.

[0076] That is to say, for each video frame to be processed, the corresponding loop memory parameters are the loop memory parameters updated during the processing of the previous video frame to be processed, so that the temporal correlation between video frames can be well preserved. For the first video frame to be processed in the video to be processed, the loop memory parameters can be initialized and assigned initial values. Then, in the subsequent process, the loop memory parameters are updated frame by frame.

[0077] For the time series convolution processing of the semantic feature information, in some embodiments, performing time series convolution processing on the semantic feature information and the cyclic memory parameter of the first scale to obtain the fused feature information of the first scale may include:

[0078] Splitting the semantic feature information into first semantic data and second semantic data;

[0079] Performing temporal convolution processing on the second semantic data and the cyclic memory parameter of the first scale to obtain temporal semantic data and obtain updated cyclic memory parameters corresponding to the first scale;

[0080] The first semantic data and the temporal semantic data are concatenated to obtain fused feature information corresponding to the first scale.

[0081] For the time series convolution processing of each scale feature in the multi-level convolution processing process, in some embodiments, feature fusion and time series convolution processing are performed according to the initial feature information of the current scale, the down-sampled data, the cyclic memory parameter, and the fused feature information of the previous scale to obtain the fused feature information of the current scale, including:

[0082] Upsample the fused feature information of the previous scale to obtain the upsampled time series features of the current scale;

[0083] The upsampled time series features, initial feature information, and downsampled data of the current scale are concatenated to obtain the merged feature information of the current scale;

[0084] Splitting the merged feature information of the current scale into first spliced ​​data and second spliced ​​data;

[0085] Performing convolution decoding processing on the first spliced ​​data to obtain convolution feature data;

[0086] Performing a time series convolution process on the second spliced ​​data and the cyclic memory parameters of the current scale to obtain the time series output data of the current scale, and obtaining the updated cyclic memory parameters corresponding to the current scale;

[0087] The convolution feature data and the time series output data are concatenated to obtain the fused feature information of the current scale.

[0088] If it is still expressed in the above i and j way, it is:

[0089] Upsample the i+1th fused feature information to obtain the i-th upsampled time series feature with the same scale as the i-th initial feature information;

[0090] The i-th up-sampled time series feature, the i-th initial feature information, and the j-th down-sampled data are concatenated to obtain the i-th combined feature information;

[0091] Splitting the i-th merged feature information into the i-th first spliced ​​data and the i-th second spliced ​​data;

[0092] Performing convolution decoding processing on the i-th first spliced ​​data to obtain the i-th convolution feature data;

[0093] Performing time series convolution processing on the i-th second splicing data and the i-th loop memory parameter to obtain the i-th time series output data and the updated i-th loop memory parameter;

[0094] The i-th convolution feature data and the i-th time series output data are concatenated to obtain the i-th fused feature information.

[0095] It should be noted that, as can be seen from the foregoing, in the disclosed embodiment, the multi-level convolution processing can be: in order from low to high scales, sequentially performing temporal convolution processing and fusion according to the cyclic memory parameter, down-sampled data, initial feature information, and fused feature information of the previous scale, and finally fusing and convolving with the video frame to be processed to obtain a target object mask. It can be seen that the multi-level convolution processing includes sequential convolution processing and conventional convolution processing.

[0096] Specifically, semantic segmentation is first performed on the initial feature information with the smallest scale, and the result is recorded as semantic feature information. After the semantic feature information and the corresponding cyclic memory parameter are subjected to temporal convolution processing, the result is recorded as fused feature information of the corresponding scale, so as to be fused with the features of the next scale.

[0097] The processing of semantic feature information can be performed by a bottleneck module, which includes a temporal convolution module. When processing, the temporal convolution module is combined with the corresponding cyclic memory parameters to take into account the temporal correlation between video frames, so that the obtained fusion feature information is not only the processing result of the current video frame to be processed, but also combines the information of the previous frame or the previous frames. At the same time, the temporal convolution module will also update the corresponding cyclic memory parameters in combination with the current video frame to be processed for use by the next video frame to be processed in the video to be processed.

[0098] More specifically, after the semantic feature information is input into the bottleneck module, the semantic feature information can be split according to the feature channels, and the semantic feature information of one part of the channels (for example, half of the channels) is not processed. The unprocessed part of the semantic feature information is recorded as the first semantic data, and the semantic feature information of another part of the channels (for example, the other half of the channels) and the corresponding cyclic memory parameters are sent to the temporal convolution module for processing. The part of the semantic feature information sent to the temporal convolution module is recorded as the second semantic data. The temporal convolution module processes and outputs the updated cyclic memory parameters and temporal semantic data, and then the obtained temporal semantic data is spliced ​​with the first semantic data to obtain the corresponding fused feature information.

[0099] Then, upsampling, feature fusion, and time series convolution processing are performed step by step from small to large scales to restore the scales step by step. For the example of N1=3, N2=2, and 3 scales of 1 / 4, 1 / 8, and 1 / 16, the third fused feature information and the updated third loop memory parameters are obtained through the bottleneck module. Among them, upsampling and time series convolution processing can be performed in the upsampling module. In this example, there are 2 upsampling modules, and each upsampling module contains a time series convolution module.

[0100] Then, i=2, j=2, the corresponding upsampling module receives: the third fused feature information (whose scale is 1 / 16, output by the bottleneck module), the second initial feature information (whose scale is 1 / 8), the second downsampled data (whose scale is 1 / 8), and the second cyclic memory parameter (corresponding to the scale of 1 / 8). Similar to the processing of the aforementioned bottleneck module, the features are also split according to the feature channel. Specifically, the third fused feature information is first upsampled to obtain the second upsampled time series feature with a scale of 1 / 8, so that the scales between the features are unified. The second upsampled time series feature, the second initial feature information, and the second downsampled data, all of which have a scale of 1 / 8, are spliced ​​to obtain the second merged feature information, and then the splitting is performed, for example, split in half according to the feature channel, and recorded as the second first spliced ​​data and the second second spliced ​​data, respectively.

[0101] Among them, the second first concatenated feature data is sent to a common convolution decoding module (or convolution decoding block) for conventional convolution decoding processing. For example, in the convolution decoding module, it passes through the convolution layer, batch normalization layer and activation layer in sequence to obtain the second convolution feature data. The second second concatenated data and the corresponding second cyclic memory parameters are sent to the time series convolution module for processing to obtain the corresponding second time series output data and the updated second cyclic memory parameters. The second time series output data and the second convolution feature data with a scale of 1 / 8 are concatenated to obtain the second fusion feature information with a scale of 1 / 8.

[0102] Then, i=1, j=1, the corresponding upsampling module receives: the second fused feature information (whose scale is 1 / 8, output by the previous upsampling module), the first initial feature information (whose scale is 1 / 4), the first downsampled data (whose scale is 1 / 4), and the first cyclic memory parameter (corresponding to the scale of 1 / 4). Similarly, the features are split here according to the feature channel. Specifically, the second fused feature information is first upsampled to obtain the first upsampled time series feature with a scale of 1 / 4, so that the scales between the features are unified. The first upsampled time series feature, the first initial feature information, and the first downsampled data, all of which have a scale of 1 / 4, are spliced ​​to obtain the first merged feature information, and then the splitting is performed, for example, split in half according to the feature channel, and recorded as the first first spliced ​​data and the first second spliced ​​data, respectively.

[0103] Among them, the first first concatenated feature data is sent to a common convolution decoding module for conventional convolution decoding processing. For example, in the convolution decoding module, it passes through a convolution layer, a batch normalization layer, and an activation layer in sequence to obtain the first convolution feature data. The first second concatenated data and the corresponding first cyclic memory parameter are sent to a temporal convolution module for processing to obtain the corresponding first temporal output data and the updated first cyclic memory parameter. The first temporal output data and the first convolution feature data, both of which have a scale of 1 / 4, are concatenated to obtain the first fused feature information with a scale of 1 / 4.

[0104] The first fused feature information is fused with the video frame to be processed to obtain the target object mask. It can be understood that in order to ensure the uniformity of scale, the first fused feature information is still upsampled first to make its scale consistent with the video frame to be processed (if it is a pre-processed video frame to be processed, it is consistent with the scale of the pre-processed video frame to be processed), and then spliced ​​for feature fusion and convolution processing through the convolution decoding module, so that the target object can be effectively distinguished from the background to obtain the target object mask.

[0105] It can be understood that here only N1=3 and N2=2 are taken as examples. If it is other numbers, the number of iterative sampling increases and the method remains unchanged. For example, to directly fuse the features of 1 / 16 scale and 1 / 4 scale, it is only necessary to change the sampling factor during upsampling.

[0106] Alternatively, in some other embodiments, feature fusion and temporal convolution processing are performed according to the initial feature information of the current scale, the downsampled data, the cyclic memory parameter, and the fused feature information of the previous scale to obtain the fused feature information of the current scale, including:

[0107] Upsample the fused feature information of the previous scale to obtain the upsampled time series features of the current scale;

[0108] The upsampled time series features, initial feature information, and downsampled data of the current scale are concatenated to obtain the merged feature information of the current scale;

[0109] Perform convolution decoding on the combined feature information of the current scale to obtain combined convolution feature data;

[0110] Split the merged convolution feature data of the current scale into first convolution data and second convolution data;

[0111] Performing time series convolution processing on the second convolution data and the loop memory parameters of the current scale to obtain time series output data of the current scale and obtain updated loop memory parameters corresponding to the current scale;

[0112] The first convolution data and the time series output data are concatenated to obtain the fused feature information of the current scale.

[0113] It should be noted that the embodiment of the present disclosure can also perform convolution decoding processing on the merged feature information and then split it (such as still splitting it in half according to the channel), retaining part of the features after the split, and performing time series convolution processing on the other part of the features, and then merging them as the final fusion feature information.

[0114] In this way, since the cyclic memory parameters are used to record the temporal feature information between adjacent video frames and processed by the temporal convolution module, the temporal characteristics between video frames are fully considered, and the temporal convolution module is added to perform information transmission between video frames. It can learn rich background information as the target moves, thereby improving the stability and robustness of video segmentation and achieving high-precision segmentation.

[0115] In some embodiments, the target object mask image is a grayscale image, and the method may further include:

[0116] The target object mask image is binarized to segment the target object area and the background area.

[0117] It should be noted that binarization is an image processing technology that sets the grayscale value of the pixel points on the image to 0 or 255, so that the entire image presents obvious black and white colors, thereby simplifying the image data and facilitating subsequent analysis and processing. For example, for the image segmentation corresponding to the embodiment of the present disclosure: after binarization, the image can be clearly divided into two parts: the target object and the background, making the distinction between the target object and the background clearer, which is convenient for subsequent image analysis and processing. At the same time, if pre-processing is performed, resizing can also be performed to restore the size to the size of the video frame to be processed.

[0118] The disclosed embodiment provides a video image segmentation method, which, on the one hand, fully considers the timing characteristics between video frames and adds a timing convolution module to conduct information between video frames. It can learn rich background information as the target moves, thereby improving the stability and robustness of video portrait segmentation and achieving high-precision portrait segmentation; on the other hand, this method also has the advantage of fast processing speed and can be easily deployed on edge devices. In short, this technology does not require any auxiliary information and has the advantages of high accuracy, strong stability, and fast processing speed.

[0119] Based on the above embodiment, in another embodiment of the present disclosure, see Figure 2 , which shows a detailed flow chart of an image segmentation method provided by an embodiment of the present disclosure. Figure 2 As shown, the detailed process includes:

[0120] Step 1: Read the video to be processed, which can be the real-time picture taken by the camera or the local video clip, obtain continuous video sequence frames, and initialize the loop memory parameters r2, r3 and r4, whose initial value is a zero-value tensor with a shape of [1,1,1,1].

[0121] It should be noted that the video frame to be processed is a video frame in the video to be processed, and these video frames are continuous, so the video frame to be processed is also called a video sequence frame. In this embodiment, it is still assumed that N1=3, and the corresponding three loop memory parameters are recorded as r2, r3 and r4 according to the scale from small to large (1 / 16, 1 / 8, 1 / 4). Here, [1,1,1,1] refers to the tensor size (shape), which is used to store the loop memory parameters. The initial value of the loop memory parameter is generally set to 0, which is an empirical value.

[0122] It should also be noted that, taking the target object as a portrait, that is, performing portrait segmentation as an example, the image segmentation process provided in the embodiment of the present disclosure can be based on Figure 3 The video portrait segmentation network is shown in Figure 1. Figure 3 As shown, the video portrait segmentation network may include a multi-scale feature encoder 301 and an iterative decoder 302 , and may also include a pre-processing module 303 and a post-processing module 304 .

[0123] Step 2: Convert the video sequence frames ( Figure 3 The input video frame in the video sequence is sent to the pre-processing module 303 to obtain the input information s0 of the video portrait segmentation network (i.e., the video frame to be processed / video sequence frame after pre-processing). Among them, the functions of the pre-processing module 303 mainly include: color space conversion, size scaling, and normalization operations, etc., which requires converting the color space of the video sequence frame into RGB; then scaling to a certain resolution H×W size, where H represents the height of the scaled video sequence frame, and W represents the width of the scaled video sequence frame. For example, the height and width of the video sequence frame before scaling are: 1080×1920, and the H×W of the scaled video sequence frame is 512×256; and the pixels are normalized so that the pixel value of the video sequence frame is a floating point decimal controlled within the range of 0 to 1, thereby obtaining the input information s0 of the video portrait segmentation network. Among them, the size scaling is to avoid excessive calculation caused by excessive size of the video frame to be processed, and it needs to be scaled to a suitable size to reduce the amount of calculation. Pixel normalization can be to divide the actual pixel value by the maximum pixel value (255).

[0124] Step 3: Send the input information s0 and the loop memory parameters r2, r3, and r4 into the video portrait segmentation network for temporal and spatial feature extraction, and perform corresponding feature decoding to obtain updated loop memory parameters r2, r3, and r4, and output the portrait mask information fout (i.e., the target object mask map, for portraits, recorded as portrait mask information fout).

[0125] like Figure 3 As shown, the video portrait segmentation network mainly includes two modules: a multi-scale feature encoder 301 (or multi-scale feature encoding module, feature encoder, feature encoding module, etc.) and an iterative decoder 302 (or iterative decoding module, etc.). The multi-scale feature encoder 301 also includes a feature extraction module 3011, a pooling module 3012, and a semantic segmentation module 3013. Among them, the feature extraction module 3011 can use MobileNetV3-Large or MobileNetV3-Small as the backbone network to extract multi-scale features, but is not limited to this.

[0126] Corresponding to N2 being 2, the pooling module 3012 includes two parts, pooling layer 1 and pooling layer 2. Pooling layer 1 can be an average pooling layer with a pooling window of 4 and a step size of 4, and pooling layer 2 can be an average pooling layer with a pooling window of 2 and a step size of 2. Corresponding to N2 being other values, the number of pooling layers changes accordingly, and the pooling window and pooling step size also change accordingly.

[0127] The semantic segmentation module 3013 may be, for example, Figure 3 The LR-ASPP module (or LA-ASPP semantic segmentation network) shown in the figure uses the lightweight convolutional network MobileNetV3 as the backbone network. The RGB image is processed by the LR-ASPP module to obtain pixel-level semantic information, which is used to remove feature points on a priori dynamic objects in the image. At the same time, since the LR-ASPP module is a very lightweight semantic segmentation head with a high speed, it can be easily run on mobile terminals, edge devices, and some low-power devices, which is conducive to improving the computing speed and reducing the requirements for device computing power.

[0128] The iterative decoder 302 is mainly used to aggregate time information and fuse the features extracted by the multi-scale feature encoder 301 to minimize information loss. Figure 3As shown, the iterative decoder 302 mainly includes a bottleneck (BottleNeck) module 3021, an upsampling module 3022 (the two upsampling modules are respectively recorded as upsampling module 1 and upsampling module 2) and an output convolution module 3023. Among them, the bottleneck module 3021 and the upsampling module 3022 both include a temporal convolution (ConvGRU) module to capture the spatiotemporal correlation in the time series. The temporal convolution module has built-in temporal information, which can retain the feature information of the previous frame and the current frame, so that information can be transferred between the previous and next frames, thereby improving the segmentation effect. It can be understood that corresponding to N1=3, except for the initial feature information of the minimum scale, the other two initial feature information correspond to an upsampling module respectively. For N1 with other values, the number of upsampling modules 3022 also changes accordingly.

[0129] like Figure 4 As shown, the bottleneck module 3021 is composed of a ConvGRU module 3025, and may also include a channel splitting module 3026 and a splicing module 3027, which are used to split feature information and splice feature information respectively. After the feature is input, the feature will be segmented in the channel dimension, and the data of a part of the channel will keep the original information, and the other part of the channel data will be sent to the ConvGRU module 3025 for processing, and finally the results of the two parts of the channel information will be spliced ​​together as the feature output of this module.

[0130] like Figure 5 As shown, the upsampling module 3022 is composed of an upsampling layer 1 with a sampling factor of 2, a normal convolutional decoding module 3028 and a ConvGRU module 3025, and also includes two splicing modules 3027 and a channel splitting module 3026. The normal convolutional decoding module 3028 can be composed of a convolutional layer with a convolution kernel of 3 and a step size of 1 (Conv2d 3×3 step size=1), a batch normalization layer and a ReLU activation layer. When processing feature information, it is still divided in the channel dimension, a part of the channel features are sent to the normal convolutional decoding module 3028 for processing, and the remaining part of the channel features are sent to the ConvGRU module 3025 for processing, and finally the results of the two parts of the channel information after processing are spliced ​​as the output feature information.

[0131] like Figure 6 As shown, the output convolution module 3023 is composed of an upsampling layer 2 with a sampling factor of 4, a convolution decoding module 3029, and a convolution layer with a convolution kernel of 1 and a stride of 1 (Conv2d 1×1 stride=1). The convolution decoding module 3029 is composed of 2 convolution layers with a convolution kernel of 3 and a stride of 1 (Conv2d 3×3 stride=1), 2 batch normalization layers, and 2 ReLU activation layers. This module will output the portrait mask information fout.

[0132] It should be noted that the sampling factor of each sampling layer is related to the actual scale, etc. The convolution kernel, compensation, etc. of each convolution layer can also be set according to the actual situation, as well as the specific composition of each module. What is shown here is only an example and not a specific limitation.

[0133] Step 4: Post-process the portrait mask information fout to obtain the final portrait mask. Figure 7 shown.

[0134] The specific calculation method is as follows: traverse all values ​​(i.e., pixel values) of the portrait mask information fout, set the values ​​less than the set threshold to 0, and vice versa to 255, and then adjust the size of the reassigned fout to the size of the video sequence frame. At the same time, the data type can also be modified to uint8 (unsigned 8-bit integer data type) to obtain the final portrait mask.

[0135] Step 5: Repeat steps 2-4 until all video sequence frames of the current video (i.e., the video to be processed) are processed. If there is continuous video input, the processing is continued, or the processing is suspended according to the actual settings.

[0136] Specifically, combined Figures 2 to 6 As shown, step 3 can be further divided into the following steps:

[0137] Step 3-1: Figure 3 As shown, the input information s0 is sent to the feature extraction module 3011 in the multi-scale feature encoder 301 for feature extraction to obtain initial feature information f2, f3 and f4 of different scales, and their corresponding scales are 1 / 4, 1 / 8, and 1 / 16 respectively; on the other hand, it is sent to the pooling module 3012 for downsampling. After s0 passes through the pooling layer 1, the downsampled data s2 (i.e., the first downsampled data) of the first stage (i.e., the first pooling) is obtained. After s2 passes through the pooling layer 2, the downsampled data s3 (i.e., the second downsampled data) of the second stage (i.e., the second pooling) is obtained.

[0138] Step 3-2: Figure 3 As shown, the initial feature information f4 of 1 / 16 scale is sent to the LR-ASPP module 3013 to obtain higher-level semantic feature information f4';

[0139] Step 3-3, such as Figure 3 or Figure 4As shown, the semantic feature information f4` and the cyclic memory parameter r4 of the 1 / 16 scale are sent to the bottleneck module 3021, and the semantic feature information f4` is split in half in the channel dimension, and the data of half of the channels retain the original feature information f4`_a (i.e., the first semantic data), and the other half of the channel data (i.e., the second semantic data) and the cyclic memory parameter r4 are sent to the ConvGRU module 3025 for processing to obtain feature information f4`_b (i.e., temporal semantic data) and the cyclic memory parameter r4 updated at the 1 / 16 scale, and then the feature information of the first semantic data f4`_a and the temporal semantic data f4`_b are spliced ​​together according to the original channels to obtain new feature information f4`` (i.e., the corresponding fused feature information), and finally the fused feature information f4`` and the updated cyclic memory parameter r4 of the 1 / 16 scale are output from the bottleneck module;

[0140] Step 3-4, such as Figure 3 or Figure 5 As shown, the initial feature information f3 (i.e. Figure 5 The features output by the multi-scale feature encoder in the middle), the 1 / 16 size fusion feature information f4`` (i.e. Figure 5 The feature information output by the previous module in the middle), the 1 / 8 scale down-sampled data s3 and the cyclic memory parameter r3 are sent to the up-sampling module 1. First, the 1 / 16 scale fused feature information f4`` is passed through the up-sampling layer to obtain feature information with a size doubled (1 / 8 scale sampling timing feature). At this time, the size of this feature information and the size of f3 and s3 are consistent (all 1 / 8 scale). Then, this feature information and the data of f3 and s3 are spliced ​​together as the input of the ordinary convolution decoding module 3028 to obtain the 1 / 8 scale merged feature information f3`, and f3` is cut in half in the channel dimension. The data of half of the channels (first spliced ​​data) are sent to the common convolution decoding module for processing to obtain feature information f3`_a (convolution feature data), and the data of the other half of the channels (second spliced ​​data) and the cyclic memory parameter r3 are sent to the ConvGRU module 3025 for processing to obtain feature information f3`_b (time series output data) and the cyclic memory parameter r3 updated at a scale of 1 / 8, and then the feature information of f3`_a and f3`_b are spliced ​​together according to the original channels to obtain new feature information f3`` (fused feature information at a scale of 1 / 8), and finally f3`` and r3 are output from the upsampling module 1;

[0141] Steps 3-5: Figure 3 or Figure 5 As shown, the initial feature information f2 (i.e. Figure 5 The features output by the multi-scale feature encoder in the middle), the fused feature information f3`` of 1 / 8 scale (i.e. Figure 5The feature information output by the previous module in the above example), the downsampled data s2 and the cyclic memory parameter r2 are sent to the upsampling module 2. First, f3`` passes through the upsampling layer to obtain feature information with a size doubled (1 / 4 scale upsampled time series feature). At this time, the size of this feature information and the size of f2 and s2 are consistent, both of which are 1 / 4 scale. Then, this feature information and the data of f2 and s2 are spliced ​​together to obtain the merged feature information f2` of 1 / 4 scale. f2` is split in half in the channel dimension, and the data of half of the channel is used as The input of the ordinary convolution decoding module 3028 is used to obtain feature information f2`_a (convolution feature data), and the other half of the channel data and r2 are sent to the ConvGRU module 3025 for processing to obtain feature information f2`_b (time series output data) and the cyclic memory parameter r2 updated at the 1 / 4 scale, and then the feature information of f2`_a and f2`_b is spliced ​​together according to the original channel to obtain new feature information f2`` (1 / 4 scale fusion feature information), and finally f2`` and r2 are output from the upsampling module 2;

[0142] Steps 3-6, such as Figure 3 or Figure 6 As shown, the 1 / 4 scale fused feature information f2`` and s0 are sent to the output convolution module 3023. First, f2`` passes through the upsampling layer 2 to obtain feature information with a size twice as large. At this time, the size of this feature information is consistent with the size of s0. Then, the feature information and s0 data are concatenated as the input of the convolution decoding module 3029 to obtain the hidden feature information f0. Then, f0 passes through the last convolution layer to obtain the portrait mask information fout.

[0143] In addition, in this embodiment, in order to reduce the amount of calculation, the feature information of 1 / 2 scale is not used. In practice, it is also possible to combine the actual needs so that the feature information of each scale can participate in the multi-level convolution processing process to further improve the image segmentation accuracy.

[0144] In summary, the disclosed embodiment proposes a real-time, high-precision video portrait segmentation method in a video scene. The method does not require any auxiliary information, has the advantages of high precision, strong stability, and fast processing speed, and can meet the needs of real-time video portrait segmentation, and can be well applied to scenes with complex backgrounds and complex character movements; this method adds a temporal convolution module to conduct information between video sequence frames, which can learn rich background information as the target moves, thereby improving the stability and robustness of video portrait segmentation and achieving high-precision portrait segmentation; and it also has the advantage of fast processing speed, and can be easily deployed on edge devices, for example, the inference speed on the Jetson development board can reach 76 frames per second (fps). It can meet the needs of real-time video portrait segmentation, and can be well applied to scenes with complex backgrounds and complex character movements.

[0145] In another embodiment of the present disclosure, see Figure 8 , which shows a schematic diagram of the composition structure of a video image segmentation device provided by an embodiment of the present disclosure. Figure 8 As shown, the video image segmentation device 30 includes:

[0146] The feature encoding module 301 is used to perform multi-scale feature extraction on the video frame to be processed to obtain N1 initial feature information corresponding to N1 scales; perform multi-scale downsampling on the video frame to be processed to obtain N2 downsampling data; wherein the video frame to be processed is an image frame in the video to be processed;

[0147] Iterative decoding module 302, used for performing multi-level convolution processing according to N1 loop memory parameters, N1 initial feature information, and N2 down-sampled data to obtain a target object mask map; the N1 loop memory parameters correspond to the N1 scales one by one, the loop memory parameters carry the temporal feature information of the previous frame or multiple frames of the video to be processed corresponding to the scale, and the target object mask map includes the target object area and the background area in the video frame to be processed;

[0148] Among them, N1 and N2 are both positive integers.

[0149] It should be noted that Figure 3 The video portrait segmentation network shown is a specific implementation of the video image segmentation device 30. Figure 3 As shown, in some embodiments, the video image segmentation device 30 may further include: a pre-processing module 303, for pre-processing the video frame to be processed; wherein the pre-processing includes at least one of the following processing methods: color space conversion, size scaling, and normalization operation.

[0150] In some embodiments, Figure 3As shown, the feature encoding module 301 includes a feature extraction module 3011, a pooling module 3012, and a semantic segmentation module 3013; wherein:

[0151] A feature extraction module 3011 is used to perform multi-scale feature extraction on the video frame to be processed to obtain N1 initial feature information corresponding to N1 scales;

[0152] A pooling module 3012 is used to perform multi-scale downsampling on the video frame to be processed to obtain N2 downsampled data;

[0153] A semantic segmentation module 3013 is used to perform semantic segmentation on the initial feature information of the first scale to generate semantic feature information; the first scale is the smallest one among the N1 scales;

[0154] The iterative decoding module 302 includes a bottleneck module 3021, an output convolution module 3023 and at least one up-sampling module 3022. In the order of small to large scale, at least one up-sampling module 3022 is cascaded in sequence, wherein:

[0155] The bottleneck module 3021 is used to perform temporal convolution processing on the semantic feature information and the recurrent memory parameter of the first scale to obtain fused feature information of the first scale;

[0156] The up-sampling module 3022 is used to perform feature fusion and temporal convolution processing according to the initial feature information of the current scale, the down-sampled data, the cyclic memory parameter and the fused feature information of the previous scale to obtain the fused feature information of the current scale; until the fused feature information of the second scale is obtained; the second scale is the largest one among the N1 scales;

[0157] The output convolution module 3023 is used to perform feature fusion and convolution processing on the fused feature information of the second scale and the video frame to be processed to obtain a target object mask image.

[0158] In some embodiments, Figure 4 As shown, the bottleneck module 3021 includes:

[0159] A channel splitting module 3026, used for splitting the semantic feature information into first semantic data and second semantic data;

[0160] A temporal convolution module 3025 is used to perform temporal convolution processing on the second semantic data and the cyclic memory parameters of the first scale to obtain temporal semantic data and obtain updated cyclic memory parameters corresponding to the first scale;

[0161] The splicing module 3027 is used to splice the first semantic data and the temporal semantic data to obtain fusion feature information corresponding to the first scale.

[0162] In some embodiments, Figure 5 As shown, the up-sampling module 3022 includes:

[0163] Upsampling layer 1 is used to upsample the fusion feature information of the previous scale to obtain the upsampled time series features of the current scale;

[0164] The first splicing module 3027 is used to splice the upsampled time series features, the initial feature information, and the downsampled data of the current scale to obtain the merged feature information of the current scale;

[0165] A channel splitting module 3026 is used to split the merged feature information of the current scale into first spliced ​​data and second spliced ​​data;

[0166] The common convolution decoding module 3028 (or called the common convolution decoding block) is used to perform convolution decoding processing on the first spliced ​​data to obtain convolution feature data;

[0167] A time series convolution module 3025 is used to perform time series convolution processing on the second spliced ​​data and the cyclic memory parameters of the current scale to obtain time series output data of the current scale and obtain updated cyclic memory parameters corresponding to the current scale;

[0168] The second splicing module 3027 is used to splice the convolution feature data and the time series output data to obtain the fusion feature information of the current scale.

[0169] In some embodiments, the temporal convolution module 3025 is also used to update the loop memory parameters of the corresponding scale so that the loop memory parameters carry the temporal feature information of the current video frame to be processed at the corresponding scale; the updated loop memory parameters are used for multi-level convolution processing of the next video frame to be processed in the video to be processed.

[0170] In some embodiments, Figure 3 As shown, the video image segmentation device 30 further includes a post-processing module 304 for performing a binarization process on the target object mask image to segment the target object area and the background area.

[0171] It should be noted that the video image segmentation device 30 provided in the embodiment of the present disclosure is used to implement the video image segmentation in the aforementioned embodiment. For details not disclosed in this embodiment, please refer to the description of the aforementioned embodiment for understanding, and no further details will be given here.

[0172] It is understandable that in this embodiment, each component can be integrated into a processing unit, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of software function modules.

[0173] If the integrated module is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0174] Therefore, this embodiment provides a computer storage medium, which stores a computer program. When the computer program is executed by at least one processor, the steps of the video image segmentation method described in any one of the above embodiments are implemented.

[0175] This embodiment further provides a computer program product, which includes a computer program. When the computer program is executed by at least one processor, the steps of the video image segmentation method described in any one of the above embodiments are implemented.

[0176] Based on the above computer storage medium and computer program product, see Fig. 9 , which shows a schematic diagram of the composition structure of an electronic device 90 provided by an embodiment of the present disclosure. Fig. 9 As shown, the electronic device 90 may include: a communication interface 901, a memory 902 and a processor 903; each component is coupled together via a bus system 904. It is understood that the bus system 904 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 904 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Fig. 9 Various buses are labeled as bus system 904. Among them, the communication interface 901 is used for receiving and sending signals in the process of sending and receiving information between other external network elements;

[0177] Memory 902, used to store computer programs that can be run on processor 903;

[0178] The processor 903 is configured to execute any of the aforementioned video image segmentation methods when running the computer program.

[0179] It can be understood that the memory 902 in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The memory 702 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0180] The processor 703 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 703. The above processor 703 may be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 702, and the processor 703 reads the information in the memory 702 and completes the steps of the above method in conjunction with its hardware.

[0181] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present disclosure, or a combination thereof.

[0182] For software implementation, the techniques described herein can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0183] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the protection scope of the present disclosure.

[0184] It should be noted that in the present disclosure, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0185] The serial numbers of the above-mentioned embodiments of the present disclosure are only for description and do not represent the advantages or disadvantages of the embodiments.

[0186] The methods disclosed in several method embodiments provided in the present disclosure can be arbitrarily combined without conflict to obtain new method embodiments.

[0187] The features disclosed in several product embodiments provided in the present disclosure can be arbitrarily combined without conflict to obtain new product embodiments.

[0188] The features disclosed in several method or device embodiments provided in the present disclosure may be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0189] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A video image segmentation method, characterized in that: The method comprises: Obtain a video frame to be processed and N1 loop memory parameters; the video frame to be processed is an image frame in the video to be processed, the N1 loop memory parameters correspond to the N1 scales one by one, and the loop memory parameters carry temporal feature information of the scales corresponding to the previous frame or multiple frames of the video to be processed; Performing multi-scale feature extraction on the video frame to be processed to obtain N1 initial feature information corresponding to the N1 scales; Perform multi-scale downsampling on the video frame to be processed to obtain N2 downsampled data; Performing multi-level convolution processing according to the N1 loop memory parameters, the N1 initial feature information, and the N2 down-sampled data to obtain a target object mask map; the target object mask map includes a target object area and a background area in the video frame to be processed; Among them, N1 and N2 are both positive integers.

2. The method according to claim 1, characterized in that Before performing multi-scale feature extraction on the video frame to be processed, the method further includes: Pre-processing is performed on the video frame to be processed; wherein the pre-processing includes at least one of the following processing methods: color space conversion, size scaling, and normalization operation.

3. The method according to claim 1, characterized in that The method further comprises: performing semantic segmentation on the initial feature information of a first scale to generate semantic feature information; the first scale is the smallest one of the N1 scales; Accordingly, a multi-level convolution process is performed according to the N1 cyclic memory parameters, the N1 initial feature information, and the N2 down-sampled data to obtain a target object mask map, including: Performing temporal convolution processing on the semantic feature information and the cyclic memory parameter of the first scale to obtain fused feature information of the first scale; According to the order of the scales from small to large, feature fusion and temporal convolution processing are performed according to the initial feature information of the current scale, the downsampled data, the cyclic memory parameter and the fused feature information of the previous scale to obtain the fused feature information of the current scale; until the fused feature information of the second scale is obtained; the second scale is the largest one of the N1 scales; The fused feature information of the second scale and the video frame to be processed are subjected to feature fusion and convolution processing to obtain the target object mask image.

4. The method according to claim 3, characterized in that Performing temporal convolution processing on the semantic feature information and the cyclic memory parameter of the first scale to obtain fused feature information of the first scale includes: Splitting the semantic feature information into first semantic data and second semantic data; Performing temporal convolution processing on the second semantic data and the cyclic memory parameter of the first scale to obtain temporal semantic data, and obtaining the updated cyclic memory parameter corresponding to the first scale; The first semantic data and the temporal semantic data are concatenated to obtain fused feature information corresponding to the first scale.

5. The method according to claim 3, characterized in that: N1 is greater than N2 by 1, and except for the first scale, the initial feature information of the remaining scales corresponds to the scale of each downsampled data one by one; feature fusion and temporal convolution processing are performed according to the initial feature information of the current scale, the downsampled data, the cyclic memory parameter, and the fused feature information of the previous scale to obtain the fused feature information of the current scale, including: Upsampling the fused feature information of the previous scale to obtain upsampled time series features of the current scale; The upsampled time series features of the current scale, the initial feature information, and the downsampled data are concatenated to obtain the merged feature information of the current scale; Splitting the merged feature information of the current scale into first spliced ​​data and second spliced ​​data; Performing convolution decoding processing on the first spliced ​​data to obtain convolution feature data; Performing time series convolution processing on the second spliced ​​data and the cyclic memory parameter of the current scale to obtain time series output data of the current scale, and obtaining the updated cyclic memory parameter corresponding to the current scale; The convolution feature data and the time series output data are concatenated to obtain the fusion feature information of the current scale.

6. The method according to any one of claims 1 to 5, characterized in that: During the multi-stage convolution processing, the method further includes: The loop memory parameters of each scale are updated so that the loop memory parameters carry the temporal feature information of the current video frame to be processed at the corresponding scale; the updated loop memory parameters are used for multi-level convolution processing of the next video frame to be processed in the video to be processed.

7. The method according to any one of claims 1 to 5, characterized in that: The target object mask image is a grayscale image, and the method further includes: The target object mask image is binarized to segment the target object area and the background area.

8. An image processing device, characterized in that: include: A feature encoding module is used to extract multi-scale features from the video frame to be processed, and obtain N1 initial feature information corresponding to N1 scales; Perform multi-scale downsampling on the video frame to be processed to obtain N2 downsampled data; wherein the video frame to be processed is an image frame in the video to be processed; an iterative decoding module, configured to perform multi-stage convolution processing according to N1 loop memory parameters, the N1 initial feature information, and the N2 down-sampled data to obtain a target object mask image; the N1 loop memory parameters correspond one-to-one to the N1 scales, the loop memory parameters carry temporal feature information of the previous frame or multiple frames of the video to be processed corresponding to the scale, and the target object mask image includes a target object area and a background area in the video frame to be processed; Among them, N1 and N2 are both positive integers.

9. The image processing device according to claim 8, characterized in that: The image processing device further comprises: The pre-processing module is used to pre-process the video frame to be processed; wherein the pre-processing includes at least one of the following processing methods: color space conversion, size scaling, and normalization operation.

10. The image processing device according to claim 8, characterized in that: The feature encoding module includes a feature extraction module, a pooling module and a semantic segmentation module; the iterative decoding module includes a bottleneck module, an output convolution module and at least one upsampling module, and the at least one upsampling module is cascaded in sequence in order of scale from small to large; wherein: The feature extraction module is used to perform multi-scale feature extraction on the video frame to be processed to obtain N1 initial feature information corresponding to N1 scales; The pooling module is used to perform multi-scale downsampling on the video frame to be processed to obtain N2 downsampled data; The semantic segmentation module is used to perform semantic segmentation on the initial feature information of the first scale to generate semantic feature information; the first scale is the smallest one among the N1 scales; The bottleneck module is used to perform temporal convolution processing on the semantic feature information and the cyclic memory parameters of the first scale to obtain fused feature information of the first scale; The upsampling module is used to perform feature fusion and temporal convolution processing according to the initial feature information of the current scale, the downsampled data, the cyclic memory parameter and the fused feature information of the previous scale to obtain the fused feature information of the current scale; until the fused feature information of the second scale is obtained; the second scale is the largest one among the N1 scales; The output convolution module is used to perform feature fusion and convolution processing on the fused feature information of the second scale and the video frame to be processed to obtain a target object mask image.

11. The image processing device according to claim 10, characterized in that: The bottleneck module comprises: A channel splitting module, used for splitting the semantic feature information into first semantic data and second semantic data; A temporal convolution module, used for performing temporal convolution processing on the second semantic data and the cyclic memory parameters of the first scale to obtain the temporal semantic data and obtain the updated cyclic memory parameters corresponding to the first scale; The splicing module is used to splice the first semantic data and the temporal semantic data to obtain fusion feature information corresponding to the first scale.

12. The image processing device according to claim 10, characterized in that: N1 is greater than N2 by 1, and except for the first scale, the initial feature information of the remaining scales corresponds to the scale of each down-sampled data one by one; the up-sampling module includes: The upsampling layer is used to upsample the fused feature information of the previous scale to obtain the upsampled time series features of the current scale; The first splicing module is used to splice the upsampled time series features, initial feature information, and downsampled data of the current scale to obtain the merged feature information of the current scale; A channel splitting module, used for splitting the merged feature information of the current scale into first spliced ​​data and second spliced ​​data; A common convolution decoding module, used for performing convolution decoding processing on the first spliced ​​data to obtain convolution feature data; A time series convolution module, used for performing time series convolution processing on the second spliced ​​data and the cyclic memory parameters of the current scale to obtain time series output data of the current scale and obtain updated cyclic memory parameters corresponding to the current scale; The second splicing module is used to splice the convolution feature data and the time series output data to obtain the fused feature information of the current scale.

13. The image processing device according to claim 11 or 12, characterized in that: The temporal convolution module is further used to update the cyclic memory parameters of the corresponding scale, so that the cyclic memory parameters carry the temporal feature information of the current video frame to be processed at the corresponding scale; The updated loop memory parameters are used for multi-stage convolution processing of the next video frame to be processed in the video to be processed.

14. The image processing device according to any one of claims 8 to 12, characterized in that: The image processing device further comprises: The post-processing module is used to perform binarization processing on the target object mask image to segment the target object area and the background area.

15. An electronic device, characterized in that: The electronic device comprises a memory and a processor; wherein, The memory is used to store a computer program that can be run on the processor; The processor is configured to execute the method according to any one of claims 1 to 8 when running the computer program.

16. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by at least one processor, the method according to any one of claims 1 to 8 is implemented.

Images