A segmentation method, device, apparatus and computer readable storage medium

Abstract

The application discloses a segmentation method, device and equipment and a computer readable storage medium. The method comprises the following steps: obtaining video data to be segmented and a trained segmentation model; determining a plurality of video frames included in the video data, and sequentially determining the plurality of video frames as video frames to be segmented; when there is at least one historical video frame before the video frame to be segmented, determining a reference transparency template of the video frame to be segmented from a target transparency template of the at least one historical video frame; inputting the video frame to be segmented and the reference transparency template into the trained segmentation model to obtain a target transparency template of the video frame to be segmented; and performing extraction processing on the video frame to be segmented based on the target transparency template to obtain a target foreground image of the video frame to be segmented. In this way, the spatio-temporal features of the input are increased through the reference transparency template corresponding to the historical video frame, thereby reducing the calculation complexity, improving the accuracy of the target transparency template, and improving the segmentation effect.

Description

Technical Field

[0001] This application relates to the field of image technology, and to, but is not limited to, a segmentation method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the continuous development of network technology, various Internet communication methods have emerged. People's communication needs have changed from the initial single voice needs to the needs of video and audio communication. Video communication services that integrate the transmission of voice data and video data have become a hot spot in the field of communication and are being used more and more widely in areas such as video conferencing, remote video medical care, and remote video education.

[0003] The resulting demands for both entertainment and privacy protection in video communication have led to the development of background replacement functionality. This involves video portrait segmentation methods to extract the portrait area in the video image, and then using image fusion to replace the background.

[0004] Video portrait segmentation algorithms in related technologies do not consider the temporal continuity between consecutive frames, which may cause segmentation jitter, thus affecting the segmentation effect and reducing the segmentation accuracy. Summary of the Invention

[0005] In view of the above, embodiments of this application provide a segmentation method, apparatus, device, and computer-readable storage medium.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides a segmentation method, the method comprising:

[0008] Acquire the video data to be segmented and the trained segmentation model;

[0009] The video data includes multiple video frames, which are then sequentially identified as video frames to be segmented.

[0010] If there is at least one historical video frame before the video frame to be segmented, determine the reference transparency template of the video frame to be segmented from the target transparency template of the at least one historical video frame.

[0011] The video frame to be segmented and the reference transparency template are input into the trained segmentation model to obtain the target transparency template of the video frame to be segmented.

[0012] The target foreground image of the video frame to be segmented is obtained by extracting the target transparency template.

[0013] This application provides a segmentation device, the segmentation device comprising:

[0014] The acquisition module is used to acquire the video data to be segmented and the trained segmentation model;

[0015] The first determining module is used to determine multiple video frames included in the video data, and to sequentially determine the multiple video frames as video frames to be segmented.

[0016] The second determining module is used to determine a reference transparency template of the video frame to be segmented from the target transparency templates of the at least one historical video frame when there is at least one historical video frame before the video frame to be segmented.

[0017] The segmentation module is used to input the video frame to be segmented and the reference transparency template into the trained segmentation model to obtain the target transparency template of the video frame to be segmented.

[0018] The extraction module is used to extract the video frame to be segmented based on the target transparency template to obtain the target foreground image of the video frame to be segmented.

[0019] This application provides a segmentation device, the segmentation device comprising:

[0020] Processor; and

[0021] Memory for storing computer programs that can run on the processor;

[0022] The computer program implements the above-mentioned segmentation method when executed by a processor.

[0023] This application provides a computer-readable storage medium storing computer-executable instructions configured to perform the above-described segmentation method.

[0024] This application provides a segmentation method, apparatus, device, and computer-readable storage medium. The segmentation method includes: first, acquiring video data to be segmented and a trained segmentation model; then, determining multiple video frames included in the video frame data and sequentially identifying these multiple video frames as video frames to be segmented; next, if at least one historical video frame exists before the video frame to be segmented, determining a reference transparency template corresponding to the video frame to be segmented from the target transparency templates corresponding to the at least one historical video frame, where the reference transparency template is the spatiotemporal feature of the video frame to be segmented; then, inputting the video frame to be segmented and the reference transparency template into the trained segmentation model, thereby obtaining the target transparency template of the video frame to be segmented through the output of the trained segmentation model. Thus, when performing segmentation using the trained segmentation model, the video frame to be segmented and its spatiotemporal features are fully considered, simplifying the segmentation process and enabling the rapid acquisition of a highly accurate target transparency template; finally, based on the target transparency template, the video frame to be segmented is extracted to obtain a target foreground image corresponding to the video frame to be segmented, which is more realistic and conforms to the actual situation. Attached Figure Description

[0025] In the accompanying drawings (which are not necessarily drawn to scale), similar reference numerals may describe similar parts in different views. The drawings illustrate, by way of example and not limitation, the various embodiments discussed herein.

[0026] Figure 1 This is a schematic diagram illustrating an implementation flow of the segmentation method provided in an embodiment of this application;

[0027] Figure 2 A schematic diagram illustrating an implementation process for determining a reference transparency template provided in an embodiment of this application;

[0028] Figure 3 This is a schematic diagram illustrating an implementation process for determining a target transparency template provided in an embodiment of this application.

[0029] Figure 4 This is a schematic diagram illustrating an implementation process for determining target semantic information provided in an embodiment of this application.

[0030] Figure 5 A schematic diagram illustrating another implementation flow of the segmentation method provided in the embodiments of this application;

[0031] Figure 6 This is a schematic diagram illustrating an implementation process for determining a trained segmentation model, as provided in an embodiment of this application.

[0032] Figure 7 A schematic diagram illustrating an implementation process for determining a preliminarily trained segmentation model, provided in an embodiment of this application;

[0033] Figure 8 A schematic diagram illustrating another implementation process for determining a trained segmentation model provided in an embodiment of this application;

[0034] Figure 9 A schematic diagram illustrating another implementation of the segmentation method provided in this application embodiment;

[0035] Figure 10 A schematic diagram of an image data transparency template transformation provided in an embodiment of this application;

[0036] Figure 11 A schematic diagram of the composition architecture of the human image segmentation network model provided in this application embodiment;

[0037] Figure 12 This is a schematic diagram of the composition of the segmentation device provided in the embodiments of this application;

[0038] Figure 13 This is a schematic diagram of the composition structure of the segmentation device provided in the embodiments of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0041] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the scope of this application.

[0043] In related technologies, one approach is to determine whether to use a preset model for portrait segmentation by judging the difference between consecutive frames of a video. If the difference is greater than a preset threshold, portrait segmentation is performed on the first image according to the preset model; if the difference is less than or equal to the preset threshold, the portrait segmentation result is determined based on the portrait segmentation result of the second image and a motion estimation algorithm. This scheme considers the spatiotemporal relationship between consecutive frames in the video during the post-processing stage, reducing the computational load to some extent. However, the setting of the difference between consecutive frames is strongly related to the video scene, making it difficult to extend to different scenarios.

[0044] Other methods, such as incorporating dilated convolutions into the MobileNet network model, designing detail enhancement modules, fusing batch normalized nodes with convolutional layers before and after the model, and adjusting downsampling rates, have reduced the computational cost of the model. However, the input to the model is still a single image, and there is no richer prior information for supervision, such as the correlation between consecutive frames in a video. Therefore, it is difficult to obtain better segmentation results when segmenting human figures in video.

[0045] Related technologies also include the following methods: acquiring the image to be processed; performing semantic segmentation on the image to obtain a semantically segmented image, where the semantically segmented image includes target and non-target regions separated by semantic segmentation; then, performing pose recognition on the image to obtain a pose recognition image with identified skeleton regions; next, fusing the target and non-target regions of the semantically segmented image with the skeleton regions of the pose recognition image to obtain a three-part image that divides the foreground region, background region, and region to be recognized; finally, generating a transparency mask image for image separation of the image to be processed based on the image to be processed and the three-part image. While this method obtains a good segmentation result by combining the human skeleton region obtained through pose recognition with the resulting image segmentation to obtain a three-part image, and then obtains the transparency mask image, the computational load for human pose recognition and obtaining the transparency mask from the three-part image is large, the computational process is complex, and it is difficult to apply to real-time computing scenarios.

[0046] Related technologies also propose that electronic devices can update the initial segmentation model to obtain an updated segmentation model. The electronic device can use the updated segmentation model to perform foreground and background segmentation on the image, obtaining a more accurate foreground and background segmentation result. Based on this accurate foreground and background segmentation result, the user can then perform post-processing using a graph cut algorithm with frame skipping to achieve image segmentation. This scheme can achieve high segmentation accuracy, but it requires manual user interaction, making it difficult to apply to real-time communication scenarios and resulting in poor real-time performance.

[0047] To address the problems existing in related technologies, this application provides a segmentation method. This method can be implemented by a computer program, which, when executed, performs the segmentation method provided in this application. In some embodiments, the computer program can be executed by a processor in a segmentation device. Figure 1 An implementation flow of the segmentation method provided in the embodiments of this application is as follows: Figure 1 As shown, the segmentation method includes:

[0048] Step S101: Obtain the video data to be segmented and the trained segmentation model.

[0049] Here, the video data to be segmented can be live video data, conference video data, online training video data, etc. This video data can be collected through monitoring equipment, smartphones, computers, etc., and then stored in cloud storage or local storage. Based on this, the video data to be segmented can be obtained through data location or data content identification commands.

[0050] In this embodiment, the trained segmentation model can be a model based on an artificial intelligence algorithm, such as a neural network algorithm, a genetic algorithm, or a Bayesian network algorithm. In practice, the input to the trained segmentation model includes the video data to be segmented and its corresponding spatiotemporal features. Based on the spatiotemporal features, the segmentation speed and accuracy of the trained segmentation model can be improved, thereby enhancing the segmentation effect.

[0051] Step S102: Determine the multiple video frames included in the video data, and sequentially determine the multiple video frames as video frames to be segmented.

[0052] In practice, a video data set consists of multiple video frames. The multiple video frames included in the video data can be determined by multimedia video processing technology. For example, this multimedia video processing technology can be Fast ForwardMpeg, abbreviated as ffmpeg. ffmpeg is a set of open-source computer programs that can be used to record, convert digital audio and video, and convert them into streams.

[0053] In this embodiment, multiple video frames can be sequentially determined as video frames to be segmented according to their order in the video data; alternatively, they can be determined according to a custom-specified order. This application does not limit the method for determining the video frames to be segmented. In actual implementation, one video frame from the multiple video frames is used as the video frame to be segmented each time, until every video frame has been used as the video frame to be segmented.

[0054] Step S103: When there is at least one historical video frame before the video frame to be segmented, determine the reference transparency template of the video frame to be segmented from the target transparency template of the at least one historical video frame.

[0055] Here, "at least one historical video frame exists before the video frame to be segmented" means that the video frame to be segmented is not the first video frame in the video data, but rather the second video frame or a video frame after the second frame in the video data. Based on this, the number of such at least one historical video frame is at least one, and can be one, two, three, etc.

[0056] In this embodiment, a target historical video frame can be determined from the historical video frames preceding the video frame to be segmented (ni frames) according to default or preset rules. Since there is a sequential relationship between this target historical video frame and the video frame to be segmented, the target transparency template corresponding to this target historical video frame can serve as a reference for the target transparency template of the video frame to be segmented. Therefore, the target transparency template corresponding to this target historical video frame is determined as the reference transparency template for the video frame to be segmented. Here, n is the number of at least one historical video frame, and n is an integer greater than or equal to 1; i is an integer greater than or equal to 0 and less than n.

[0057] Taking n as an example with a value of 10, indicating that the video frame to be segmented is the 11th video frame, then i can be 1, which means that the video frame one frame ahead of the video frame to be segmented is identified as the target historical video frame; i can be 3, which means that the video frame three frames ahead of the video frame to be segmented is identified as the target historical video frame; i can be 5, which means that the video frame five frames ahead of the video frame to be segmented is identified as the target historical video frame.

[0058] In some embodiments, if there are no previous historical video frames before the video frame to be segmented, it indicates that the video frame to be segmented is the first video frame in the video data, and the Y component of the video frame to be segmented is determined as the reference transparency of the video frame to be segmented.

[0059] Step S104: Input the video frame to be segmented and the reference transparency template into the trained segmentation model to obtain the target transparency template of the video frame to be segmented.

[0060] In this embodiment of the application, the encoding format of the video frame to be segmented determined in step S102 can be YUV encoding. Then, the video frame to be segmented includes Y component, U component and V component.

[0061] In step S104, the Y component, U component, V component, and reference transparency template are input into the trained segmentation model. Then, on the one hand, semantic information is extracted from the Y component, U component, V component, and reference transparency template to obtain target semantic information; on the other hand, edge information is extracted from the Y component, U component, V component, and reference transparency template to obtain target edge information. Then, the target semantic information and target edge information are fused to obtain target fusion information. Finally, feature extraction is performed on the target fusion information to obtain the target transparency template corresponding to the video frame to be segmented.

[0062] In practice, the target transparency template can be a black and white image composed of 0s and 1s, where black represents the background image (represented by 0s) and white represents the foreground image (represented by 1s). Furthermore, the size of this target transparency template is the same as the size of the video frame to be segmented.

[0063] Step S105: Extract the target foreground image of the video frame to be segmented based on the target transparency template.

[0064] Here, the target transparency template can be ANDed with the video frame to be segmented, so that the background image has a value of 0, while the target foreground image has a value of the foreground image itself, thereby realizing the extraction and processing of the video frame to be segmented.

[0065] This application provides a segmentation method. First, it acquires the video data to be segmented and a trained segmentation model. Then, it identifies multiple video frames included in the video frame data and sequentially determines these multiple video frames as the video frames to be segmented. Next, if at least one historical video frame exists before the video frame to be segmented, a reference transparency template corresponding to the video frame to be segmented is determined from the target transparency templates corresponding to at least one historical video frame. Here, the reference transparency template represents the spatiotemporal features of the video frame to be segmented. Then, the video frame to be segmented and the reference transparency template are input into the trained segmentation model, thereby obtaining the target transparency template of the video frame to be segmented through the output of the trained segmentation model. Thus, when performing segmentation using the trained segmentation model, the video frame to be segmented and its spatiotemporal features are fully considered, simplifying the segmentation process and enabling the rapid acquisition of a highly accurate target transparency template. Finally, based on the target transparency template, the video frame to be segmented is extracted to obtain the target foreground image corresponding to the video frame to be segmented. This target foreground image is more realistic and conforms to the actual situation.

[0066] In some embodiments, when implementing step S103, "when at least one historical video frame exists before the video frame to be segmented, determine the reference transparency template of the video frame to be segmented from the target transparency template of the at least one historical video frame", such as Figure 2As shown, this can be achieved through the following steps S1031 to S1033:

[0067] Step S1031: Determine the number of frames of at least one historical video frame and record the number of frames as n.

[0068] Here, n is an integer greater than or equal to 1. In some embodiments, any video frame in the video data may include a frame identifier that can characterize the position of that video frame among multiple video frames. For example, if the frame identifier indicates that the video frame to be segmented is at the position of the eleventh frame, then there are ten historical video frames preceding the video frame to be segmented, that is, the number of frames n is 10.

[0069] Step S1032: Determine the historical video frame that precedes the video frame to be segmented (ni frame) as the target historical video frame.

[0070] Here, i is an integer greater than or equal to zero and less than n. Taking n as 10 as an example, theoretically, the value of i can be any integer from 0 to 9. In practice, in order to improve the correlation between the video frame to be segmented and the target historical video frame, and to make the target historical video frame have strong reference significance, the target historical video frame should not be too far away from the video frame to be segmented. That is, the value of i should not be too large. Generally, the value of i can be 1, 2, 3, 5, etc.

[0071] In this embodiment, when i is 1, a historical video frame that is one frame ahead of the video frame to be segmented is determined as the target historical video frame; when i is 2, a historical video frame that is two frames ahead of the video frame to be segmented is determined as the target historical video frame; when i is 3, a historical video frame that is three frames ahead of the video frame to be segmented is determined as the target historical video frame; similarly, when i is 5, a historical video frame that is five frames ahead of the video frame to be segmented is determined as the target historical video frame.

[0072] In actual implementation, i can be determined first, and then the historical video frame that precedes the video frame to be segmented ni can be determined as the target historical video frame.

[0073] Step S1033: Determine the target transparency template corresponding to the target historical video frame as the reference transparency template for the video frame to be segmented.

[0074] Here, the target transparency template corresponding to the target historical video frame can be obtained first, and the target transparency corresponding to the target historical video frame is also determined based on the above steps S101 to S105; then, the target transparency template corresponding to the target historical video frame is determined as the reference transparency template of the video frame to be segmented.

[0075] In this embodiment of the application, through the above steps S1031 to S1033, the number of at least one historical video frame n is first determined; then, the historical video frame that precedes the video frame to be segmented by ni frames is determined as the target historical video frame, where i is an integer greater than or equal to zero and less than n, thereby determining the historical video frame that is associated with the video frame to be segmented; finally, the target transparency template corresponding to the target historical video frame is determined as the reference transparency template of the video frame to be segmented. Since the target transparency template corresponding to the target historical video frame is the spatiotemporal feature of the video data, the spatiotemporal feature of the video data can be obtained simply and efficiently.

[0076] In some embodiments, step S104, "inputting the video frame to be segmented and the reference transparency template into the trained segmentation model to obtain the target transparency template of the video frame to be segmented," is actually implemented using the reference... Figure 3 This can be achieved through the following steps S1041 to S1044:

[0077] Step S1041: Extract semantic information from the video frame to be segmented and the reference transparency template to obtain the target semantic information.

[0078] In actual implementation, refer to Figure 4 Step S1041 can be achieved through the following steps S411 to S413:

[0079] Step S411: Local feature extraction and splicing are performed on the video frame to be segmented and the reference transparency template in sequence to obtain the first local feature.

[0080] Here, local features can be extracted from the video frame to be segmented and the reference transparency template through convolutional layers to obtain the extraction results. Each convolutional layer consists of several convolutional units, and the parameters of each convolutional unit are optimized through the backpropagation algorithm. The purpose of convolution is to extract different features from the input. The first convolutional layer can extract some low-level features such as edges, lines, and corners. The structure of more layers can iteratively extract more complex features from low-level features. Step S411 is to extract more abstract features, such as classification information in the video frame, that is, whether a certain position in the video frame belongs to the background or the foreground. For example, this convolutional layer can be a 3*3 convolutional layer.

[0081] In this embodiment, the extraction results can be concatenated based on the order of Y, U, V and the reference template to obtain the first local feature. Of course, the extraction results can also be concatenated based on the order of the reference template, Y, U and V. This embodiment does not limit the concatenation order.

[0082] Step S412: Downsample and extract features from the first local features to obtain the second local features.

[0083] Here, the first local feature can be downsampled and feature extracted multiple times. In this embodiment, taking two downsampling and feature extraction as an example, the first local feature is downsampled for the first time to obtain the first downsampling result, and then the first downsampling result is used for feature extraction to obtain the first extraction result; then the first extraction result is downsampled for the second time to obtain the second downsampling result, and then the second downsampling result is used for feature extraction to obtain the second local feature.

[0084] The downsampling factor can be 4x, 16x, 64x, etc. Continuing with the example above, the first downsampling factor can be 4x. After the first downsampling, the length and width of the first local feature are reduced to 1 / 2 of the input size, thereby reducing the computational cost of each part. Then, multiple dense convolutional modules (e.g., 2 dense convolutional modules) can be used to continue feature extraction from the first downsampling result, thereby extracting more abstract classification features. Next, the second downsampling factor can be 16x. After the second downsampling, the length and width of the first extracted result are reduced to 1 / 4 of the input size, further reducing the computational cost of each part. Then, two dense convolutional modules are used for secondary feature extraction to obtain the second local feature.

[0085] Step S413: Upsample the second local features to obtain target semantic information.

[0086] Here, the upsampling factor corresponds to the downsampling factor in step S412. For example, if 4x downsampling and 16x downsampling are performed in step S412, the upsampling factor is 64x, thereby restoring the size to the input size. After upsampling restoration, the target semantic information is obtained. This target semantic information includes the relatively abstract classification feature information of the video frame to be segmented, that is, the foreground and background information in the video frame to be segmented.

[0087] Step S1042: Extract edge information from the video frame to be segmented and the reference transparency template to obtain the target edge information.

[0088] In practice, step S1042 can be achieved through the following steps S421 and S422:

[0089] Step S421: Local feature extraction is performed on the video frame to be segmented and the reference transparency template to obtain the third local feature.

[0090] Here, the method for local feature extraction in step S421 is similar to the method for local feature extraction in step S411. Therefore, the method for local feature extraction in step S421 can refer to the method for local feature extraction in step S411.

[0091] Step S422: Extract features from the third local features to obtain target edge information.

[0092] Here, a dense convolutional module can be used to extract features from the third local features to obtain target edge information, which refers to the image information at the boundary between the foreground and background in the video frame to be segmented.

[0093] Step S1043: Fuse target semantic information and target edge information to obtain target fusion information.

[0094] Here, the semantic information and edge information of the target can be fused by adding the feature maps element by element to obtain the target fusion information that includes abstract classification feature information and edge information.

[0095] Step S1044: Extract features from the target fusion information to obtain the target transparency template.

[0096] Here, the feature extraction method in step S1044 is similar to the local feature extraction method in step S411. Therefore, the local feature extraction method in step S1044 can refer to the local feature extraction method in step S411.

[0097] In this embodiment, through steps S1041 to S1044, on the one hand, abstract classification features are extracted from the video frame to be segmented and the reference transparency template through feature extraction, stitching, downsampling, upsampling, etc.; on the other hand, target edge information of the video frame to be segmented and the reference transparency template is obtained through feature extraction; finally, the abstract classification features and target edge information are fused, and the target transparency template of the video frame to be segmented is obtained after feature extraction. Since the spatiotemporal feature of the reference transparency template is taken into account during the processing, the segmentation process can be simplified and the segmentation accuracy can be improved.

[0098] In some embodiments, the target video frame includes a Y component, a U component, and a V component, after step S102, as follows: Figure 5 As shown, the following steps can also be performed:

[0099] Step S103': Determine whether there is at least one historical video frame before the video frame to be segmented.

[0100] Here, it can be determined whether there is at least one instance video frame before the video frame to be segmented by judging whether the video frame to be segmented is the first video frame of the video data. If the video frame to be segmented is the first video frame of the video data, it indicates that there is no historical video frame before the video frame to be segmented, and proceed to step S104'; if the video frame to be segmented is not the first video frame of the video data, it indicates that there is at least one historical video frame before the video frame to be segmented, and proceed to step S106'.

[0101] Step S104': Determine the Y component as the reference transparency template for the video frame to be segmented.

[0102] At this point, there are no historical video frames preceding the video frame to be segmented. To improve the reliability of the segmentation, the Y component of the video frame to be segmented is determined as the reference transparency template for that video frame.

[0103] Step S105': Input the video frame to be segmented and the reference transparency template into the trained segmentation model to obtain the target transparency template of the video frame to be segmented.

[0104] Here, the implementation process of step S105' is similar to that of step S104. Therefore, the implementation process of step S105' can refer to the implementation process of step S104.

[0105] Step S106', proceed to step S103.

[0106] At this point, it indicates that there is at least one historical video frame before the video frame to be segmented. Since step S103 is for the case where there is at least one historical video frame before the video frame to be segmented, step S103 is performed.

[0107] Through steps S103' to S106' above, if there are no historical video frames before the video frame to be segmented, the Y component of the video frame to be segmented is determined as the reference transparency template of the video frame to be segmented; if there are at least one historical video frame before the video frame to be segmented, the reference transparency template of the video frame to be segmented is determined from the target transparency templates corresponding to at least one historical video frame; finally, the video frame to be segmented and the reference transparency template are input into the trained segmentation model to obtain the target transparency template of the segmentation model. That is, regardless of whether there are at least one historical video frame before the video frame to be segmented, a relatively accurate segmentation of the video frame to be segmented can be achieved, thereby improving the completeness of the segmentation scheme.

[0108] In some embodiments, before performing step S101 "obtaining the video data to be segmented and the trained segmentation model", it is also necessary to determine the trained segmentation model, referring to... Figure 6The trained segmentation model can be determined through the following steps S001 to S003:

[0109] Step S001: Obtain video training data, image training data, and a preset segmentation model.

[0110] Here, the image training data includes training images, the first training label corresponding to the training images, and the training image transparency template corresponding to the training images; the video training data includes training video frames, the second training label corresponding to the training video frames, and the training video frame transparency template corresponding to the training video frames.

[0111] In this embodiment, video training data and image training data can be obtained from a general-purpose server or a dedicated server. In practice, image training data can also be obtained from the Portrait Matting_dataset dataset. A preset classification model refers to a model where the weight values ​​are not yet determined; these weight values ​​are currently either default initial values ​​or randomly generated values.

[0112] In some embodiments, the training images and their corresponding first training labels in the image training data can be obtained through step S001, while the training image transparency template corresponding to the training image can be obtained through the following steps S0011 and S0012:

[0113] Step S0011: Annotate the training images to obtain the original transparency template.

[0114] Here, taking the example of obtaining training images from the Portrait Matting_dataset dataset, since this dataset already has a coarse transparency template, image processing software is used to further refine the transparency template, thereby completing the annotation process of the training image and obtaining the original transparency template corresponding to the training image.

[0115] Step S0012: Perform rigid transformation and / or non-rigid transformation on the original transparency template to obtain the training image transparency template.

[0116] Here, since the training image is a single image without any other images that are temporally related, the motion of the training image can be simulated through rigid transformations and / or non-rigid transformations to obtain the transparency template for the training image. A rigid transformation refers to a translation and rotation of the image within the original transparency template without changing the image shape; non-rigid transformations are more complex than rigid transformations, such as scaling, affine transformations, projective transformations, and polynomial transformations.

[0117] In the embodiments of this application, the training image transparency template can be considered as the spatiotemporal features of the training image.

[0118] Step S002: Use the image training data to train the preset segmentation model to obtain a pre-trained segmentation model.

[0119] In actual implementation, refer to Figure 7 Step S002 can be achieved through the following steps S021 to S023:

[0120] Step S021: Input the training image and the training image transparency template into the preset segmentation model to obtain the first predicted label corresponding to the training image.

[0121] Here, the preset segmentation model is trained for the first time, and the result is the first predicted label corresponding to the training image. In practice, the training process is similar to the inference process. Therefore, the implementation process of step S021 is similar to the implementation process of step S104. Therefore, the implementation process of step S021 can refer to the implementation process of step S104.

[0122] Step S022: Obtain the first error information between the first training label and the first predicted label.

[0123] Here, both the first training label and the first predicted label are matrices. The distance between the first training label and the first predicted label can represent the difference between them. Therefore, the distance between the first training label and the first predicted label can be determined as the first error information.

[0124] Step S023: Based on the first error information and the first error threshold, backpropagation training is performed on the preset segmentation model to obtain a pre-trained segmentation model.

[0125] Here, the first error threshold can be a default value or a custom setting. If the first error information is less than the first error threshold, it means that the first error information is within the allowable range, the training stopping condition is met, and no further training is needed. This indicates that a preliminary training segmentation model has been obtained. However, if the first error information is greater than or equal to the first error threshold, it means that the first error information is not within the allowable range, the training stopping condition is not met, and a preliminary training segmentation model has not yet been obtained. Further training is still needed.

[0126] Step S003: Use the video training data to further train the pre-trained segmentation model to obtain a trained segmentation model.

[0127] In actual implementation, refer to Figure 8 Step S003 can be achieved through the following steps S031 to S033:

[0128] Step S031: Input the training video frame and the training video frame transparency template into the pre-trained segmentation model to obtain the second predicted label corresponding to the training video frame.

[0129] Here, the implementation process of step S031 is similar to that of step S021. Therefore, the implementation process of step S031 can refer to the implementation process of step S021.

[0130] Step S032: Obtain the second error information between the second training label and the second predicted label.

[0131] Here, the implementation process of step S032 is similar to that of step S022. Therefore, the implementation process of step S032 can refer to the implementation process of step S022.

[0132] Step S033: Based on the second error information and the second error threshold, backpropagation training is performed on the initially trained segmentation model to obtain the trained segmentation model.

[0133] Here, the second error threshold can be a default value or a custom setting value. In addition, the second error threshold can be the same as the first error threshold or different from the first error threshold. This application embodiment does not limit this.

[0134] If the second error information is less than the second error threshold, it indicates that the second error information is within the allowable range, the training stopping condition is met, and no further training is required. This indicates that a well-trained segmentation model has been obtained. However, if the second error information is greater than or equal to the second error threshold, it indicates that the second error information is not within the allowable range, the training stopping condition is not met, and a well-trained segmentation model has not yet been obtained. Further training is still required.

[0135] Through steps S001 to S003, video training data, image training data, and a preset segmentation model are first acquired. Since image training data can be used to train the preset segmentation model simply and efficiently, the preset segmentation model is first trained using the image training data to obtain a pre-trained segmentation model. Then, the pre-trained segmentation model is trained again using video training data to finally obtain a fully trained segmentation model. Through this phased model training, a more accurate trained segmentation model is obtained.

[0136] Based on the above embodiments, this application further provides a segmentation method. This segmentation method is an efficient video portrait segmentation method based on the YUV color space combined with spatiotemporal features, which can be used in video communication, live streaming, and other scenarios. The segmentation method includes:

[0137] First, data collection and annotation. The data is mainly divided into image data and video data. Image data corresponds to the image training data in the above embodiments, and video data corresponds to the video training data in the above embodiments. The image data uses open-source datasets, and the video data mainly consists of video data collected by acquisition devices in telecommunications scenarios. Here, the acquisition devices can be Universal Serial Bus (USB) cameras, smartphones, or set-top boxes with cameras. For data annotation, the annotation information provided by the open-source datasets often contains noise and edge defects, requiring further refinement of the transparency template using image processing software (e.g., Adobe Photoshop). For video data, a frame extraction tool (e.g., ffmpeg) is first used to obtain all video frames, and then image processing software (e.g., Adobe Photoshop) is used to extract the portrait transparency template.

[0138] Second, video portrait segmentation model training; here, the video portrait segmentation model corresponds to the preset segmentation model in the above embodiments. Since the time required for video frame image segmentation and annotation is high, this application proposes training the video portrait segmentation method using single images. Rigid and non-rigid transformations are added to the transparency template of a single image to simulate the difference in transparency template caused by the movement of consecutive frames in the video. Rigid transformations include translation, rotation, and combinations thereof. Translation simulates the horizontal movement of the portrait relative to the lens plane. Non-rigid transformations include scaling, radial transformation, projection, polynomial transformation, and local motion transformation. Scaling simulates the vertical movement of the portrait relative to the lens plane. Local motion transformations of the portrait can be simulated by using thin plate spline (TPS) deformation with K control points. Thus, training is first performed on image data to obtain the first model weights; based on these first model weights, transfer learning is then performed using real video portrait segmentation data to obtain the second model weights.

[0139] Third, the trained model is deployed to smart devices for video portrait segmentation. Here, the trained model corresponds to the trained segmentation model in the above embodiments.

[0140] In the embodiments of this application, such as Figure 9 As shown, the segmentation method includes the following steps S901 to S905 during the model training phase:

[0141] Step S901: Acquire image and video data.

[0142] Here, image and video data includes image data and video data. Image data can be directly obtained from the open-source Portrait Matting_dataset dataset, while video data can be obtained from video data collected by an acquisition device, such as a USB camera, smartphone, or set-top box with a camera.

[0143] Step S902, data annotation.

[0144] Here, the current frame transparency template annotation includes: after obtaining the raw data, data annotation and cleaning are performed on the raw data. For image data, the Portrait Matting_dataset dataset is used, which already has a coarse transparency template. Adobe Photoshop can then be used to further refine the transparency template. For video data, tools such as ffmpeg are first used to split the video into video frames, and then Adobe Photoshop can be used to segment and obtain the transparency template. To utilize the spatiotemporal features in the video to supervise the network training, this embodiment proposes using the current video frame combined with the transparency template (mask) of the previous frame as input to the video data portrait segmentation model algorithm. Furthermore, the video frames stored in the video are often in the YUV color space rather than the RGB color space commonly found in images. To simplify the calculation process, this embodiment uses a four-channel data input of YUV + Mask for network training and inference.

[0145] Step S903: Image data transparency template transformation.

[0146] Here, to fully utilize image data, this application proposes training a video portrait segmentation method using a single image. Since image data does not contain previous frame data, the determination of the input Mask is as follows: Figure 10 As shown, for reference Figure 10 a) is the original image, and b) is the annotated transparency template obtained after labeling a) in step S902. Based on this, a non-rigid transformation of scaling is performed on b) to obtain the scaled transparency template, i.e., c). In practice, the scaling scale is uniformly distributed between [0.9, 1.1]. Next, a rigid transformation of translation is performed on c) to obtain the translated transparency template, i.e., d). In practice, the translation coordinates in the four directions of up, down, left, and right are uniformly distributed between [0, 0.05] times the size of the original image. Then, a non-rigid transformation of TPS deformation is performed on d) to obtain the TPS deformed transparency template, i.e., e). When performing TPS deformation, the value of K can be 5, which is equivalent to simulating the local motion transformation of the human face through TPS of 5 control points.

[0147] In addition, to address the issue that the first frame of a simulated video does not contain the transparency template of the previous frame, when the current frame is the first frame of the video, the Y channel in the YUV color space of the current frame can be used as the mask, and then training can be performed on the current frame based on YUV and the mask.

[0148] Step S904: Training the single-image portrait segmentation network.

[0149] Here, after obtaining the labeled training data and performing transparency template transformation, the first step is to train a portrait segmentation network model on the image data. This portrait segmentation network model corresponds to the preset segmentation model in the above embodiments, and its structure is as follows: Figure 11 As shown, the input of the portrait segmentation network differs from the RGB color space data used in traditional portrait segmentation. Since video frame data is often stored in YUV420 or YUV420P format in video encoding, traditional image segmentation algorithms require converting YUV format video frames to RGB format for input into the model algorithm, introducing additional computational overhead. To simplify computation, YUV color space data is used here, meaning the network is trained directly using the YUV format, eliminating the computational overhead of the conversion during the inference phase. Furthermore, to enhance the robustness of the segmentation results and reduce jitter between frames during video segmentation, this application proposes a segmentation method incorporating video spatiotemporal features. This is reflected in the fact that the input of the segmentation method in this application considers the segmentation result of the previous frame, i.e., the transparency template of the previous frame. Therefore, the input of the segmentation method in this application mainly consists of two parts: the current frame encoded using YUV and the transparency template (Mask) of the previous frame.

[0150] The backbone of the human image segmentation network model has two bi-branch structures, such as... Figure 11 As shown, Figure 11The upper branch is the coarse branch 1101, which has a large number of network layers and is used to extract the generated high-level semantic information. The lower branch is the refined branch 1102, which uses only a small number of convolutional layers to extract shallow edge details. The coarse branch 1101 first passes through a 3*3 convolutional layer 11011, then concatenates the input data and downsamples it by 4 times 11012, so that the length and width of Y, U, V, and Mask are each reduced to 1 / 2 of the input size. Then, it passes through two dense convolutional modules 11013 and concatenates the input data and downsamples it by 16 times 11014, so that the length and width of Y, U, V, and Mask are each reduced to 1 / 4 of the input size. Next, it passes through two dense convolutional modules 11015, and then uses an upsampling layer 11016 to upsample the feature map to 1 / 4 of the input area, resulting in the coarse branch result. Here, the upsampling layer 11016 can use bilinear interpolation. The refined branch 1102 first passes through a 3*3 convolutional layer 11021, then through a dense comprehensive convolutional module 11022, to obtain the refined branch result. Finally, the feature maps of the coarse branch result and the refined branch result are added element-wise 1103 and passed through a 3*3 convolutional layer 1104 to obtain the final transparency template 110.

[0151] In the training process described above, the stochastic gradient descent algorithm with momentum can be used as the optimization function, and L2 regularization can be used to constrain the model weights. For the loss function, considering both the edge loss from portrait segmentation and the cross-entropy loss to supervise the network's learning, the total loss function is shown in Equation 1 below:

[0152] L total =λ*L ce +(1-λ)*L b (1);

[0153] Among them, L total Let L represent the total loss function, λ represent the weighting coefficients, and L represent the total loss function. ce The cross-entropy loss is represented by the cross-entropy function with focal loss coefficients. This function can alleviate the imbalance between the foreground and background of the portrait in the dataset. The calculation method is shown in Equation 2, where L... b To represent the boundary loss, the image morphology operation can be performed using the transparency template and the labeled template output by the human image segmentation network model. This includes: first, dilating the transparency to obtain the dilated template, then performing erosion to obtain the eroded template, and finally subtracting the two to obtain the edge information of the image. The boundary loss is then calculated using Formula 3.

[0154]

[0155]

[0156] Where y represents the label information of a specific location in the image. This represents the prediction result of the human face segmentation network model. γ is greater than 0, and adjusting the value of γ affects the loss weights of the human face segmentation network model for samples that are difficult to segment. The gradient of the existing neural network parameters is calculated based on the above loss function, and the parameters are updated using stochastic gradient descent with momentum until the human face segmentation network converges, thus obtaining the first model weights.

[0157] Step S905: Video portrait segmentation transfer training.

[0158] Here, after obtaining the first model weights, the video data portrait segmentation model is trained. First, considering the selection of the transparency template of the previous frame of the video, the current frame is recorded as frame t. The transparency templates of frames t-1, t-3, and t-5 can be randomly used as masks to simulate the differences caused by the movement of the person under different frame rates, different people, and different scenes. When t-1, t-3, and t-5 are less than 0, the Y channel in the YUV color space of the current frame can be used as a mask to combine the current frame for training. The structure of the video data portrait segmentation model is consistent with the model result in step S604. Since the input mask of the video data portrait segmentation model is relatively accurate at this time, and in order to achieve better segmentation results, the cross-entropy loss function with focal loss coefficient in the loss function of the video data portrait segmentation model can be replaced with the Lovasz loss function, as shown in Equations 4, 5, and 6.

[0159]

[0160]

[0161]

[0162] Where y represents the label information of a specific location in the image. This represents the prediction result of the video data portrait segmentation model. The hinge loss predicted for the current position. for The Lovasz extension. Compared to the Jaccard loss shown in Equation 6, the Lovasz loss can more effectively reduce the loss. Load the first network weight file, and calculate the gradient with respect to the existing neural network parameters according to the above loss function. Use stochastic gradient descent with momentum to update the parameters until the network converges, and the second model weights, i.e., the final training result, can be obtained.

[0163] In this embodiment, firstly, the YUV color space is used for model inference. Compared to the traditional RGB color space, this avoids the computational overhead of converting video frames from YUV to RGB format, saving computational resources and time, and improving the efficiency of video portrait segmentation. Secondly, a YUV+Mask input format is used for video portrait segmentation. Compared to traditional single RGB data, this adds spatiotemporal features from the video to the model's prediction, while simplifying the difficulty of predicting the transparency template, further reducing the model's complexity. This improves both algorithm performance and robustness while also enhancing algorithm efficiency. Thirdly, a multi-stage training approach is used, sequentially training the video portrait segmentation model with image data and then video data. During the training phase, image data is applied to the video portrait segmentation task through image transparency template transformation, alleviating to some extent the problems of insufficient video portrait segmentation data and high annotation costs. Simultaneously, different loss functions are used based on the characteristics of the data for more effective supervised learning.

[0164] Based on this, the embodiments of this application do not require color space conversion during video portrait segmentation inference, achieving efficient video non-compliance detection. Furthermore, compared to traditional single-frame portrait segmentation algorithms, by adding the transparency template of the previous frame to the input, the spatiotemporal features in the video are increased. This not only solves the robustness issues such as transparency template jitter and insufficient segmentation quality during video segmentation, but also simplifies the difficulty of predicting the current frame. A better video portrait segmentation result can be obtained with a smaller network, improving the model's robustness while achieving efficient prediction. Finally, during the training phase, image data is applied to the video portrait segmentation task through image transparency template transformation, alleviating to some extent the problems of insufficient video portrait segmentation data and high annotation costs.

[0165] Based on the foregoing embodiments, this application provides a segmentation device. The modules and units included in the device can be implemented by a processor in a computer device; of course, they can also be implemented by corresponding logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0166] This application embodiment further provides a segmentation device, Figure 12 This is a schematic diagram of the composition of the segmentation device provided in the embodiments of this application, such as... Figure 12As shown, the dividing device 1200 includes:

[0167] The acquisition module 1201 is used to acquire the video data to be segmented and the trained segmentation model;

[0168] The first determining module 1202 is used to determine multiple video frames included in the video data, and to sequentially determine the multiple video frames as video frames to be segmented;

[0169] The second determining module 1203 is used to determine a reference transparency template of the video frame to be segmented from the target transparency template of the at least one historical video frame when there is at least one historical video frame before the video frame to be segmented.

[0170] The segmentation module 1204 is used to input the video frame to be segmented and the reference transparency template into the trained segmentation model to obtain the target transparency template of the video frame to be segmented.

[0171] The extraction module 1205 is used to extract the video frame to be segmented based on the target transparency template to obtain the target foreground image of the video frame to be segmented.

[0172] In some embodiments, the second determining module 1203 includes:

[0173] The first determining submodule is used to determine the number of frames of the at least one historical video frame, and to record the number of frames as n, where n is an integer greater than or equal to 1;

[0174] The second determining submodule is used to determine the historical video frame that precedes the video frame to be segmented ni as the target historical video frame, wherein i is greater than or equal to zero and less than n, and i is an integer.

[0175] The third determining submodule is used to determine the target transparency template corresponding to the target historical video frame as the reference transparency template of the video frame to be segmented.

[0176] In some embodiments, the segmentation module 1204 includes:

[0177] The first extraction submodule is used to extract semantic information from the video frame to be segmented and the reference transparency template to obtain target semantic information.

[0178] The second extraction submodule is used to extract edge information from the video frame to be segmented and the reference transparency template to obtain target edge information;

[0179] The fusion submodule is used to fuse the target semantic information and the target edge information to obtain target fusion information;

[0180] The feature extraction submodule is used to extract features from the target fusion information to obtain the target transparency template.

[0181] In some embodiments, the first extraction submodule includes:

[0182] The first extraction unit is used to sequentially extract and stitch local features from the video frame to be segmented and the reference transparency template to obtain the first local features.

[0183] The second extraction unit is used to downsample and extract features from the first local features to obtain the second local features;

[0184] An upsampling unit is used to upsample the second local feature to obtain the target semantic information.

[0185] In some embodiments, the second extraction submodule includes:

[0186] The third extraction unit is used to extract local features from the video frame to be segmented and the reference transparency template to obtain the third local features;

[0187] The fourth extraction unit is used to extract features from the third local features to obtain the target edge information.

[0188] In some embodiments, the target video frame includes Y, U, and V components. The segmentation module 1204 is further configured to input the video frame to be segmented and the reference transparency template into the trained segmentation model to obtain the target transparency template of the video frame to be segmented. The segmentation device 1200 includes:

[0189] The third determining module is used to determine the Y component as the reference transparency template of the video frame to be segmented when there is no previous historical video frame before the video frame to be segmented.

[0190] In some embodiments, the acquisition module 1201 is further configured to acquire video training data, image training data, and a preset segmentation model;

[0191] The segmentation module 1204 is further configured to train the preset segmentation model using the image training data to obtain a pre-trained segmentation model; and to continue training the pre-trained segmentation model using the video training data to obtain the trained segmentation model.

[0192] In some embodiments, the image training data includes training images, a first training label corresponding to the training images, and a training image transparency template corresponding to the training images; the segmentation module 1204 includes:

[0193] The first input submodule is used to input the training image and the training image transparency template into the preset segmentation model to obtain the first predicted label corresponding to the training image;

[0194] The first acquisition submodule is used to acquire first error information between the first training label and the first predicted label;

[0195] The first training submodule is used to perform backpropagation training on the preset segmentation model based on the first error information and the first error threshold to obtain the pre-trained segmentation model.

[0196] In some embodiments, the segmentation device 1200 further includes:

[0197] The annotation submodule is used to annotate the training images to obtain the original transparency template;

[0198] The transformation submodule is used to perform rigid transformation and / or non-rigid transformation processing on the original transparency template to obtain the training image transparency template.

[0199] In some embodiments, the video training data includes training video frames, second training labels corresponding to the training video frames, and training video frame transparency templates corresponding to the training video frames. The segmentation module 1204 further includes:

[0200] The second input submodule is used to input the training video frame and the transparency template of the training video frame into the pre-trained segmentation model to obtain the second predicted label corresponding to the training video frame.

[0201] The second acquisition submodule is used to acquire the second error information between the second training label and the second predicted label;

[0202] The second training submodule is used to perform backpropagation training on the pre-trained segmentation model based on the second error information and the second error threshold to obtain the trained segmentation model.

[0203] It should be noted that the description of the segmentation device in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects. For technical details not disclosed in this device embodiment, please refer to the description of the method embodiment of this application for understanding.

[0204] It should be noted that, in the embodiments of this application, if the above-described segmentation method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0205] Accordingly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the segmentation method provided in the above embodiments.

[0206] This application provides a segmentation device. Figure 13 This is a schematic diagram of the composition structure of the segmentation device provided in the embodiments of this application, such as... Figure 13 As shown, the segmentation device 1300 includes: a processor 1301, at least one communication bus 1302, a user interface 1303, at least one external communication interface 1304, and a memory 1305. The communication bus 1302 is configured to enable communication between these components. The user interface 1303 may include a display screen, and the external communication interface 1304 may include standard wired and wireless interfaces. The processor 1301 is configured to execute a program of a segmentation method stored in the memory to implement the segmentation method provided in the above embodiment.

[0207] The descriptions of the above-described partitioning device and storage medium embodiments are similar to those of the above-described method embodiments, and have similar beneficial effects. For technical details not disclosed in the embodiments of the partitioning device and storage medium of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0208] It should be noted that the descriptions of the storage medium and partitioning device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium and partitioning device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0209] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0210] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0211] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0212] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0213] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0214] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0215] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an AC to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0216] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A segmentation method, characterized in that, The method includes: Acquire the video data to be segmented and the trained segmentation model; The video data includes multiple video frames, which are then sequentially identified as video frames to be segmented. Each video frame to be segmented includes a Y component, a U component, and a V component. If there is at least one historical video frame before the video frame to be segmented, determine the reference transparency template of the video frame to be segmented from the target transparency template of the at least one historical video frame. When there are no previous historical video frames before the video frame to be segmented, the Y component is determined as the reference transparency template of the video frame to be segmented. The video frame to be segmented and the reference transparency template are input into the trained segmentation model, and the segmentation is processed through the coarse branches of the trained segmentation model. Three convolutional layers, 4x downsampling of concatenated input data, dense comprehensive convolutional module, 16x downsampling of concatenated input data, dense comprehensive convolutional module, and upsampling are used to obtain coarse branch results; The refined branch of the trained segmentation model is processed through 3... Three convolutional layers are used, followed by a densely synthesized convolutional module, to obtain a refined branching result; The feature maps of the coarse branch result and the refined branch result are summed element-wise, and then processed using 3... 3 convolutional layers are used to obtain the target transparency template; The target foreground image of the video frame to be segmented is obtained by extracting the target transparency template.

2. The method according to claim 1, characterized in that, When at least one historical video frame exists before the video frame to be segmented, determining a reference transparency template for the video frame to be segmented from the target transparency templates of the at least one historical video frame includes: Determine the number of frames in the at least one historical video frame, and denot the number of frames as n, where n is an integer greater than or equal to 1; The historical video frame that precedes the video frame to be segmented ni is determined as the target historical video frame, where i is greater than or equal to zero and less than n, and i is an integer. The target transparency template corresponding to the target historical video frame is determined as the reference transparency template for the video frame to be segmented.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Acquire video training data, image training data, and preset segmentation models; The preset segmentation model is trained using the image training data to obtain a pre-trained segmentation model; The pre-trained segmentation model is further trained using the video training data to obtain the trained segmentation model.

4. The method according to claim 3, characterized in that, The image training data includes training images, a first training label corresponding to the training images, and a training image transparency template corresponding to the training images. The step of training the preset segmentation model using the image training data to obtain a pre-trained segmentation model includes: The training image and the training image transparency template are input into the preset segmentation model to obtain the first predicted label corresponding to the training image; Obtain the first error information between the first training label and the first predicted label; Based on the first error information and the first error threshold, the preset segmentation model is backpropagated and trained to obtain the pre-trained segmentation model.

5. The method according to claim 4, characterized in that, The method further includes: The training images are annotated to obtain the original transparency template; The original transparency template is subjected to rigid transformation and / or non-rigid transformation to obtain the training image transparency template.

6. The method according to claim 3, characterized in that, The video training data includes training video frames, second training labels corresponding to the training video frames, and training video frame transparency templates corresponding to the training video frames. The step of further training the pre-trained segmentation model using the video training data to obtain a trained segmentation model includes: The training video frame and the transparency template of the training video frame are input into the pre-trained segmentation model to obtain the second predicted label corresponding to the training video frame; Obtain the second error information between the second training label and the second predicted label; The pre-trained segmentation model is backpropagated and trained based on the second error information and the second error threshold to obtain the trained segmentation model.

7. A dividing device, characterized in that, The segmentation device includes: The acquisition module is used to acquire the video data to be segmented and the trained segmentation model; The first determining module is used to determine multiple video frames included in the video data, and to sequentially determine the multiple video frames as video frames to be segmented; the video frames to be segmented include Y components, U components and V components; The second determining module is configured to determine a reference transparency template for the video frame to be segmented from the target transparency templates of the at least one historical video frame when there is at least one historical video frame before the video frame to be segmented; and to determine the Y component as the reference transparency template for the video frame to be segmented when there is no historical video frame before the video frame to be segmented. The segmentation module is used to input the video frame to be segmented and the reference transparency template into the trained segmentation model, and process them through the coarse branches of the trained segmentation model. A coarse branch result is obtained by performing a 3-convolutional layer, concatenating input data downsampled by 4 times, a dense comprehensive convolutional module, concatenating input data downsampled by 16 times, a dense comprehensive convolutional module, and upsampling. The refined branch result is then processed through the trained segmentation model. Three convolutional layers are applied, followed by a dense comprehensive convolutional module to obtain a refined branch result. The feature maps of the coarse branch result and the refined branch result are then element-wise summed, and the result is further processed using a 3-layer convolutional layer. 3 convolutional layers are used to obtain the target transparency template; The extraction module is used to extract the video frame to be segmented based on the target transparency template to obtain the target foreground image of the video frame to be segmented.

8. A dividing device, characterized in that, The segmentation device includes: Processor; and Memory for storing computer programs that can run on the processor; When the computer program is executed by a processor, it implements the segmentation method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions configured to perform the segmentation method according to any one of claims 1 to 6.

Images