Human body image segmentation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a human body image segmentation method and device, electronic equipment and a storage medium, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a target video frame sequence containing a target object; wherein the target video frame sequence has a timestamp; classifying the target video frame sequence based on the timestamp to obtain a reference video frame and a target video frame; wherein the timestamp of the reference video frame is before the timestamp of the target video frame; generating a segmentation mask according to the reference video frame to obtain reference mask data; performing image segmentation mask prediction on the target video frame based on a preset human body segmentation mask prediction model, the reference video frame and the reference mask data to obtain target mask data; and performing human body image segmentation on the target video frame according to the target mask data. According to the embodiment of the invention, the human body image segmentation precision of the video frame sequence can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for segmenting human images, an electronic device, and a storage medium. Background Art

[0002] Human image segmentation is to extract the human body area from a static image, and to achieve accurate segmentation of the human body by classifying each pixel in the image as "human body" or "background". In the existing human image segmentation technology, the static image segmentation method based on deep learning is mainly used. Since the static image segmentation method only relies on the information of a single frame image, the segmentation accuracy is low under fast motion or complex movements, and it is difficult to ensure the human body segmentation effect in a series of video frame sequences. Therefore, how to improve the accuracy of human image segmentation in video frame sequences has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a human image segmentation method and device, an electronic device and a storage medium, aiming to improve the accuracy of human image segmentation in a video frame sequence.

[0004] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a human body image segmentation method, the method comprising:

[0005] Acquire a target video frame sequence containing a target object; wherein the target video frame sequence has a timestamp;

[0006] Classifying the target video frame sequence based on the timestamp to obtain a reference video frame and a target video frame; wherein the timestamp of the reference video frame is before the timestamp of the target video frame;

[0007] Generating a segmentation mask according to the reference video frame to obtain reference mask data;

[0008] Performing image segmentation mask prediction on the target video frame based on a preset human body segmentation mask prediction model, the reference video frame and the reference mask data to obtain target mask data;

[0009] Perform human image segmentation on the target video frame according to the target mask data.

[0010] In some embodiments, the preset human body segmentation mask prediction model includes an encoder and a decoder; the performing image segmentation mask prediction on the target video frame based on the preset human body segmentation mask prediction model, the reference video frame and the reference mask data to obtain the target mask data includes:

[0011] Based on the encoder, segment the reference video frame and the reference mask data to obtain reference key encoding features and reference value encoding features;

[0012] Based on the encoder, segment the target video frame to encode the image, and obtain a target key encoding feature and a target value encoding feature;

[0013] Based on the decoder, segmentation mask prediction is performed on the reference key encoding feature, the reference value encoding feature, the target key encoding feature and the target value encoding feature to obtain the target mask data.

[0014] In some embodiments, performing segmentation mask prediction on the reference key encoding feature, the reference value encoding feature, the target key encoding feature, and the target value encoding feature based on the decoder to obtain the target mask data includes:

[0015] Generate a similarity graph based on the reference key encoding feature, the target key encoding feature and the reference value encoding feature to obtain a spatiotemporal similarity graph feature;

[0016] Perform data splicing according to the spatiotemporal similarity graph feature and the target value coding feature to obtain a target coding feature;

[0017] Based on the decoder, segmentation mask prediction is performed on the target coding feature to obtain the target mask data.

[0018] In some embodiments, generating a similarity graph according to the reference key encoding feature, the target key encoding feature, and the reference value encoding feature to obtain a spatiotemporal similarity graph feature includes:

[0019] Calculating the product between the reference key encoding feature and the target key encoding feature to obtain a similar graph feature;

[0020] Performing weight normalization processing on the similar graph features to obtain normalized features;

[0021] The spatiotemporal similarity graph feature is obtained by performing weighted summation processing on the normalized feature and the reference value coding feature.

[0022] In some embodiments, the encoding of the reference video frame and the reference mask data based on the encoder to obtain reference key encoding features and reference value encoding features includes:

[0023] Calculating the sum of the reference video frame and the reference mask data to obtain reference mask frame data;

[0024] The reference mask frame data is segmented and encoded based on the encoder to obtain the reference key encoding feature and the reference value encoding feature.

[0025] In some embodiments, the encoder includes a convolutional layer, a residual layer, a pooling layer, and a fully connected layer; the segmentation image encoding of the reference mask frame data based on the encoder to obtain the reference key encoding feature and the reference value encoding feature includes:

[0026] Performing feature convolution on the reference mask frame data based on the convolution layer to obtain reference mask frame convolution features;

[0027] Performing residual connection on the convolutional features of the reference mask frame based on the residual layer to obtain residual features of the reference mask frame;

[0028] Based on the pooling layer, the reference mask frame residual features are pooled and connected to obtain reference mask frame pooling features;

[0029] Performing image encoding on the reference mask frame pooling features based on the fully connected layer to obtain reference mask frame key-value coding features;

[0030] Convolutional dimension reduction is performed based on the reference mask frame key-value coding features to obtain the reference key coding features and the reference value coding features.

[0031] In some embodiments, acquiring a target video frame sequence containing a target object includes:

[0032] Acquire original video data containing the target object;

[0033] Extracting video frames according to the original video data to obtain an original video frame sequence;

[0034] Performing stable picture screening according to the original video frame sequence to obtain an initial video frame sequence;

[0035] Background elimination is performed according to the initial video frame sequence to obtain the target video frame sequence.

[0036] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a human body image segmentation device, the device comprising:

[0037] An acquisition sequence module is used to acquire a target video frame sequence containing a target object; wherein the target video frame sequence has a timestamp;

[0038] A video classification module, used for classifying the target video frame sequence based on the timestamp to obtain a reference video frame and a target video frame; wherein the timestamp of the reference video frame is before the timestamp of the target video frame;

[0039] A mask generation module, used to generate a segmentation mask according to the reference video frame to obtain reference mask data;

[0040] A mask prediction module, configured to perform image segmentation mask prediction on the target video frame based on a preset human body segmentation mask prediction model, the reference video frame and the reference mask data to obtain target mask data;

[0041] An image segmentation module is used to perform human image segmentation on the target video frame according to the target mask data.

[0042] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0044] The present application proposes a method and device for human image segmentation, an electronic device and a storage medium, which obtain a target video frame sequence containing a target object, and then classify the target video frame sequence based on the timestamp of the target video frame sequence to obtain a reference video frame and a target video frame, and then generate a segmentation mask based on the reference video frame to obtain reference mask data, and then perform image segmentation mask prediction on the target video frame based on a preset human segmentation mask prediction model, the reference video frame and the reference mask data to obtain target mask data, thereby realizing mask prediction of the target video frame based on combining spatiotemporal information, thereby improving the human body segmentation effect in a series of video frame sequences, and having accurate segmentation accuracy even under fast motion or complex actions, and finally performing human body image segmentation on the target video frame according to the target mask data to obtain a target human body image, thereby realizing accurate human body image segmentation of the video frame sequence. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a human body image segmentation method provided in an embodiment of the present application;

[0046] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0047] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.

[0048] Figure 4 yes Figure 3 Flow chart of step S301 in FIG.

[0049] Figure 5 yes Figure 4 Flow chart of step S402 in FIG.

[0050] Figure 6 yes Figure 3 Flow chart of step S303 in FIG.

[0051] Figure 7 yes Figure 6 Flow chart of step S601 in FIG.

[0052] Figure 8 is a structural schematic diagram of a human body image segmentation device provided in an embodiment of the present application;

[0053] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] First, some nouns involved in this application are analyzed:

[0058] Image segmentation: It is used to subdivide a digital image into multiple image regions. Image segmentation is a branch of artificial intelligence and belongs to the research field of pattern recognition and image analysis. This technology is often used in applications such as medical image processing, autonomous vehicles, video surveillance, and robot perception, involving multiple processes such as image preprocessing, feature extraction, and target recognition. Through image segmentation, computers can identify independent objects in an image and perform further analysis on each object to support more complex image understanding tasks such as object detection and scene understanding.

[0059] Image masking: is used to selectively manipulate specific areas in an image. Image masks are often used to hide parts of an image or highlight certain parts, which allows for detailed editing and processing of images. Image masking is an application branch of artificial intelligence and belongs to the technical category of image editing and graphic design. Image masking is widely used in image synthesis, visual effects production, medical imaging analysis, and any scenario that requires precise image editing. By using image masks, users can easily isolate objects in an image and enhance, modify, or delete them without affecting other parts of the image. The key steps involved in this technology include the creation, application, and modification of masks. Image masking supports not only static image processing, but also the editing of dynamic video content.

[0060] Encoder: It is a kind of machine that converts input data into a compact vector representation. It belongs to the technical field of artificial intelligence and deep learning, and is usually used as part of a more complex system, such as an autoencoder or a component in a sequence-to-sequence model. The main function of the encoder is data dimensionality reduction and feature extraction. It generates an efficient representation by learning the intrinsic structure of the input data, which captures the key information of the data while removing redundancy. In image processing, the encoder can convert an image into a vector, which can then be used for various tasks such as image classification, image reconstruction, and image retrieval. The application of encoders is not limited to image processing, but is also widely used in natural language processing, speech recognition, and other fields that require feature conversion. This technology involves algorithm development and optimization of deep learning and machine learning.

[0061] Decoder: A decoder is a machine that converts the vector representation generated by an encoder or other network structure back to the format of the original data or into a new output format. Decoders are core components in artificial intelligence and deep learning technologies, and are commonly found in autoencoders, sequence-to-sequence models, and generative adversarial networks. The main function of a decoder is data reconstruction and generation. For example, in natural language processing, it can convert the vector representation of the input text into text in the target language. In image processing, a decoder can reconstruct a high-resolution image from a low-dimensional representation of the image. In addition, decoders are also widely used in speech synthesis, video generation, and other fields that require recovering detailed information or creating new information from compact data representations.

[0062] Human image segmentation is to extract the human body area from a static image, and to achieve accurate segmentation of the human body by classifying each pixel in the image as "human body" or "background". In the existing human image segmentation technology, the static image segmentation method based on deep learning is mainly used. Since the static image segmentation method only relies on the information of a single frame image, the segmentation accuracy is low under fast motion or complex movements, and it is difficult to ensure the human body segmentation effect in a series of video frame sequences. Therefore, how to improve the accuracy of human image segmentation in video frame sequences has become an urgent problem to be solved.

[0063] Based on this, the embodiments of the present application provide a human image segmentation method and device, an electronic device and a storage medium, aiming to improve the accuracy of human image segmentation in a video frame sequence.

[0064] A human image segmentation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the human image segmentation method in the embodiments of the present application is described.

[0065] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0066] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0067] The human image segmentation method provided in the embodiment of the present application relates to the field of image processing technology. The human image segmentation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the human image segmentation method, etc., but is not limited to the above forms.

[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0069] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0070] Figure 1 is an optional flow chart of the human body image segmentation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0071] Step S101, obtaining a target video frame sequence containing a target object; wherein the target video frame sequence has a timestamp;

[0072] Step S102, classifying the target video frame sequence based on the timestamp to obtain a reference video frame and a target video frame; wherein the timestamp of the reference video frame is before the timestamp of the target video frame;

[0073] Step S103, generating a segmentation mask according to the reference video frame to obtain reference mask data;

[0074] Step S104, performing image segmentation mask prediction on the target video frame based on the preset human body segmentation mask prediction model, the reference video frame and the reference mask data to obtain target mask data;

[0075] Step S105 , performing human body image segmentation on the target video frame according to the target mask data.

[0076] Steps S101 to S105 shown in the embodiment of the present application are performed by acquiring a target video frame sequence containing a target object, and then classifying the target video frame sequence based on the timestamp of the target video frame sequence to obtain a reference video frame and a target video frame, and then performing segmentation mask generation based on the reference video frame to obtain reference mask data, and then performing image segmentation mask prediction on the target video frame based on a preset human body segmentation mask prediction model, the reference video frame and the reference mask data to obtain target mask data, thereby achieving mask prediction of the target video frame based on combining spatiotemporal information, thereby improving the human body segmentation effect in a series of video frame sequences, and having accurate segmentation accuracy even under fast motion or complex movements, and finally performing human body image segmentation on the target video frame according to the target mask data to obtain a target human body image, thereby achieving accurate human body image segmentation of the video frame sequence.

[0077] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S204:

[0078] Step S201, obtaining original video data containing a target object;

[0079] Step S202, extracting video frames according to the original video data to obtain an original video frame sequence;

[0080] Step S203, performing stable picture screening according to the original video frame sequence to obtain an initial video frame sequence;

[0081] Step S204: performing background removal according to the initial video frame sequence to obtain a target video frame sequence.

[0082] Steps S201 to S204 shown in the embodiment of the present application are obtained by acquiring original video data containing the target object, then extracting video frames based on the original video data to obtain an original video frame sequence, and then screening stable images based on the original video frame sequence to obtain an initial video frame sequence, thereby screening stable images, reducing inaccurate segmentation caused by motion blur or jitter, and improving the overall quality of subsequent segmentation results; further, background elimination is performed based on the initial video frame sequence to obtain a target video frame sequence, thereby achieving a preliminary reduction in background images and reducing interference caused by background images, thereby improving the overall quality of subsequent segmentation results.

[0083] In step S201 of some embodiments, the original video data is a video containing a human body of the target object, and the way of obtaining the original video data includes but is not limited to recorded video, public data set, monitoring system or synthetic video.

[0084] In step S202 of some embodiments, the original video frame sequence refers to a set of continuous single-frame images extracted from the original video data in chronological order. Each frame image represents a static picture of the video at a specific time point, and is arranged in a playback order to form an orderly image sequence. The video frame extraction method includes but is not limited to extracting frame images from the video at fixed time intervals, extracting key frames in the video, or extracting frames based on event triggering.

[0085] In step S203 of some embodiments, the initial video frame sequence refers to a set of frame images after stable images are screened out from the original video frame sequence. In one embodiment, the original video frame sequence is screened for stable images based on the optical flow method, thereby filtering out frames with poor quality in the initial video frame sequence due to jitter, blur or other unstable factors, and obtaining the initial video frame sequence.

[0086] In step S204 of some embodiments, the target video frame sequence refers to a set of frame images obtained after background removal technology is used on the basis of the initial video frame sequence, and the background part of each frame in the initial video frame sequence is removed, and only the human body area of ​​the target object is retained. The background removal method includes but is not limited to image difference method or background removal method based on deep learning, so as to achieve preliminary removal of the background, provide a data basis for subsequent human body image segmentation, and thus improve the segmentation accuracy of subsequent human body image segmentation.

[0087] In step S102 of some embodiments, the target video frames are classified based on the timestamp to obtain reference video frames and target video frames. The classification method is to classify the target video frames according to a preset timestamp threshold. For example, the target video frames include 10 seconds of video frames, and the preset timestamp threshold is 3 seconds. Then, the video of 0-3 seconds is determined as the reference video frame, and the video frame of the 4th second is determined as the target video frame.

[0088] It should be noted that the target mask data obtained after the subsequent image segmentation mask prediction of the target video frame will be used as the reference video frame. For example, the target video frame includes 10 seconds of video frames, the preset timestamp threshold is 3 seconds, the video of 0-3 seconds is determined as the reference video frame, the video frame of the 4th second is used as the target video frame, and then the image segmentation mask prediction is performed on the video frame of the 4th second, the video frame of 1-3 seconds is used as the reference video frame, the video frame of the 5th second is used as the target video frame, and the cycle is iterated until the last time the video frame of 6-9 seconds is used as the reference video frame and the video frame of the 10th second is used as the target video frame.

[0089] In step S103 of some embodiments, the reference mask data refers to binary data generated on the reference video frame for identifying the human body area, the data dimension of the binary data is consistent with the dimension of the frame in the reference video frame, and each pixel point on the binary data is 0 or 1, for example, 0 represents the background area, and 1 represents the human body area. For example, the segmentation mask can be generated by manual annotation, or the segmentation mask is generated for the reference video frame based on a pre-trained model. For example, in a fitness video of an indoor gym, the reference mask data is obtained based on the manual annotation of the athletes who are exercising.

[0090] It should be noted that the specific value of the binary data can be set as needed, and this application does not impose any specific restrictions.

[0091] See also Figure 3 In some embodiments, the preset human body segmentation mask prediction model includes an encoder and a decoder, and step S104 may include but is not limited to steps S301 to S303:

[0092] Step S301, performing segmented image encoding on a reference video frame and reference mask data based on an encoder to obtain reference key encoding features and reference value encoding features;

[0093] Step S302, performing segmentation image encoding on the target video frame based on the encoder to obtain target key encoding features and target value encoding features;

[0094] Step S303: Based on the decoder, segmentation mask prediction is performed on the reference key encoding feature, the reference value encoding feature, the target key encoding feature and the target value encoding feature to obtain target mask data.

[0095] In steps S301 to S303 shown in the embodiment of the present application, the reference video frame and the reference mask data are segmented and encoded by the encoder to obtain reference key encoding features and reference value encoding features, and then the target video frame is segmented and encoded by the encoder to obtain target key encoding features and target value encoding features. Finally, based on the decoder, segmentation mask prediction is performed on the reference key encoding features, reference value encoding features, target key encoding features and target value encoding features to obtain target mask data, so that the image feature information of the reference video frame and the mask feature information in the reference mask data are extracted by the encoder and converted into reference key encoding features and reference value encoding features, and then the image feature information of the target video frame is extracted by the encoder and converted into target key encoding features and target value encoding features. Finally, the reference key encoding features, reference value encoding features, target key encoding features and target value encoding features are segmented and predicted by the decoder to accurately predict the mask data of the target video frame.

[0096] See also Figure 4 In some embodiments, step S301 may include but is not limited to steps S401 to S402:

[0097] Step S401, calculating the sum of the reference video frame and the reference mask data to obtain reference mask frame data;

[0098] Step S402, performing segmented image encoding on the reference mask frame data based on the encoder to obtain reference key encoding features and reference value encoding features.

[0099] In steps S401 to S402 shown in the embodiment of the present application, the reference mask frame data is obtained by calculating the sum of the reference video frame and the reference mask data, thereby fusing the video frame and the mask data, so that the reference mask frame data can more comprehensively express the shape and position of the human body, and improve the expression ability of the reference mask frame data for the human body area. Then, the encoder is used to perform deep feature extraction on the reference mask frame data to generate reference key encoding features and reference value encoding features for capturing the temporal and spatial information and semantic features of the human body as part of the model memory, so as to assist the segmentation task of the target video frame, thereby improving the accuracy of human image segmentation in the target video frame.

[0100] In step S401 of some embodiments, the sum of the reference video frame and the reference mask data is calculated to obtain the reference mask frame data, so that the reference mask frame data can more comprehensively express the shape and position of the human body and enhance the expression ability of the reference mask frame data for the human body area.

[0101] See also Figure 5 In some embodiments, the encoder includes a convolutional layer, a residual layer, a pooling layer, and a fully connected layer, and step S402 includes but is not limited to steps S501 to S505:

[0102] Step S501, performing feature convolution on the reference mask frame data based on the convolution layer to obtain reference mask frame convolution features;

[0103] Step S502, performing residual connection on the convolutional features of the reference mask frame based on the residual layer to obtain residual features of the reference mask frame;

[0104] Step S503, performing pooling connection on the reference mask frame residual features based on the pooling layer to obtain reference mask frame pooling features;

[0105] Step S504, performing image encoding on the reference mask frame pooling features based on the fully connected layer to obtain reference mask frame key-value coding features;

[0106] Step S505, performing convolution dimensionality reduction based on the reference mask frame key-value coding features to obtain reference key coding features and reference value coding features.

[0107] Steps S501 to S505 shown in the embodiment of the present application, through multi-level convolution operations, deeply capture the subtle features and complex patterns in the mask data, extract features that can express the detailed differences between the human body and the background, and then alleviate the gradient vanishing problem of the network model through residual connections, promote the effective transmission of information, and make feature extraction more stable and efficient. Then, high-level semantic information is further extracted through the pooling layer, while reducing the spatial dimension of the feature map and reducing the computational complexity. Finally, the multi-dimensional features are mapped to the high-dimensional semantic space through the fully connected layer, and the expressive power of the features is enhanced to obtain the reference mask frame key-value coding features. Finally, convolution dimensionality reduction is performed based on the reference mask frame key-value coding features to obtain reference key coding features and reference value coding features, retaining the main information of the features from different dimensions, and providing a data basis for the subsequent human image segmentation of the target video frame.

[0108] In steps S501 to S105 of some embodiments, the encoder is a ResNet-18 network. ResNet-18 is a convolutional neural network architecture, which is composed of a convolutional layer, a residual layer, a pooling layer and a fully connected layer connected in sequence. The convolutional layer is composed of a convolutional kernel sublayer, a batch normalization sublayer, an activation function sublayer, and a maximum pooling sublayer. The residual layer is composed of four residual sublayers, each of which includes two convolutional sublayers and filters, and the fully connected layer is composed of at least one linear layer. The reference mask frame data is input into the convolution layer for feature convolution to obtain the reference mask frame convolution feature, and then the reference mask frame convolution feature is input into the residual layer for residual connection to obtain the reference mask frame residual feature, and then the reference mask frame residual feature is input into the pooling layer for pooling connection to obtain the reference mask frame pooling feature, and finally the reference mask frame pooling feature is input into the fully connected layer to obtain the reference mask frame key-value coding feature, and then the reference mask frame key-value coding feature is downsampled twice, the first downsampling is to downsample the reference mask frame key-value coding feature by dividing the number of channels by 8 to obtain the reference key coding feature, and the second downsampling is to downsample the reference mask frame key-value coding feature by dividing the number of channels by 2 to obtain the reference value coding feature.

[0109] In step S302 of some embodiments, the target video frame is input into the encoder, encoded through the convolutional layer, residual layer, pooling layer and fully connected layer of the encoder to obtain a target encoding feature, and then the target encoding feature is downsampled by the number of channels divided by 8 to obtain a target key encoding feature, and then the target encoding feature is downsampled by the number of channels divided by 2 to obtain a target value encoding feature.

[0110] See also Figure 6 In some embodiments, step S303 includes but is not limited to steps S601 to S603:

[0111] Step S601, generating a similarity graph according to the reference key coding feature, the target key coding feature and the reference value coding feature to obtain a spatiotemporal similarity graph feature;

[0112] Step S602, performing data splicing according to the spatiotemporal similarity graph feature and the target value coding feature to obtain the target coding feature;

[0113] Step S603: perform segmentation mask prediction on the target coding feature based on the decoder to obtain target mask data.

[0114] In steps S601 to S603 shown in the embodiment of the present application, a similarity graph is generated based on the reference key coding features, the target key coding features and the reference value coding features to obtain spatiotemporal similarity graph features, and then data is spliced ​​based on the spatiotemporal similarity graph features and the target value coding features to obtain target coding features. Finally, segmentation mask prediction is performed on the target coding features based on the decoder to obtain target mask data, thereby achieving accurate image segmentation mask prediction for the target video frame.

[0115] See also Figure 7 In some embodiments, step S601 may include but is not limited to steps S701 to S703:

[0116] Step S701, calculating the product between the reference key encoding feature and the target key encoding feature to obtain a similar graph feature;

[0117] Step S702, performing weight normalization processing on similar graph features to obtain normalized features;

[0118] Step S703, performing weighted sum processing according to the normalized features and the reference value coding features to obtain spatiotemporal similarity graph features.

[0119] In steps S701 to S703 shown in the embodiment of the present application, similarity graph features are obtained by calculating the product between the reference key coding features and the target key coding features, and then the similarity graph features are weighted normalized to obtain normalized features, which accurately capture the key features related to the target object, and weighted summation is performed based on the normalized features and the reference value coding features to obtain spatiotemporal similarity graph features, so that the spatiotemporal similarity graph features can make full use of the information of the reference video frame and the target video frame, enhance the ability to understand the dynamic changes and complex movements of the human body, and reduce the segmentation error caused by background interference or rapid movement.

[0120] In step S701 of some embodiments, the product between the reference key encoding feature and the target key encoding feature is calculated to obtain a similarity graph feature, as shown in formula (1):

[0121] S=KeyMemory·KeyQuery T (1)

[0122] Among them, S is the similarity graph feature, KeyMemory is the reference key encoding feature, KeyQuery is the target key encoding feature, and KeyQuery T Encode the transpose of features for the target key.

[0123] In step S702 of some embodiments, weight normalization is performed on similar graph features to obtain normalized features, as shown in formula (2):

[0124]

[0125] Among them, S is the similarity graph feature, the reference video frame contains multiple video frames, the total number of frames is N, and the i-th frame corresponds to S i ,j is the preset spatial size of the similarity graph. a i,j For S i,j The corresponding normalized features.

[0126] In step S703 of some embodiments, a weighted summation process is performed based on the normalized features and the reference value encoding features to obtain a spatiotemporal similarity graph feature, and the weighted summation is shown in formula (3):

[0127]

[0128] Among them, WeightedMemory is the spatiotemporal similarity graph feature, ValueMemory is the reference value encoding feature, and Value_Memory i The reference encoding value features of the i-th frame in .

[0129] In step S602 of some embodiments, data splicing is performed according to the spatiotemporal similarity graph feature and the target value coding feature to obtain the target coding feature. The data splicing is shown in formula (4):

[0130] Combined=Concat(WeightedMemory,ValueQuery) (4),

[0131] Among them, Combined is the target coding feature, ValueQuery is the target value coding feature, and Concat() is the preset data concatenation function.

[0132] In step S603 of some embodiments, segmentation mask prediction is performed on the target coding feature based on the decoder, and the decoder is composed of a first upsampling and convolution layer, a first residual block, a second upsampling and convolution layer, a second residual block, a third upsampling and convolution layer, a third residual block and an output layer connected in sequence. Among them, the first upsampling and convolution layer includes a bilinear interpolation layer, a convolution layer, a batch normalization layer and an activation function connected in sequence. The residual block is composed of multiple residual layers in sequence and residual connections are performed. This application does not limit the number of residual layers. The output layer is composed of a convolution layer and a Sigmoid activation function, and Sigmoid outputs a binary mask, that is, outputs the target mask data.

[0133] The first upsampling and convolution layer, the second upsampling and convolution layer, and the third upsampling and convolution layer network structures are similar, and will not be repeated here. The first residual block, the second residual block, and the third residual block network structures are similar, and will not be repeated here.

[0134] The image size is continuously restored through upsampling and convolution layers in the decoder, and residual connections are used to help the decoder better extract and retain feature information, avoiding information loss or degradation during multiple upsampling and convolution processes. This corresponds to the residual blocks in the encoder, keeping the overall network structurally consistent, thereby improving the decoding accuracy of the decoder.

[0135] In step S105 of some embodiments, human body image segmentation is performed on the target video frame according to the target mask data to obtain a target human body image. Human body image segmentation refers to pixel-level image segmentation of the target video frame. The target mask data is consistent with the image size of the target video frame. The target mask data is a matrix composed of 0 or 1, and each pixel is 0 or 1. In one embodiment, 0 represents the background of the target video frame, and 1 represents the human body of the target video frame. The pixel point at the corresponding position of each 1 in the target mask data is taken out to obtain the target human body image.

[0136] See also Figure 8 The embodiment of the present application further provides a human image segmentation device, which can implement the above human image segmentation method, and the device includes:

[0137] The sequence acquisition module 801 is used to acquire a target video frame sequence containing a target object; wherein the target video frame sequence has a timestamp;

[0138] The video classification module 802 is used to classify the target video frame sequence based on the timestamp to obtain a reference video frame and a target video frame; wherein the timestamp of the reference video frame is before the timestamp of the target video frame;

[0139] The mask generation module 803 is used to generate a segmentation mask according to a reference video frame to obtain reference mask data;

[0140] The mask prediction module 804 is used to perform image segmentation mask prediction on the target video frame based on the preset human body segmentation mask prediction model, the reference video frame and the reference mask data to obtain the target mask data;

[0141] The image segmentation module 805 is used to perform human image segmentation on the target video frame according to the target mask data.

[0142] The specific implementation of the human image segmentation device is basically the same as the specific embodiment of the above-mentioned human image segmentation method, and will not be repeated here.

[0143] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned human image segmentation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0144] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0145] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0146] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the human body image segmentation method of the embodiment of this application;

[0147] Input / output interface 903, used to implement information input and output;

[0148] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0149] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0150] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0151] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the human body image segmentation method is implemented.

[0152] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0153] The human image segmentation method, human image segmentation device, electronic device and storage medium provided in the embodiments of the present application obtain a target video frame sequence containing a target object, and then classify the target video frame sequence based on the timestamp of the target video frame sequence to obtain a reference video frame and a target video frame, and then generate a segmentation mask based on the reference video frame to obtain reference mask data, and then perform image segmentation mask prediction on the target video frame based on a preset human segmentation mask prediction model, the reference video frame and the reference mask data to obtain target mask data, thereby realizing mask prediction of the target video frame based on combining spatiotemporal information, thereby improving the human body segmentation effect in a series of video frame sequences, and having accurate segmentation accuracy even under fast motion or complex actions, and finally performing human body image segmentation on the target video frame according to the target mask data to obtain a target human image, thereby realizing accurate human body image segmentation of the video frame sequence.

[0154] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0155] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0156] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0157] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0158] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0159] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0160] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0161] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0163] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0164] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A human body image segmentation method, characterized in that: The method comprises: Acquire a target video frame sequence containing a target object; wherein the target video frame sequence has a timestamp; Classifying the target video frame sequence based on the timestamp to obtain a reference video frame and a target video frame; wherein the timestamp of the reference video frame is before the timestamp of the target video frame; Generating a segmentation mask according to the reference video frame to obtain reference mask data; Performing image segmentation mask prediction on the target video frame based on a preset human body segmentation mask prediction model, the reference video frame and the reference mask data to obtain target mask data; Perform human image segmentation on the target video frame according to the target mask data.

2. The method according to claim 1, characterized in that The preset human body segmentation mask prediction model includes an encoder and a decoder; the image segmentation mask prediction is performed on the target video frame based on the preset human body segmentation mask prediction model, the reference video frame and the reference mask data to obtain the target mask data, including: Based on the encoder, segment the reference video frame and the reference mask data to obtain reference key encoding features and reference value encoding features; Based on the encoder, the target video frame is segmented and encoded to obtain a target key encoding feature and a target value encoding feature; Based on the decoder, segmentation mask prediction is performed on the reference key encoding feature, the reference value encoding feature, the target key encoding feature and the target value encoding feature to obtain the target mask data.

3. The method according to claim 2, characterized in that The performing segmentation mask prediction on the reference key encoding feature, the reference value encoding feature, the target key encoding feature and the target value encoding feature based on the decoder to obtain the target mask data includes: Generate a similarity graph based on the reference key encoding feature, the target key encoding feature and the reference value encoding feature to obtain a spatiotemporal similarity graph feature; Perform data splicing according to the spatiotemporal similarity graph feature and the target value coding feature to obtain a target coding feature; Based on the decoder, segmentation mask prediction is performed on the target coding feature to obtain the target mask data.

4. The method according to claim 3, characterized in that: Generate a similarity graph according to the reference key encoding feature, the target key encoding feature and the reference value encoding feature to obtain a spatiotemporal similarity graph feature, including: Calculating the product between the reference key encoding feature and the target key encoding feature to obtain a similarity graph feature; Performing weight normalization processing on the similar graph features to obtain normalized features; The spatiotemporal similarity graph feature is obtained by performing weighted summation processing on the normalized feature and the reference value coding feature.

5. The method according to claim 2, characterized in that: The segmenting image encoding of the reference video frame and the reference mask data based on the encoder to obtain reference key encoding features and reference value encoding features includes: Calculating the sum of the reference video frame and the reference mask data to obtain reference mask frame data; The reference mask frame data is segmented and encoded based on the encoder to obtain the reference key encoding feature and the reference value encoding feature.

6. The method according to claim 5, characterized in that The encoder includes a convolutional layer, a residual layer, a pooling layer and a fully connected layer; the segmentation image encoding of the reference mask frame data based on the encoder to obtain the reference key encoding feature and the reference value encoding feature includes: Performing feature convolution on the reference mask frame data based on the convolution layer to obtain reference mask frame convolution features; Performing residual connection on the convolutional features of the reference mask frame based on the residual layer to obtain residual features of the reference mask frame; Based on the pooling layer, the reference mask frame residual features are pooled and connected to obtain reference mask frame pooling features; Performing image encoding on the reference mask frame pooling features based on the fully connected layer to obtain reference mask frame key-value coding features; Convolutional dimension reduction is performed based on the reference mask frame key-value coding features to obtain the reference key coding features and the reference value coding features.

7. The method according to any one of claims 1 to 6, characterized in that: The step of obtaining a target video frame sequence containing a target object comprises: Acquire original video data containing the target object; Extracting video frames according to the original video data to obtain an original video frame sequence; Performing stable picture screening according to the original video frame sequence to obtain an initial video frame sequence; Background elimination is performed according to the initial video frame sequence to obtain the target video frame sequence.

8. A human body image segmentation device, characterized in that: The device comprises: An acquisition sequence module is used to acquire a target video frame sequence containing a target object; wherein the target video frame sequence has a timestamp; A video classification module, used for classifying the target video frame sequence based on the timestamp to obtain a reference video frame and a target video frame; wherein the timestamp of the reference video frame is before the timestamp of the target video frame; A mask generation module, used to generate a segmentation mask according to the reference video frame to obtain reference mask data; A mask prediction module, configured to perform image segmentation mask prediction on the target video frame based on a preset human body segmentation mask prediction model, the reference video frame and the reference mask data to obtain target mask data; An image segmentation module is used to perform human image segmentation on the target video frame according to the target mask data.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the human body image segmentation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the human body image segmentation method according to any one of claims 1 to 7 is implemented.

Images