A method for training a human target segmentation model, a human target segmentation method, and related devices

Abstract

The present application relates to a kind of training human body target segmentation model method, human body target segmentation method and related device, human body target segmentation network includes background coding network and human body segmentation network, the embodiment of the present application when training human body target segmentation model, background area in original image, background area is cut after using multiple different sizes, then based on multiple different scene images, modal shift is carried out to the background area after cutting, obtain the background image set of the background image including multiple sizes, multiple modal, background image set is input into background coding network and is encoded to obtain background feature map, and background feature map is used for model training. So that human body target segmentation network is more focused on the change of feature information and boundary feature of background image when training, and then the human body target segmentation model obtained can accurately segment out human body target, reduce the interference of complex background and the generation of artifact, reduce the dithering of human body target segmentation boundary.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target segmentation, in particular to a method for training a human target segmentation model, a human target segmentation method and related devices. BACKGROUND

[0002] In the digital era, multimedia digital content has become an indispensable part of people's lives, covering text, audio, images and video, etc. With the popularity of mobile devices equipped with cameras and sensors, video has become a new way of communication between Internet users, such as online video communication methods such as video conferencing, video calls, etc. This trend has enabled a series of video content understanding technologies and related applications to develop rapidly, enabling users in different locations to have face-to-face conversations through communication devices and networks. Human body segmentation of video is one of the core technologies and has been widely developed in the field of sports health, requiring real-time segmentation of the main human body and separation of the main human body from the complex background, so as to accurately generate human motion scenes in different backgrounds and enhance entertainment. However, in actual application, the scene of human motion is complex, and when segmenting the human region, some complex background images are also segmented, resulting in obvious artifacts and obvious jitter of the motion target segmentation boundary. SUMMARY

[0003] The embodiments of the present application provide a method for training a human target segmentation model, a human target segmentation method and related devices, which can accurately segment the human target, reduce the interference of complex background and the generation of artifacts, and reduce the jitter of the human target segmentation boundary.

[0004] To solve the above technical problems, in a first aspect, the embodiments of the present application provide a method for training a human target segmentation model, a human target segmentation network comprising a background encoding network and a human segmentation network, the human segmentation network comprising an encoding module, a feature fusion module and a decoding module, the method comprising:

[0005] obtaining a training set, the training set comprising a plurality of original images containing human targets, the original images having real labels corresponding to the human targets;

[0006] extracting a background region in the original image, and obtaining a background image set based on the background region, wherein the background image set is obtained by cropping the background region with a plurality of different sizes and then performing modal transfer, and the modal transfer comprises image transfer of a plurality of different scenes;

[0007] inputting the background image set into the background encoding network for encoding to obtain a background feature map;

[0008] inputting the original image into the encoding module for encoding to obtain an original feature map;

[0009] inputting the background feature map and the original feature map into the feature fusion module for feature fusion to obtain a fusion feature map;

[0010] inputting the fusion feature map into the decoding module for decoding to obtain a predicted label corresponding to the human target;

[0011] calculating a loss between the real label and the predicted label based on a loss function, and iteratively training the human target segmentation network according to the loss until the human target segmentation network converges to obtain the human target segmentation model.

[0012] In some embodiments, the background region in the original image is extracted, and a background image set is obtained based on the background region, including:

[0013] obtaining a human key point of the original image;

[0014] based on the human key point, determining a human rectangular frame, and extracting an image outside the human rectangular frame as a background region;

[0015] cropping the background region in multiple different sizes to obtain background images of multiple different sizes;

[0016] based on images of multiple different scenes, performing modal transfer on the background images of each size to obtain a background image set, the background image set including background images of multiple sizes and multiple modalities.

[0017] In some embodiments, the background image set is obtained by performing modal transfer on the background images of each size based on images of multiple different scenes, including:

[0018] based on images of multiple different scenes, performing modal transfer on the background images of each size according to a first formula to obtain a background image set, the first formula being:

[0019] M bg ,S bg =F(θ(I bg ))

[0020] M style ,S style =F(θ(I style ))

[0021]

[0022] wherein θ() represents converting an image into LAB format, Istyle represent images in different scenes, I bg represents a first background image, F() is a function of extracting mean and standard deviation of an image in LAB format, M bg bg are the mean and standard deviation of F(θ(I bg )), respectively, M style style are the mean and standard deviation of F(θ(I style )), respectively, is a background image after modal migration.

[0023] In some embodiments, the modal migration is performed on the background image of each size based on a plurality of images in different scenes to obtain a set of background images, and the method further comprises:

[0024] adjusting display parameters of the background image, and performing Gaussian noise blur processing, wherein the display parameters include brightness, contrast, and saturation.

[0025] In some embodiments, the inputting the background feature map and the original feature map into the feature fusion module to perform feature fusion to obtain a fusion feature map comprises:

[0026] inputting the background feature map and the original feature map into the feature fusion module, performing convolution operation on the original feature map to obtain a first original feature map;

[0027] performing dilated convolution operation on the background feature map to obtain a first background feature map;

[0028] performing residual connection on the first original feature map and the first background feature map to obtain a first fusion feature map;

[0029] connecting the first fusion feature map and the original feature map to obtain the fusion feature map.

[0030] In some embodiments, the loss function is:

[0031]

[0032] wherein x i represents a true label of a human target in the i-th frame of the original image, represents a predicted label of a human target in the i-th frame of the original image, L2 represents a mean standard deviation loss of calculating the true label and the predicted label, L1 represents a mean absolute difference loss of calculating the true label and the predicted label, and ∝ is a constant.

[0033] To solve the above technical problems, in a second aspect, an embodiment of the present application provides a human target segmentation method, comprising:​​

[0034] obtaining an image to be processed;

[0035] inputting the image to be processed into a human target segmentation model to obtain a predicted label of a human target in the image to be processed, wherein the human target segmentation model is obtained by using any one of the above methods for training a human target segmentation model;

[0036] segmenting a human target image from the image to be processed according to the predicted label of the human target.

[0037] To solve the above technical problems, in a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory connected in communication, the memory storing computer program instructions executable by the processor, and when the computer program instructions are called by the processor, the processor executes any one of the above methods for training a human target segmentation model or any one of the above human target segmentation methods.

[0038] To solve the above technical problems, in a fourth aspect, an embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing computer program instructions, and a processor running the computer program instructions to execute any one of the above methods for training a human target segmentation model or any one of the above human target segmentation methods.

[0039] The embodiment of the present application has the following beneficial effects: Different from the prior art, the method for training a human target segmentation model provided by the embodiment of the present application, the human target segmentation network comprises a background encoding network and a human segmentation network, the human segmentation network comprises an encoding module, a feature fusion module and a decoding module, the method comprises: obtaining a training set, the training set comprising a plurality of original images containing human targets, the original images having real labels corresponding to the human targets; extracting a background region in the original image, obtaining a background image set based on the background region, wherein the background image set is obtained by cropping the background region with a plurality of different sizes and then performing modal transfer, and the modal transfer comprises image transfer in a plurality of different scenes; inputting the background image set into the background encoding network for encoding to obtain a background feature map; inputting the original image into the encoding module for encoding to obtain an original feature map; inputting the background feature map and the original feature map into the feature fusion module for feature fusion to obtain a fusion feature map; inputting the fusion feature map into the decoding module for decoding to obtain a predicted label corresponding to the human target; calculating a loss between the real label and the predicted label based on a loss function, and iteratively training the background encoding network and the human segmentation network according to the loss until the human target segmentation network converges to obtain a human target segmentation model.

[0040] The embodiment of the present application is used for training a human target segmentation model. When training the human target segmentation model, the background region in the original image is cropped in multiple different sizes, and then the cropped background region is subjected to modality migration based on images of multiple different scenes to obtain a background image set, wherein the background image set includes background images of multiple sizes and multiple modalities. The background image set is input into a background encoding network for encoding to obtain a background feature map, and the background feature map is used for model training. Thus, the human target segmentation network pays more attention to the feature information and boundary feature changes of the background image during training, and the human target segmentation model obtained in this way can accurately segment the human target, reduce the interference of complex background and the generation of artifacts, reduce the jitter of the human target segmentation boundary, and the segmentation result is clearer and more reliable, improving the stability of the segmentation and the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings described below only show some of the embodiments of the present application, and should not be regarded as a limitation to the protection scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0042] Figure 1 is an application scenario diagram of the method for training a human target segmentation model provided by some embodiments of the present application;

[0043] Figure 2 is a structural diagram of an electronic device provided by some embodiments of the present application;

[0044] Figure 3 is a whole network structure diagram of a human target segmentation model provided by some embodiments of the present application;

[0045] Figure 4 is a flow diagram of the method for training a human target segmentation model provided by some embodiments of the present application;

[0046] Figure 5 is Figure 4 is a sub-flow diagram of step S200 in the method for training a human target segmentation model shown in the embodiments;

[0047] Figure 6 is Figure 4 is a sub-flow diagram of step S500 in the method for training a human target segmentation model shown in the embodiments;

[0048] Figure 7 is a flow diagram of a human target segmentation method provided by some embodiments of the present application. DETAILED DESCRIPTION

[0049] In order to make the objects and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. The following detailed description of the embodiments of the present application in the drawings is not intended to limit the scope of the present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present application.

[0050] It should be noted that, if there is no conflict, each feature in the embodiments of the present application can be combined with each other, and all within the protection scope of the present application. In addition, although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the order in the flowchart. In addition, the "first", "second", "third" and the like used herein do not limit the data and execution order, but only distinguish the same items or similar items with basically the same function and effect.

[0051] Unless otherwise defined, all technical and scientific terms used in the specification are the same as those commonly understood by those skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments and are not used to limit the present application. The term "and / or" used in the specification includes any and all combinations of one or more related listed items. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as there is no conflict.

[0052] In order to facilitate the understanding of the method provided by the embodiments of the present application, first, the terms involved in the embodiments of the present application are introduced:

[0053] (1) Neural network

[0054] The neural network can be composed of neural units, and can specifically be a neural network with an input layer, a hidden layer and an output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The neural network with many layers of hidden layers is called a deep neural network (DNN). The work of each layer in the neural network can be described by the mathematical expression y = a(Wx + b). From a physical perspective, the work of each layer in the neural network can be understood as completing the transformation of the input space (a set of input vectors) to the output space (i.e., the row space of the matrix to the column space) through five operations on the input space, which include: 1, dimensionality increase / dimensionality decrease; 2, magnification / reduction; 3, rotation; 4, translation; and 5, "bending". The operations of 2, 3 are completed by "Wx", the operation of 4 is completed by "+b", and the operation of 5 is completed by "a()". The reason why "space" is used here is that the objects to be classified are not single things, but a class of things, and the space refers to the set of all individuals of this class. W is a weight matrix of each layer of the neural network, and each value in the matrix represents the weight value of a neuron in the layer. The matrix W determines the spatial transformation of the input space to the output space described above, that is, the W of each layer of the neural network controls how to transform the space. The purpose of training the neural network is to obtain the weight matrix of all layers of the trained neural network. Therefore, the training process of the neural network is essentially learning the way to control the spatial transformation, and more specifically, learning the weight matrix.

[0055] It should be noted that in the embodiments of the present application, the models used based on machine learning tasks are essentially neural networks. Common components in neural networks include convolution layers, pooling layers and normalization layers, etc. By assembling these common components in neural networks, a model is designed. When the model parameters (weight matrix of each layer) are determined so that the model error meets the preset condition or the number of adjusted model parameters reaches the preset threshold, the model converges.

[0056] (2) Convolution

[0057] Convolution is a mathematical operation widely used in signal processing, image processing and machine learning. Convolution is very common in image processing. When a convolution is performed on an image, a small filter or kernel function can be applied to each pixel of the image, and a new pixel is generated by weighted sum of adjacent pixels. This process can be used to implement many image processing tasks, such as edge detection, image enhancement and blurring, etc.

[0058] In machine learning, a Convolutional Neural Network (CNN) is a deep learning model based on convolution operations. CNNs are widely used in image recognition, object detection, and speech processing tasks, and their main advantage is their ability to automatically learn and extract features from input data. The convolutional layer is a core component of CNNs, which performs convolution operations by sliding a convolution kernel over the input data to effectively capture local patterns and structural information in the input data.

[0059] (3) Convolutional Layer

[0060] A convolutional layer (Convolutional Layer) is a core component of a convolutional neural network, used for feature extraction and convolution operations on input data. The convolutional layer consists of a set of convolution kernels, each of which can be regarded as a feature detector for detecting a specific feature in the input data. The convolutional layer generates an output feature map by sliding the convolution kernel over the input data and performing convolution operations on different positions of the input data. The convolutional layer uses convolution operations to extract and map features from input data to capture spatial local features in the input data, and reduces the number of parameters through weight sharing, thereby achieving effective feature learning and representation.

[0061] Specifically, the input to the convolutional layer is a multi-channel feature map (such as an image or the output of the previous layer), each channel corresponds to a feature, and the convolutional layer supports multi-channel input data and multi-channel convolution kernels, which can extract and integrate multiple feature information. The convolution kernel performs element-wise multiplication with the input data and sums up the results to obtain a single pixel value on the output feature map. By sliding the convolution kernel over the input data and performing convolution operations at each position, an output feature map corresponding to the size of the input data can be generated. Each position in the output feature map corresponds to a local region in the input data, and through convolution operations, the convolutional layer can extract local patterns and features in the input data.

[0062] The convolutional layer plays an important role in deep learning, as multiple convolution kernels extract features from input data at different positions and scales, gradually establishing high-level abstract representations and feature learning of data. The convolutional layer is usually combined with other types of neural network layers (such as pooling layers, activation function layers, fully connected layers, etc.) to form a complete convolutional neural network, which is used to solve computer vision tasks such as image segmentation, image classification, and object detection, and is also widely used in natural language processing and other deep learning tasks in other fields.

[0063] (4) Convolution Kernel

[0064] Convolutional Kernel, also known as Filter or Feature Detector, is the core component of Convolutional Neural Networks. In image processing and computer vision, Convolutional Kernel is a small matrix or tensor used for convolution operation on images. Convolutional Kernel is usually a square matrix, and its size can be defined according to the requirements of the task, with common sizes being 1*1, 3*3, 5*5, and 7*7. Convolutional Kernel contains a set of weight parameters, which are used for weighted summation with input data in convolution operation. In convolution operation, Convolutional Kernel slides on input data and performs element-wise multiplication and summation with the corresponding region of input data at each position to generate a single pixel of output.

[0065] Convolutional Kernel plays a crucial role in Convolutional Neural Networks (CNN). By designing different Convolutional Kernels, the network can learn different features, such as edges, textures, corners, etc. Each Convolutional Kernel can be regarded as a kind of feature detector, which is sensitive to a certain specific feature of input data and calculates it through sliding window to extract the existence degree of the feature at different positions. In deep learning tasks, the parameters of Convolutional Kernel can be automatically learned through training process, or can be pre-set through experience. Through backpropagation algorithm, neural network can automatically adjust the weight parameters in Convolutional Kernel according to the feedback signal of loss function, so that the neural network can better adapt to the requirements of the task and learn more high-level feature representation.

[0066] (5) Dilated Convolution

[0067] In the field of computer vision and deep learning, Dilated Convolution, also known as Atrous Convolution, is a technique that modifies the traditional convolution operation. Dilated Convolution introduces intervals (also known as dilation rates) inside the convolution kernel, so that the convolution kernel performs a jump operation on the input feature map, thereby expanding the receptive field while reducing the number of parameters. In traditional convolution operation, each element of the convolution kernel performs dot product operation with the corresponding region of the input feature map. For example, a 3*3 convolution kernel will perform dot product operation with each 3*3 region of the input feature map to obtain a pixel of the output feature map.

[0068] Dilated convolution introduces the concept of dilation rate (or interval), which determines the distance between elements within the convolution kernel. For example, if the dilation rate of a 3*3 convolution kernel is 1, each element performs dot product operation with the corresponding input region, which is the same as traditional convolution. But if the dilation rate is 2, there is a pixel interval between the elements of the convolution kernel, that is, only every other pixel region of the input feature map is dot product operation. In this way, the receptive field of the convolution kernel can be expanded without increasing the size of the convolution kernel, so as to better capture the global information in the image. The application scenarios of dilated convolution in deep learning usually include image segmentation and target detection tasks. By introducing the dilation rate, the neural network can obtain broader context information under a smaller receptive field, which helps to improve the performance of the model.

[0069] The following illustrates an exemplary application of the electronic device provided by the embodiments of the present application for training a human target segmentation model or for human target segmentation. The electronic device provided by the embodiments of the present application can be various suitable types of devices with certain computing and control capabilities, such as notebook computers, desktop computers, or mobile devices. As an example, refer to Figure 1 Figure 1 is a schematic diagram of an application scenario of the method for training a human target segmentation model provided by some embodiments of the present application.

[0070] Specifically, when the electronic device 100 is used for training a human target segmentation model, the electronic device 100 can be used to obtain training image data and build a human target segmentation model. For example, a person skilled in the art can download prepared training image data on the electronic device 100, and build a network structure of the human target segmentation model, and can train the human target segmentation model when obtaining the training image data. The training image data includes a plurality of original images containing human targets. It can be understood that the electronic device 100 can also be used to obtain image data to be processed. For example, a person skilled in the art packs the image data to be processed and sends it to the electronic device 100 through a communication network, so that the electronic device 100 obtains the image data to be processed. In some embodiments, when the electronic device 100 is used for human target segmentation, the electronic device 100 sends the training image data or the image data to be processed to the controller (not shown in the figure) in the electronic device 100, so that the controller uses the built-in human target segmentation model to perform human target segmentation on the training image data or the image data to be processed, and obtains the human target segmentation result. Figure 1

[0071] ​​In some embodiments, the electronic device 100 can locally execute the method for training the human target segmentation model provided in the embodiment of the present invention to complete the training of the designed human target segmentation model using training image data, determine the final model parameters, and thus the human target segmentation model is configured with the final model parameters to obtain the human target segmentation model. In other embodiments, the electronic device 100 can be connected to a server through a communication network, and send the training image data and the constructed human target segmentation model stored on the electronic device 100 by a person skilled in the art to the server, the server receives the training image data and the human target segmentation model, and iteratively trains the human target segmentation model using the training image data to determine the final model parameters, and then sends the final model parameters to the electronic device 100, the electronic device 100 receives and saves the final model parameters, so that the human target segmentation model is configured with the final model parameters to obtain the human target segmentation model. It is easy to understand that the aforementioned communication network can be a wide area network or a local area network, or a combination of the two.

[0072] The structure of the electronic device in the embodiment of the present invention is described below. Figure 2 As shown, Figure 2 1 is a schematic diagram of the structure of an electronic device 100 provided in some embodiments of the present invention. The electronic device 100 includes at least one processor 110 and a memory 120 ( Figure 2 In the example of bus system connection and one processor, the components in the electronic device 100 are coupled together through the bus system 130, which is used to realize the connection and communication between these components. It is easy to understand that the bus system 130 includes not only the data bus, but also the power bus, control bus and status signal bus. However, for the sake of clarity and brevity, the following are omitted. Figure 2 In FIG, various buses are labeled as bus system 130. Those skilled in the art will understand that Figure 2 The structure shown is only exemplary and does not limit the structure of the electronic device 100. For example, the electronic device 100 may further include Figure 2 More or fewer components than shown, or with Figure 2 Different configurations shown.

[0073] The processor 110 is configured to provide operation and control capabilities to control the electronic device 100 to perform corresponding tasks, for example, to control the electronic device 100 to perform any of the above methods for training a human target segmentation model, or any of the above human target segmentation methods. It can be understood that the processor 110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; or a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0074] The memory 120 is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the methods for training a human target segmentation model or the human target segmentation methods in the embodiments of the present application. The processor 110 can implement any of the above methods for training a human target segmentation model, or any of the above human target segmentation methods by running the non-transitory software programs, instructions and modules stored in the memory 120. The memory 120 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 120 can also include a memory remotely arranged with respect to the processor, which can be connected to the processor 110 through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0075] Referring to Figure 3 illustrated, Figure 3 is a schematic diagram of the overall network structure of the human target segmentation model provided by some embodiments of the present application. Specifically, after obtaining a training image I, the training image I is input into a background extraction model, and the background region of the training image I is cropped with multiple different sizes, and then the cropped background region is respectively subjected to modal transfer based on images of multiple different scenes, to obtain background images of multiple sizes and multiple modalities, and form a background image set Then the background image set is input into a background encoding network for encoding to obtain a background feature map corresponding to the frame image and the background feature map Used for subsequent model iterative training.

[0076] The human body segmentation network includes an encoding module, an attention feature fusion module (i.e., a feature fusion module), and a decoding module. Specifically, after obtaining the training image I, the training image I is input into the encoding module to obtain the original feature map of the corresponding frame image. Then the background feature map of the corresponding frame image and the original feature map Input into the attention feature fusion module for feature fusion to obtain the fusion feature map Then the fusion feature map Input to the decoding module, classify each pixel into background category or human target category, generate human target segmentation results, and output the predicted label of each pixel in the corresponding frame image Get the predicted label corresponding to the human target in the corresponding frame image

[0077] It can be understood from the above that the method for training a human target segmentation model or the method for human target segmentation provided by the embodiments of the present invention can be implemented by various suitable types of electronic devices with certain computing and control capabilities, such as the above-mentioned electronic devices, or by other devices with certain computing, processing and control capabilities that are communicatively connected to the electronic devices, such as servers, smart terminals, etc. The following describes the method for training a human target segmentation model or the method for human target segmentation provided by the embodiments of the present invention in conjunction with exemplary applications and implementations of electronic devices provided by the embodiments of the present invention.

[0078] See Figure 4 As shown, Figure 4 : is a flow chart of a method for training a human target segmentation model provided by some embodiments of the present invention. It is understandable that the execution subject of the method for training a human target segmentation model can be the above-mentioned electronic device, and the method for training a human target segmentation model includes but is not limited to the following steps S100-S600:

[0079] S100: Acquire a training set, where the training set includes a plurality of original images containing human targets, and the original images have real labels corresponding to the human targets.

[0080] In a specific application, a user or a training personnel can collect a plurality of different original images from various data sources and form a data set for training the human target segmentation model by using the plurality of original images. Understandably, in the obtained training image data set, each original image contains one or more human targets, and the human targets in the original images can be static or moving. Obviously, the method for training the human target segmentation model is applicable to segmenting human targets in any original image, and has better segmentation effect especially when the segmented human target is a moving human target.

[0081] After obtaining the original image data for training the human target segmentation model, the pixels in the original image are labeled with real labels according to the human targets in the original image. In some embodiments, the labels are divided into two categories: human target and background, and it can be agreed that 0 represents the background category and 1 represents the human target category. Therefore, the pixels belonging to the human target category will be labeled as 1, and the pixels belonging to the background category will be labeled as 0. After labeling the real labels of the pixels in the original image, each pixel in the original image is assigned a specific category or label for identifying the semantic category or target type to which each pixel in the original image belongs, so that each pixel in each original image of the training data set has a corresponding real label. Understandably, in the process of obtaining the training set, the labels of the original images can be manually labeled by professionals or labeled by using automatic labeling technology. Obviously, when the original image data is obtained as the training set, the labels of the original images are the real labels of the human targets and the backgrounds in the original images, and the human target segmentation result obtained by inputting the original image into the human target segmentation model is the predicted label output after segmenting the human target in the original image.

[0082] S200: Extract the background region in the original image, and obtain a background image set based on the background region, wherein the background image set is obtained by cropping the background region in a plurality of different sizes and then performing modal transfer, and the modal transfer includes image transfer in a plurality of different scenes.

[0083] The human target segmentation network is a computer model for semantic segmentation of images or videos. The target of the human target segmentation network is to label each pixel in an image as a target category or a semantic category to which it belongs, and to realize fine classification of image pixels. For example, in an embodiment of the present application, the human target segmentation network is used to classify each pixel in an input original image as belonging to a background category or a human target category. In some embodiments, the human target segmentation network is composed of a background encoding network and a human segmentation network, which are respectively responsible for extracting background features and human target features, and combining them for pixel-level classification.

[0084] The background encoding network focuses on learning the feature representation of the background image and is composed of multiple convolutional layers and pooling layers, which can capture different features of the image at different levels. The background encoding module is the core component of the background encoding network and is responsible for converting the input background image into a corresponding background feature map to capture important semantic information of the background image.

[0085] The human segmentation network is a model based on a convolutional neural network (CNN) and is composed of an encoding module, a feature fusion module, and a decoding module. The human segmentation network adopts an encoding module-feature fusion module-decoding module structure and is used for learning human target features and realizing pixel-level segmentation. The encoding module is responsible for extracting the original feature representation of the human target and the background in the original image to obtain an original feature map and gradually reducing it to a lower resolution while retaining important semantic information. The encoding module is composed of a series of convolutional layers and pooling layers and is used for learning high-level semantic features. The feature fusion module fuses the background feature map extracted by the background encoding network and the original feature map extracted by the encoding module through connection, weighted summation, or other ways to combine the feature information of the background and the feature information of the human target together to obtain a fused feature map. The decoding module receives the fused feature map after fusion and restores the low-resolution features to the original resolution through a series of deconvolution, upsampling, and jump connection operations and generates a final human target segmentation result, i.e., outputs the predicted label of each pixel in the original image. The goal of the decoding module is to restore the human target segmentation result and output a pixel-level segmentation mask to classify each pixel as background or human target.

[0086] In some embodiments, the human segmentation network can be a model constructed based on a U2NET (Unified U-Net) network, wherein the U2NET model adopts a U-Net architecture. The U2NET model can segment the input original image into foreground and background with details and accuracy and generate a binary mask with the same size as the original image by learning a large amount of image data. The U2NET model performs well in image segmentation tasks, especially in handling complex edges and details with high accuracy, and can be used in image editing, computer vision research, and autonomous driving, etc.

[0087] After obtaining the original image, since the original image collected in the data source is usually a real image without processing or modification, the obtained original image needs to be preprocessed and cleaned, including image size adjustment, data enhancement, pixel value normalization and standardization processing, etc., to ensure that the original image meets the input requirements of the background coding network. When importing the original image data into the target segmentation model for training, since the relative importance of each dimension is not clear, the original image data needs to be standardized first. In some embodiments, the original image can be standardized by using a normalization method, and the formula of the normalization method is as follows:

[0088] wherein,

[0089] is the mean, s is the standard deviation, the standardization processing is to subtract the mean value from the pixel value of the image and divide by the standard deviation, so that the pixel value distribution of the image is close to the standard normal distribution with a mean of 0 and a standard deviation of 1, and the pixel value of the original image is processed into an array with a mean of 0 and a standard deviation of 1. If the original image has multiple channels (such as an RGB image), each channel can be standardized independently. In this case, the mean and standard deviation of each channel can be calculated, and independent mean and standard deviation standardization can be performed on each channel. When using image processing libraries (such as OpenCV, PIL in Python, etc.) or deep learning frameworks (such as TensorFlow, PyTorch, etc.), image standardization processing can be realized through corresponding functions and methods. For mean and standard deviation standardization, corresponding calculation and operation functions can be used. In some embodiments, after obtaining the mean and standard deviation, the Normalize function of transforms can be used for standardization processing. It is easy to understand that in addition to using the normalization method to standardize the original image, other ways such as normalization, normalization method, etc. can also be used for standardization processing.

[0090] After standardizing the obtained original image, the human body target detection or human body segmentation algorithm is used to mark the human body target bounding box in the original image, and then the background area unrelated to the human body target is cropped or extracted from the original image, and the background area is saved or processed as an independent background image, so as to obtain the background area in the original image. It can be understood that the background image refers to the part of the image that does not contain any human body target or region of interest, and it can also be understood as the image content in the original image except the human body target.

[0091] After obtaining the background region of the original image, the background region is cropped with multiple different sizes. By changing the size of the cropping size, applying different size transformations, or using image pyramid techniques, the background region is represented at different sizes, making the background image more robust and rich. By cropping the background region at multiple sizes, multiple size background images are obtained. Each size background image contains different background feature information in the original image, making each background image different in semantics and details. After obtaining multiple size background images, they can be saved as independent image files or used for training in subsequent training and learning tasks.

[0092] Modal migration refers to combining the visual features of one image with the semantic content of another image to generate a new image with the style or features of the second image. Multiple different scene images refer to image data collected from multiple different scenes or sources, which come from different places, environments or times, and have their own unique background information.

[0093] Based on multiple different scene images, modal migration is performed on each size background image. The style or features of each size background image are respectively fused with another predefined modal image to obtain multiple modal background images corresponding to each size. Understandably, the predefined modal image can be multiple images from different scenes, or images in artistic styles generated by a specific model. By performing modal migration on each size background image respectively, multiple modal background images corresponding to each size are obtained, and a background image set of the original image is finally obtained. Obviously, the background image set includes multiple size, multiple modal background images, and the multiple size, multiple modal background images fuse the styles or features of multiple different scene images, making them show diversity and individuality in vision and semantics, suitable for human target segmentation tasks in different scenes, enhancing the diversity of training data, and helping to improve the generalization ability and robustness of human target segmentation network, and improving the performance and accuracy of the model.

[0094] Referring to Figure 5 as shown, Figure 5 A sub-process flowchart of step S200 of the method for training a human target segmentation model provided by some embodiments of the present application is provided, which extracts the background region in the original image, and obtains a background image set based on the background region, and specifically includes but is not limited to the following steps S210-S240:

[0095] S210: Obtain the human key points of the original image.

[0096] In the human target segmentation task, human body keypoints (Human Body Keypoints), also known as pose estimation (Pose Estimation), refers to marking the key point positions of specific parts of the human body for the human body category target, which is usually used to represent the key parts of the human body, such as the head, eyes, nose, ears, shoulders, elbows, wrists, hips, knees, ankles, etc. In some human target segmentation tasks, specific deep learning models or algorithms, such as pose estimation models, key point detection models, etc., are needed to detect human body keypoints in images. Specifically, pose estimation models, key point detection models, etc. are composed of convolutional neural networks and can automatically detect human body keypoints in images or videos after training. When human body keypoints are detected, the model outputs the position information of each human body keypoint in the form of pixel coordinates. Among them, the position information describes the accurate position of the human body keypoint in the image, such as (x, y) coordinates representing the horizontal and vertical positions of the keypoint on the image. It is easy to understand that when extracting human body keypoints from the original image, different models or algorithms can be selected and adjusted according to actual needs.

[0097] In some embodiments, the OpenPose algorithm can be used to extract human body keypoints from the original image. The OpenPose algorithm is an open-source pose estimation algorithm based on deep learning, which can detect human body keypoints in input original images or videos in real time, including key positions of body parts such as head, arms, and legs. The OpenPose algorithm uses a convolutional neural network to analyze images or videos and outputs the accurate position of each human body keypoint, for example, outputs a series of coordinates (x, y) of human body keypoints to represent the position of human body keypoints on the image. In some embodiments, the position information of 6 human body keypoints in the original image can be extracted by the OpenPose algorithm, which are the left eye, right eye, left hand, right hand, left shoe, and right shoe, and the coordinates of the 6 human body keypoints are output correspondingly. It is easy to understand that the position information of other quantities of human body keypoints in the original image can also be extracted by the OpenPose algorithm, for example, in some other embodiments, the position information of 8 human body keypoints in the original image can be extracted, which are the left eye, right eye, left hand, right hand, left little toe, right little toe, left big toe, and right big toe, and the coordinates of the 8 human body keypoints are output correspondingly. Those skilled in the art can understand that the number of human body keypoints extracted from the original image can be selected and adjusted according to actual needs.

[0098] S220: Based on the human body keypoints, determine a human body rectangular frame, and extract the image outside the human body rectangular frame as a background area.

[0099] After the human body key points in the original image are obtained by a specific deep learning model or algorithm, a human body rectangular frame containing the human subject can be calculated based on the obtained human body key points. The human body target segmentation model separates the human body region and the background region in the original image by taking the image in the human body rectangular frame as the human body region and the image outside the human body rectangular frame as the background region according to the calculated human body rectangular frame. The image outside the human body rectangular frame is extracted as the background region through cropping or masking operation. The background region is a part that is mutually exclusive with the human body region and contains all image contents in the original image except the human body region.

[0100] In some embodiments, after the six human body key points required in the original image are obtained by using the OpenPose algorithm, a human body rectangular frame containing the human subject can be calculated based on the six obtained human body key points. Then, the image in the human body rectangular frame is taken as the human body region and the image outside the human body rectangular frame is taken as the background region according to the calculated human body rectangular frame, and the image outside the human body rectangular frame is cropped as the background region through cropping or masking operation, so as to separate the human body region and the background region in the original image.

[0101] S230: cropping the background region in multiple different sizes to obtain multiple background images with different sizes.

[0102] After the background image outside the human body rectangular frame is extracted as the background region, multiple different sizes are randomly selected for the extracted background region, and the background region is cropped. It is easy to understand that multiple different cropping sizes can be set, such as small size, medium size and large size, or other sizes can also be set to obtain multiple background images with different sizes. In some embodiments, the cropping size can be set to a minimum of 16*16 and a maximum of 64*64, and a total of 100 background images are cropped. By cropping the background region in different sizes, multiple background images with different sizes are obtained, which have different features and spatial distributions, thereby providing more diverse background feature information for the human body target segmentation model, and being beneficial to improve the generalization ability and robustness of the human body target segmentation model.

[0103] S240: performing modal transfer on the background image of each size based on images of multiple different scenes to obtain a set of background images.

[0104] Specifically, the style or feature of each size of the background image is respectively fused with another predefined modal image, a plurality of modal background images corresponding to each size are obtained, and finally a plurality of size and modal background images are obtained, and a background image set of the original image is formed. In the embodiment of the present application, the predefined modal image is a plurality of images from different scenes. Alternatively, in some other embodiments, the predefined modal image can also be an artistic style image generated by a specific model. Based on a plurality of images from different scenes, the background image set including a plurality of size and modal background images is obtained by performing modal migration on the background image of each size, wherein the plurality of size and modal background images fuse the styles or features of the plurality of images from different scenes, so that they show diversity and individuality in vision and semantics, and the diversity of the training data is enhanced to adapt to the human target segmentation task in different scenes.

[0105] In some embodiments, based on a plurality of images from different scenes, the background image set is obtained by performing modal migration on the background image of each size, specifically including but not limited to the following steps S241:

[0106] S241: based on a plurality of images from different scenes, the background image set is obtained by performing modal migration on the background image of each size according to a first formula, and the first formula is:

[0107] M bg ,S bg =F(θ(I bg ))

[0108] M style ,S style =F(θ(I style ))

[0109]

[0110] Wherein, θ() represents converting an image into LAB format, I style represents an image under different scenes, I bg represents a first background image, F() is to extract mean and standard deviation of the LAB format image, M bg ,S bg are the mean and standard deviation of F(θ(I bg )), respectively, M style ,S style are the mean and standard deviation of F(θ(I style )), respectively, and is the background image after modal migration. The idea of the above formula is to make the mean and standard deviation of the pixels in the first background image close to the mean and standard deviation of the images under different scenes.

[0111] In some embodiments, the images in different scenes are collected in advance from different scenes and background images, which are different in shooting location, shooting time, lighting condition, weather, etc., representing diversified background scenes. In some embodiments, the OpenCV image processing library in Python can be used to perform modal migration processing on each size of the background image respectively, to obtain a plurality of modal background images corresponding to each size, and finally obtain background images of multiple sizes and multiple modalities, and form a background image set of the original image. Among them, the OpenCV image processing library includes three different color spaces: RGB, HSV and LAB. It is easy to understand that the LAB color space is a device-independent color model for describing human perception of color, which contains luminance (L) and two chrominance channels (A and B), and can be used to more accurately represent and process color information of images. Among them, the L channel represents luminance, the value range of the L channel is 0-100, and the A and B channels represent chrominance, the value range of the A and B channels is between-128 and 127.

[0112] After obtaining the background image of each size, the background image of each size and a plurality of images from different scenes are input into the OpenCV image processing library, and then the cv2.imread() function in the OpenCV image processing library is used to read the background image of each size and a plurality of images from different scenes. After reading the corresponding images, the cv2.cvtColor() function in the OpenCV image processing library is used to convert the background image of each size and a plurality of images from different scenes from BGR (the default OpenCV color representation) to LAB color representation.

[0113] After converting each size of the background image and the images of multiple different scenes into LAB color representation, in some embodiments, modal migration processing can be performed on each size of the background image by employing color conversion, style conversion, style transfer and the like, so as to fuse the style or feature of each size of the background image with another predefined modal image respectively, to convert the background image into a background image with a predefined image modal, and obtain multiple modal background images corresponding to each size. By performing modal migration processing on each size of the background image respectively, multiple modal background images corresponding to each size are obtained, and finally multiple size and multiple modal background images are obtained, and a background image set of the original image is formed. It can be understood that the multiple size and multiple modal background images fuse the styles or features of the images of multiple different scenes, represent the diversity of the backgrounds of different scenes, make them show diversity and individuality in vision and semantics, enhance the diversity of the training data, are suitable for human target segmentation tasks in different scenes, are conducive to training the human target segmentation model to accurately segment in various background conditions, improve the generalization ability and robustness of the human target segmentation model, and improve the performance and accuracy of target segmentation.

[0114] In some embodiments, based on the images of multiple different scenes, the background image of each size is subjected to modal migration to obtain a background image set, which further includes but is not limited to the following step S242:

[0115] S242: adjusting display parameters of the background image, and performing Gaussian noise blurring processing, wherein the display parameters include brightness, contrast and saturation.

[0116] Adjusting the display parameters of the obtained multiple size and multiple modal background images includes adjusting the brightness, contrast and saturation of the background image. In some embodiments, brightness adjustment can be achieved by linear or nonlinear transformation, such as multiplying a brightness factor or applying Gamma correction, etc. By adjusting the brightness of the background image, the background image can become brighter or dimmer. Adjusting the contrast can usually be achieved by stretching the pixel value range or applying histogram equalization, etc. Contrast adjustment is used to enhance the color difference in the background image, so that the colors in the background image are more vivid. Saturation adjustment is used to increase or decrease the color saturation in the background image. Increasing the saturation can make the colors of the background image more vivid, and decreasing the saturation can make the colors of the background image more soft. By adjusting the display parameters of the multiple size and multiple modal background images, diversified background images can be obtained, which have different characteristics and forms in vision.

[0117] After adjusting the display parameters of the background images of multiple sizes and multiple modalities, in some embodiments, Gaussian noise blur can be applied to each background image. Gaussian noise blur is achieved by applying Gaussian filtering to the image to blur the image, thereby reducing the details and noise in the image, making the image more smooth and realistic. Among them, Gaussian filtering is a linear smoothing filter that uses a Gaussian function to perform weighted averaging on each pixel in the image to reduce the difference between pixels. The introduction of Gaussian noise can simulate random noise in the image, while Gaussian filtering can reduce the impact of these noises on the image, making the image more smooth and blurred. After Gaussian noise blur processing, each background image will become softer, and the details and noise of the image will be reduced, thereby enhancing the visual effect of the image.

[0118] S300: input the background image set into the background encoding network for encoding to obtain a background feature map.

[0119] After extracting the background image set in the original image, the background image set is input into the background encoding network. The background encoding network converts the background image set into a more compact representation of the background feature map through a series of convolution and pooling layers. In some embodiments, the size of the convolution kernel in the convolution layer of the background encoding network is 3*3, the step size is set to 2, the activation function used is Relu, and the number of convolution kernels is 128 and 256 respectively. The specific network structure is convolution layer 128-activation function layer-batch normalization layer-convolution layer 256-activation function layer-batch normalization layer. The background encoding network scales the size of the background feature map converted from the background image set to 16*16*256. It can be understood that the obtained background feature map is an encoded representation of the input background image set, capturing important semantic feature information in the background image set. These feature information is usually a two-dimensional matrix or a multi-dimensional tensor, depending on the structure and design of the background encoding network.

[0120] S400: input the original image into the encoding module for encoding to obtain an original feature map.

[0121] The encoding module is composed of a convolutional neural network (CNN) or its variants, which gradually reduces the spatial dimensions of the original image through a series of convolution and pooling layers, extracts and encodes the features of the input original image, and obtains a series of abstract original feature maps, such as an original feature map with a size of 16*16*512. Obviously, the original feature map is a representation of the original image at a high-level semantic level, which can capture the semantic and structural information in the original image. Each pixel or feature in the original feature map corresponds to the semantic information or abstract feature value of the corresponding position in the original image, which can be regarded as an effective encoding of the original image. In some embodiments, the size of the convolution kernel in the network structure of the encoding module is 3*3, the step is set to 2, the activation function used is Relu, and the number of convolution kernels is 64, 128, 256, and 512, respectively. The specific network structure is: convolution layer 64-activation function layer-batch normalization layer-convolution layer 128-activation function layer-batch normalization layer-convolution layer 256-activation function layer-batch normalization layer-convolution layer 512-activation function layer-batch normalization layer, which has good image feature extraction capability.

[0122] S500: input the background feature map and the original feature map into the feature fusion module for feature fusion to obtain a fused feature map.

[0123] In the human target segmentation task, based on multiple images of different scenes, the modal transfer is performed on the background images of each size respectively, and after obtaining a background image set including background images of multiple sizes and multiple modalities, the background images of multiple modalities of each size are input into the background encoding network for encoding to obtain the background feature map of each size.

[0124] The background feature map and the original feature map of each size are input into the feature fusion module, the weight parameters of the background feature maps of different sizes are adjusted, so that the feature fusion module learns and represents the features under the background images of multiple sizes and multiple modalities, and the background feature map and the original feature map of each size are fused to better adapt to the requirements of the human target segmentation task and learn more high-level feature representations and detailed information. Through multiple iterations of training and adjustment of the weight parameters of the neural network, the feature fusion module learns the background detail information and high-level feature representation of the background images of multiple sizes and multiple modalities. The feature fusion module fuses the background feature map and the original feature map according to the weight parameters of the background feature map of each size, so as to accurately combine the human target feature information and the background feature information together to obtain the fused feature map after feature fusion, and generate a pixel-level segmentation result to classify each pixel in the image as belonging to the background category or the human target category. Understandably, the fused feature map is a key result of the human target segmentation task, which contains high-level feature representations fused with background feature information and human target feature information.

[0125] S600: input the fusion feature map into the decoding module for decoding to obtain a prediction label corresponding to the human target.

[0126] As the depth of the neural network is continuously deepening, the gradient vanishing or explosion may occur. In order to solve this problem, the way of jump cascade connection can be used. The jump cascade connection way connects the convolution features of the bottom layer and the deconvolution features of the upper layer, so that the feature maps with different spatial resolutions can transmit information. The jump connection can strengthen the information transmission of the network, while retaining the low-level detail information. By connecting the up-sampling features and the down-sampling features, the network can better learn the local and global features of the image, improve the accuracy of semantic segmentation, and accelerate the convergence of the human target segmentation model.

[0127] The decoding module is a core component of the human segmentation network. In some embodiments, the size of the convolution kernel of the convolution layer in the network structure of the decoding module is 3*3, the step is set to 1, the activation function used is Relu, and the number of convolution kernels is 256, 128, 64 and 1 respectively. The specific network structure is: convolution layer 256-up sampling-jump cascade connection layer (connected with the output result of the third convolution kernel of the encoding module)-activation function layer-batch normalization layer-convolution layer 128-up sampling-jump cascade connection layer (connected with the output result of the second convolution kernel of the encoding module)-activation function layer-batch normalization layer-convolution layer 64-up sampling-jump cascade connection layer (connected with the output result of the first convolution kernel of the encoding module)-activation function layer-batch normalization layer-convolution layer 1-activation function layer-batch normalization layer.

[0128] After obtaining the fusion feature map, the fusion feature map is input into the decoding module, and through a series of deconvolution and up-sampling operations, the fusion feature map is restored to the size of the original input image, and a pixel-level segmentation mask is output, that is, a prediction label corresponding to the human target in the original image with a size of 256*256*1, each pixel in the original image is classified as background or human target. In some embodiments, it is agreed that the pixel value of 0 represents background and the pixel value of 1 represents human target. Among them, the prediction label is a matrix or tensor with the same size as the true label of the original image, each pixel is classified as belonging to the background or the human target, and the design of the decoding module allows the model to convert the high-level semantic information learned from the feature map into the prediction result of each pixel in the image. The segmentation mask gives the prediction label of each pixel in the image, that is, the prediction label of the human target. By analyzing the segmentation mask, it can be accurately known that each pixel is classified as belonging to the background category or the human target category, and the pixels belonging to the human target are segmented.

[0129] S700: Calculate the loss between the real label and the predicted label based on the loss function, and iteratively train the human target segmentation network according to the loss until the human target segmentation network converges, obtaining the human target segmentation model.

[0130] In the human target segmentation task, the segmentation result predicted by the target segmentation model needs to be compared with the real label, the loss between the predicted label and the real label is measured by a preset loss function, and the background coding network and the human segmentation network (i.e. the human target segmentation network) are iteratively trained by an optimization algorithm according to the loss between the predicted label and the real label, the neural network parameters are constantly adjusted until the human target segmentation network converges, and an effective human target segmentation model is obtained.

[0131] In some embodiments, a cross-entropy loss function or a Dice loss function can be used to calculate the loss between the real label and the predicted label. The cross-entropy loss function measures the difference between the predicted probability distribution and the real label, while the Dice loss function measures the similarity between two sets. It can be understood that the smaller the loss function value calculated, the closer the prediction result of the human target segmentation model to the real label, and the more optimized the training process.

[0132] In other embodiments, the loss function that can be used is:

[0133]

[0134] wherein x i represents the real label of the human target in the original image of the i-th frame, represents the predicted label of the human target in the original image of the i-th frame, L2 represents the average standard deviation loss of the real label and the predicted label, and L1 represents the average absolute difference loss of the real label and the predicted label. Since the standard deviation loss is sensitive to large differences, and the absolute difference loss is sensitive to small changes, in order to ensure the stability of the model convergence result, the model loss is not blindly converged to a certain direction by adding the coefficient a before the L1 loss, thereby ensuring the stability of the model convergence. In this embodiment, preferably, the value of the coefficient a is a constant 12. It can be easily understood that other different loss functions can be used to calculate the loss function value between the real label and the predicted label, and the loss function used can be selected, adjusted and transformed as needed.

[0135] In the model training process, by inputting the original image into the human target segmentation network, the prediction label output by the human target segmentation network is obtained. Then the prediction label is compared with the real label, and the loss function value is calculated by the loss function. It is easy to understand that the optimization algorithm (such as Adam, stochastic gradient descent SGD, etc.) can be used to adjust the related weight parameters in the background coding network and the human segmentation network through the back propagation algorithm to reduce the value of the loss function and optimize the prediction ability of the human target segmentation network. In some embodiments, the Adam optimization algorithm can be used to optimize the weight parameters of the human target segmentation network, wherein the number of iterations is set to 100,000 times, the initial learning rate is set to 0.001, the weight decay is set to 0.0005, the learning rate is attenuated to 1 / 10 of the original every 1000 iterations, and the human target segmentation network converges.

[0136] After multiple iterations of training, the loss function value of the human target segmentation network gradually decreases, and the agreement between the prediction label output by the human target segmentation network and the real label gradually improves until the human target segmentation network converges. The convergence of the human target segmentation network indicates that the human target segmentation model has reached a relatively stable state, and more training will not bring significant improvement and improvement. The human target segmentation network receives the input original image and outputs the pixel-level segmentation result, accurately classifying each pixel in the original image as belonging to the background or the human target.

[0137] The method for training a human target segmentation model provided by the embodiments of the present application, the human target segmentation network includes a background coding network and a human segmentation network, the human segmentation network includes an encoding module, a feature fusion module and a decoding module, the method includes: obtaining a training set, the training set includes multiple original images containing human targets, the original image has a real label corresponding to the human target; extract the background area in the original image, obtain the background image set based on the background area, wherein the background image set is obtained by cropping the background area with multiple different sizes and then performing modal transfer, the modal transfer includes image transfer of multiple different scenes; input the background image set into the background coding network for coding to obtain the background feature map; input the original image into the encoding module for coding to obtain the original feature map; input the background feature map and the original feature map into the feature fusion module for feature fusion to obtain the fusion feature map; input the fusion feature map into the decoding module for decoding to obtain the prediction label corresponding to the human target; calculate the loss between the real label and the prediction label based on the loss function, and iteratively train the background coding network and the human segmentation network according to the loss until the human target segmentation network converges to obtain the human target segmentation model.

[0138] In the embodiment of the present application, when training a human target segmentation model, the background region in the original image is cropped using multiple different sizes, and then the cropped background region is respectively subjected to modality transfer based on images of multiple different scenes to obtain a background image set, wherein the background image set includes background images of multiple sizes and multiple modalities. The background image set is input into a background encoding network for encoding to obtain a background feature map, and the background feature map is used for model training. Thus, the human target segmentation network pays more attention to the feature information and boundary feature changes of the background image during training, and the human target segmentation model obtained thereby can accurately segment the human target, reduce the interference of complex background and the generation of artifacts, reduce the jitter of the human target segmentation boundary, and the segmentation result is clearer and more reliable, improving the stability of segmentation and the user experience.

[0139] Referring to Figure 6 as shown, Figure 6 A sub-process diagram of step S500 of the method for training a human target segmentation model provided by some embodiments of the present application is shown. The background feature map and the original feature map are input into the feature fusion module for feature fusion to obtain a fused feature map, specifically including but not limited to the following steps S510-S540:

[0140] S510: The background feature map and the original feature map are input into the feature fusion module, and the original feature map is subjected to convolution operation to obtain a first original feature map.

[0141] Convolution operation is a commonly used feature extraction method in deep learning. Convolution operation extracts features at different positions on a feature map by sliding a convolution kernel (also known as a filter), which can further enhance the expression ability of the feature map and extract higher-level feature information. Each size of the background feature map and the original feature map is input into the feature fusion module, and the feature fusion module performs convolution operation on the original feature map to obtain a first original feature map with a corresponding size of 16*16*512. In some embodiments, the size of the convolution kernel in the convolution layer of the network structure of the feature fusion module is 1*1, the step size is set to 1, the activation function used is Relu, and the number of convolution kernels is 512.

[0142] S520: The background feature map is subjected to dilated convolution operation to obtain a first background feature map.

[0143] Dilated convolution is a special convolution operation. The convolution kernel of the dilated convolution layer inserts several holes (also known as dilation rates) in the middle of the original convolution kernel to expand the receptive field of the convolution kernel and increase the perception range of the feature map, so as to capture more extensive context information of the background region, better understand the global features of the background region, and improve the accuracy and robustness of target segmentation.

[0144] The dilated convolution layer in the feature fusion module performs dilated convolution operation on each size of the background feature map to obtain each size of the first background feature map after dilated convolution processing, which contains the feature information of each size of the background feature map after dilated convolution. In some embodiments, the size of the convolution kernel of the dilated convolution layer in the network structure of the feature fusion module is 3*3, the convolution dilation rate is set to 4, the step is set to 1, the activation function used is Relu, and the number of convolution kernels is 512.

[0145] S530: performing residual connection on the first original feature map and the first background feature map to obtain a first fusion feature map.

[0146] Residual Connection refers to a technique of introducing direct or shortcut connections in neural networks. In traditional neural networks, information is transmitted through linear transformations between layers and nonlinear activation functions. However, when the network becomes very deep, there is a problem of gradient vanishing or gradient explosion, making it difficult to train deep networks, and the introduction of skip connections aims to solve this problem. Through skip connections, the network can more easily learn the identity mapping, i.e., directly passing the input information to the output, thereby making the training of the network more stable and efficient.

[0147] Residual Connection adds the input data directly to the output of the network, specifically, assuming the input is x, and the output H(x) is obtained after a series of network layer transformations and activations, then the role of residual connection is to add the input x and the output H(x) to obtain the final output F(x) = x + H(x). In this way, the network can directly learn the residual, rather than relying entirely on inter-layer transformations. Residual Connection allows the model to directly learn the residual, i.e., the difference between the first original feature map and the first background feature map, so that the feature information can be more easily transmitted at different levels. Through residual connection, the original feature and the background feature can be effectively fused, enhancing the feature representation capability of the human target segmentation model and improving the accuracy and robustness of human target segmentation.

[0148] In some embodiments, after obtaining the first original feature map and each size of the first background feature map in the feature fusion module, the first original feature map and each size of the first background feature map are connected by residual connection. By adding the input after the original convolution operation (i.e., the first original feature map) and the residual after the dilated convolution operation (i.e., each size of the first background feature map), information transmission is achieved, making information transmission in the network smoother, effectively combining background feature information and human target feature information to obtain a first fusion feature map with a size of 16*16*512 that has more rich and accurate feature representation.

[0149] S540: connecting the first fusion feature map and the original feature map to obtain the fusion feature map.

[0150] In some embodiments, the first fusion feature map and the original feature map can be feature spliced or channel-level connected in the feature fusion module to fuse the first fusion feature map and the original feature map into a more rich and comprehensive fusion feature map. Wherein, the feature splicing refers to splicing the first fusion feature map and the original feature map in the depth (channel number) direction to obtain a corresponding fusion feature map with increased channel number. The channel-level connection refers to connecting the corresponding channels of the first fusion feature map and the original feature map to obtain a fusion feature map with a size of 16*16*1024, and each channel is the superposition of the corresponding channels of the first fusion feature map and the original feature map. The fusion feature map obtained by connecting the first fusion feature map and the original feature map contains the background information and the human target information in the first fusion feature map and the detail features of the original image captured by the original feature map, has a more rich and comprehensive feature representation, and improves the performance and accuracy of the human target segmentation model.

[0151] In summary, the method for training a human target segmentation model provided by the embodiment of the present application, the human target segmentation network includes a background encoding network and a human segmentation network, the human segmentation network includes an encoding module, a feature fusion module and a decoding module, the method includes: obtaining a training set, the training set includes a plurality of original images containing human targets, the original images have real labels corresponding to the human targets; extracting a background region in the original image, obtaining a background image set based on the background region, wherein the background image set is obtained by cropping the background region with a plurality of different sizes and then performing modal transfer, and the modal transfer includes image transfer of a plurality of different scenes; inputting the background image set into the background encoding network for encoding to obtain a background feature map; inputting the original image into the encoding module for encoding to obtain an original feature map; inputting the background feature map and the original feature map into the feature fusion module for feature fusion to obtain a fusion feature map; inputting the fusion feature map into the decoding module for decoding to obtain a predicted label corresponding to the human target; calculating the loss between the real label and the predicted label based on a loss function, and iteratively training the background encoding network and the human segmentation network according to the loss until the human target segmentation network converges to obtain a human target segmentation model.

[0152] In the embodiment of the present application, when training a human target segmentation model, the background region in the original image is cropped using multiple different sizes, and then the cropped background region is subjected to modality transfer based on multiple different scene images to obtain a background image set, wherein the background image set includes multiple size and multiple modality background images. The background image set is input into a background encoding network for encoding to obtain a background feature map, and the background feature map is used for model training. Thus, the human target segmentation network pays more attention to the feature information and boundary feature changes of the background image during training, and the human target segmentation model obtained thereby can accurately segment the human target, reduce the interference of complex background and the generation of artifacts, reduce the jitter of the human target segmentation boundary, and the segmentation result is clearer and more reliable, improving the stability of the segmentation and enhancing the user experience.

[0153] Referring to Figure 7 as shown, Figure 7 is a flowchart of a human target segmentation method according to some embodiments of the present application. Obviously, the execution subject of the human target segmentation method can be the electronic device described above, and the human target segmentation method includes but is not limited to the following steps S10A-S30A:

[0154] S10A: Obtain a to-be-processed image.

[0155] When applying a human target segmentation model to perform human target segmentation on a to-be-processed image, the to-be-processed image needs to be obtained first, which can be obtained by a user or an operator from various data sources for image processing or computer vision tasks. It can be understood that the to-be-processed image can be pixel data represented in digital form, or an image file in formats such as JPEG, PNG, etc., or a real-time image obtained through an image acquisition device such as a camera, a scanner, etc. It is easy to understand that in some embodiments, the to-be-processed image can also be obtained in other ways, for example, a user can obtain image data uploaded by the user himself through an application program, such as an avatar, a photo album, etc. in a social media application. Obtaining the to-be-processed image provides a reliable data source for subsequent image data preprocessing, recognition detection, feature extraction, and human target segmentation operations.

[0156] S20A: input the to-be-processed image into the human target segmentation model to obtain a predicted label of a human target in the to-be-processed image, wherein the human target segmentation model is trained using any of the above methods for training a human target segmentation model.

[0157] The obtained to-be-processed image is loaded as input data into the human target segmentation model. The to-be-processed image is subjected to forward propagation of the human target segmentation model, and the model classifies each pixel in the to-be-processed image as a human body or a background to obtain a predicted label of a human target in the to-be-processed image. The human target segmentation model is obtained by the method for training a human target segmentation model provided in any one of the above embodiments, has the same structure and function as the human target segmentation model in the above embodiments, and will not be described again here. In the forward propagation process, each pixel in the to-be-processed image is input into the human target segmentation model and subjected to a series of convolution, pooling, feature fusion and other operations to extract feature information from the input to-be-processed image. The human target segmentation model outputs a corresponding predicted label for each pixel in the to-be-processed image based on the feature representation and weight parameters learned in the training stage, classifies each pixel as a human target category or a background category, and then obtains a predicted label image of the same size as the to-be-processed image. The obtained predicted label image is the predicted label of the human target in the to-be-processed image. It can be understood that each pixel in the predicted label image is assigned a category label, which indicates whether the pixel belongs to a human target or a background category. In some embodiments, after obtaining the predicted label image, a post-processing operation needs to be performed, such as removing noise in the predicted label image, filling a target area, and the like, to obtain a more accurate and complete human target segmentation result.

[0158] S30A: segmenting a human target image from the to-be-processed image according to the predicted label of the human target.

[0159] After obtaining the predicted label of the human target in the to-be-processed image, the human target pixels in the to-be-processed image are extracted from the background through a pixel-level operation according to the predicted label of the human target, and a separate human target image is obtained. The human target image only contains the human target part in the to-be-processed image, and the background part is removed. In some embodiments, the human target image can be simply obtained by traversing the pixels in the to-be-processed image, selecting the human target pixels according to the predicted label of the human target, or the human target image can also be generated through an image mask operation.

[0160] The human body target segmentation method provided by the embodiments of the present application comprises: obtaining a to-be-processed image; inputting the to-be-processed image into a human body target segmentation model to obtain a predicted label of a human body target in the to-be-processed image, wherein the target segmentation model is obtained by training the human body target segmentation model by using any one of the methods for training a human body target segmentation model described above; and segmenting a human body target image from the to-be-processed image according to the predicted label of the human body target. The human body target segmentation model obtained by training can accurately segment the human body target, reduce the interference of a complex background and the generation of artifacts, reduce the jitter of the human body target segmentation boundary, make the segmentation result clearer and more reliable, improve the stability of segmentation, and improve the experience of a user.

[0161] The computer readable storage medium provided by the embodiments of the present application stores computer program instructions, and a processor executes the computer program instructions to perform any one of the methods for training a human body target segmentation model provided by the embodiments of the present application, or any one of the human body target segmentation methods provided by the embodiments of the present application.

[0162] In some embodiments, the storage medium can be a flash memory, a hard disk, an optical disc, a register, a magnetic surface memory, a removable disk, a CD-ROM, a random access memory (RAM), a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, or any other form of storage medium known in the art, or various devices comprising one or any combination of the above storage mediums.

[0163] In some embodiments, the computer program instructions can be in the form of a program, software, software module, script or code, written in any form of programming language (including a compiled or interpreted language, or a declarative or procedural language), and can be deployed in any form, including being deployed as a standalone program or being deployed as a module, component, subroutine or other unit suitable for use in a computing environment.

[0164] As an example, the computer program instructions can but not necessarily correspond to a file in a file system, can be stored in a part of a file storing other programs or data, for example, stored in one or more scripts of a Hyper Text Markup Language (HTML) document, or stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subprograms or code portions).

[0165] By way of example, computer program instructions can be deployed to be executed on one computer (e.g., directly installed on integral computer of the device, including smart terminal and server), or on multiple computers (e.g., distributed on multiple computers located in one place or interconnected through a communication network). It is easily understood that all or part of the steps of the method described in the above embodiments of the present application can be implemented by using electronic hardware or computer program instructions executable by a processor, or a combination of both.

[0166] It is understood by the skilled in the art that the embodiments provided by the present application are only illustrative, and the writing order of the steps in the method of the embodiments does not mean a strict execution order and constitute any limitation on the implementation process. The order can be adjusted, combined and deleted according to actual needs. The modules or sub-modules, units or sub-units in the device or system of the embodiments can be combined, divided and deleted according to actual needs. For example, the division of the units is only a logical function division, and another division mode can be used in actual implementation. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0167] It should be noted that the above embodiments are only for illustrating the technical concepts and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and implement it accordingly, and cannot limit the protection scope of the present application. Those skilled in the art can understand that the implementation of all or part of the above-mentioned embodiments is based on the equivalent changes and modifications of the claims of the present application, which should belong to the scope of the claims of the present application.

Claims

1. A method for training a human object segmentation model, the method comprising: The human target segmentation network includes a background encoding network and a human segmentation network. The human segmentation network includes an encoding module, a feature fusion module and a decoding module. The method includes: Acquire a training set, wherein the training set includes a plurality of original images containing human targets, and the original images have true labels corresponding to the human targets; Extracting a background region from the original image and obtaining a background image set based on the background region, wherein the background image set is obtained by cropping the background region using multiple different sizes and then performing modal migration, wherein the modal migration includes migrating images of multiple different scenes; Inputting the background image set into the background encoding network for encoding to obtain a background feature map; Inputting the original image into the encoding module for encoding to obtain an original feature map; Inputting the background feature map and the original feature map into the feature fusion module for feature fusion to obtain a fused feature map; Inputting the fused feature map into the decoding module for decoding to obtain a predicted label corresponding to the human target; The loss between the true label and the predicted label is calculated based on a loss function, and the human target segmentation network is iteratively trained according to the loss until the human target segmentation network converges to obtain the human target segmentation model.

2. The method of claim 1, wherein, The extracting of the background area in the original image and obtaining a background image set based on the background area includes: Obtaining key points of a human body in the original image; Based on the human body key points, a human body rectangular frame is determined, and an image outside the human body rectangular frame is extracted as a background area; The background area is cropped using a plurality of different sizes to obtain a plurality of background images of different sizes; Based on images of multiple scenes, modality migration is performed on the background image of each size to obtain a background image set, where the background image set includes background images of multiple sizes and multiple modalities.

3. The method of claim 2, wherein, The step of performing modal migration on the background image of each size based on the images of multiple different scenes to obtain a background image set includes: Based on images of multiple different scenes, modal migration is performed on the background images of each size according to a first formula to obtain a background image set. The first formula is: M bg ,S bg =F(θ(I bg )) M style ,S style = F(θ(I style )) where θ() denotes converting the image to LAB format, I style denotes the image under different scenarios, I bg denotes the background image, F() is the mean and standard deviation extracted from the image in LAB format, M bg bg are the mean and standard deviation of F(θ(I bg )), respectively, style style are the mean and standard deviation of F(θ(I style )), respectively, is the background image after modal migration.​​ 4. The method of claim 3, wherein, The method of performing modal migration on the background image of each size based on the images of multiple different scenes to obtain a background image set further includes: The display parameters of the background image are adjusted and Gaussian noise blur processing is performed, wherein the display parameters include brightness, contrast and saturation.

5. The method of claim 1, wherein, The step of inputting the background feature map and the original feature map into the feature fusion module for feature fusion to obtain a fused feature map includes: Inputting the background feature map and the original feature map into the feature fusion module, performing a convolution operation on the original feature map to obtain a first original feature map; Performing a dilated convolution operation on the background feature map to obtain a first background feature map; Performing a residual connection on the first original feature map and the first background feature map to obtain a first fused feature map; The first fused feature map and the original feature map are connected to obtain the fused feature map.

6. The method of claim 1, wherein, The loss function is: wherein x i represents a real label of a human target in the original image of the i-th frame, represents a predicted label of a human target in the original image of the i-th frame, L2 represents a mean standard deviation loss of the real label and the predicted label, L1 represents a mean absolute difference loss of the real label and the predicted label, and a is a constant.

7. A human object segmentation method, characterized by, The method comprises: obtaining a to-be-processed image; inputting the to-be-processed image into a human target segmentation model to obtain a predicted label of a human target in the to-be-processed image, wherein the human target segmentation model is trained by using the method for training a human target segmentation model according to any one of claims 1-6; segmenting a human target image from the to-be-processed image according to the predicted label of the human target.

8. An electronic device, comprising: The method comprises: a processor and a memory connected to the processor in communication; the memory stores computer program instructions executable by the processor, and the computer program instructions, when invoked by the processor, cause the processor to execute the method for training a human target segmentation model according to any one of claims 1-6 or the human target segmentation method according to claim 7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions, and the computer program instructions are adapted to be loaded by a processor to execute the method for training a human target segmentation model according to any one of claims 1-6 or the human target segmentation method according to claim 7.

Images