Multi-task image segmentation method and device, equipment, storage medium and product

Abstract

The embodiment of the invention discloses a multi-task image segmentation method and device, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, different segmentation heads share the image feature information output by the backbone network, the foreground and background segmentation head is utilized to analyze and process the image feature information to obtain the foreground and background segmentation result, and the semantic segmentation head is utilized to analyze and process the image feature information to obtain the semantic segmentation result. The face segmentation head is used for analyzing and processing the image feature information to obtain the face image probability, the frequency of repeatedly extracting the image feature information of the to-be-processed image by different segmentation heads is reduced, the face segmentation head does not need to output a face detection frame, the face image probability is output, and the face segmentation head is more accurate. And a face detection frame of the to-be-processed image is generated according to the semantic segmentation result and the face image probability, so that the complex post-processing process of the face segmentation head is reduced, the time consumed by image segmentation reasoning is effectively shortened, and the image segmentation efficiency is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, and in particular to a multi-task image segmentation method, device, equipment, storage medium and product. BACKGROUND

[0002] In the field of live streaming and short videos, users' demand for content diversity and high quality is constantly rising. In order to enhance user experience and improve the attractiveness and interactivity of content, accurate and efficient image segmentation technology is crucial.

[0003] Currently, image segmentation schemes generally use separate image segmentation models for processing, that is, a foreground-background segmentation model, a semantic segmentation model and a face segmentation model are used to process an original image respectively, and foreground-background segmentation results, semantic segmentation results and face detection boxes are output respectively. The foreground-background segmentation model, the semantic segmentation model and the face segmentation model are independent in processing the original image. Currently, deep learning-based image segmentation models generally use complex network structures, such as stacking a large number of convolution layers to extract rich features and using complex feature fusion strategies to improve image segmentation accuracy. However, this approach makes the parameter quantity of the image segmentation model large, the image segmentation inference time-consuming and the image segmentation efficiency low. SUMMARY

[0004] Embodiments of the present application provide a multi-task image segmentation method, device, equipment, storage medium and product, which reduce the number of times of repeatedly extracting image feature information of a to-be-processed image by different segmentation heads by sharing image feature information output by a backbone network, generate a face detection box of the to-be-processed image according to a semantic segmentation result and a face image probability by outputting the face image probability, and reduce the complex post-processing process of a face segmentation head, thereby solving the technical problem of long image segmentation inference time and low image segmentation efficiency in related technologies, effectively shortening the image segmentation inference time and improving the image segmentation efficiency.

[0005] In a first aspect, embodiments of the present application provide a multi-task image segmentation method, comprising: obtaining a to-be-processed image; inputting the to-be-processed image into a trained multi-task image segmentation model, using a backbone network to extract image feature information of the to-be-processed image by the multi-task image segmentation model, using a foreground-background segmentation head to analyze and process the image feature information to obtain a foreground-background segmentation result, using a semantic segmentation head to analyze and process the image feature information to obtain a semantic segmentation result, and using a face segmentation head to analyze and process the image feature information to obtain a face image probability of the to-be-processed image belonging to a face image; generating a face detection box of the to-be-processed image according to the semantic segmentation result and the face image probability.

[0006] In a second aspect, the embodiments of the present application provide a multi-task image segmentation device, comprising an image acquisition module, an image segmentation module and a face detection module, wherein: The image acquisition module is configured to acquire a to-be-processed image. The image segmentation module is configured to input the to-be-processed image into a trained multi-task image segmentation model, use a backbone network to extract features of the to-be-processed image to obtain image feature information, analyze and process the image feature information using a foreground-background segmentation head to obtain a foreground-background segmentation result, analyze and process the image feature information using a semantic segmentation head to obtain a semantic segmentation result, and analyze and process the image feature information using a face segmentation head to obtain a face image probability that the to-be-processed image belongs to a face image. The face detection module is configured to generate a face detection frame of the to-be-processed image according to the semantic segmentation result and the face image probability.

[0007] In a third aspect, the embodiments of the present application provide a multi-task image segmentation device, comprising a memory and one or more processors. The memory is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-task image segmentation method of the first aspect.

[0008] In a fourth aspect, the embodiments of the present application provide a non-volatile storage medium storing computer executable instructions, which, when executed by a computer processor, are used to execute the multi-task image segmentation method of the first aspect.

[0009] In a fifth aspect, the embodiments of the present application provide a computer program product, which comprises a computer program stored in a computer readable storage medium, and at least one processor of a device reads and executes the computer program from the computer readable storage medium, so that the device executes the multi-task image segmentation method of the first aspect.

[0010] The embodiment of the present application inputs the to-be-processed image into the trained multi-task image segmentation model, uses the backbone network to extract features of the to-be-processed image to obtain image feature information, uses the foreground and background segmentation head to analyze and process the image feature information to obtain a foreground and background segmentation result, uses the semantic segmentation head to analyze and process the image feature information to obtain a semantic segmentation result, uses the face segmentation head to analyze and process the image feature information to obtain a face image probability that the to-be-processed image belongs to a face image, and generates a face detection frame of the to-be-processed image according to the semantic segmentation result and the face image probability. Different segmentation heads can share the image feature information output by the backbone network, the number of times of repeatedly extracting image feature information of the to-be-processed image by different segmentation heads is reduced, and the face segmentation head does not need to output a face detection frame but outputs a face image probability, and then generates a face detection frame of the to-be-processed image according to the semantic segmentation result and the face image probability, thereby reducing the complex post-processing process of the face segmentation head, effectively shortening the image segmentation inference time, and improving the image segmentation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a flowchart of a multi-task image segmentation method provided by the embodiment of the present application; Figure 2 is a flowchart of another multi-task image segmentation method provided by the embodiment of the present application; Figure 3 is a segmentation head structure diagram provided by the embodiment of the present application; Figure 4 is a structure diagram of a multi-task image segmentation device provided by the embodiment of the present application; Figure 5 is a structure diagram of a multi-task image segmentation device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0012] In order to make the objectives, technical solutions and advantages of the present application clearer, the following further describes the specific embodiments of the present application in conjunction with the drawings. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only parts related to the present application are shown in the drawings, but not all contents. Before discussing the example embodiments in more detail, it should be mentioned that some example embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The above processes can be terminated when the operations are completed, but can also have additional steps not included in the drawings. The above processes can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0013] The multi-task image segmentation method provided by the application can be applied to the image segmentation scene of live broadcast and short video, aims to make different segmentation heads share the image feature information output by the backbone network, reduce the number of times of repeatedly extracting the image feature information of the image to be processed by different segmentation heads, and output the face image probability instead of the face detection frame by the face segmentation head, and then generate the face detection frame of the image to be processed according to the semantic segmentation result and the face image probability, reduce the complex post-processing process of the face segmentation head, effectively shorten the image segmentation inference time, and improve the image segmentation efficiency.

[0014] Multi-task segmentation generally completes foreground and background segmentation, semantic segmentation and face detection frame output through a deep learning-based segmentation model, which has great value for many applications of live broadcast and short video. For example, in the live broadcast scene, the anchor end can realize virtual background replacement by means of foreground and background segmentation, and enhance the interestingness of the live broadcast picture; semantic segmentation can be used for intelligent recognition of scene elements, realize color enhancement of different colors and advertisement placement; and the face detection frame is helpful for accurate application of real-time beautification and makeup special effects. However, the live broadcast and short video applications are usually run on mobile terminal devices, and the hardware resources such as computing power, memory size and battery endurance of such devices are relatively limited. At present, the multi-task segmentation model based on deep learning adopts a complex network structure, such as using a large number of convolution layers to extract rich features and using a complex feature fusion strategy to improve the segmentation accuracy. However, this approach makes the model parameter quantity large and the calculation complexity increases sharply. When running on a mobile terminal, the model inference time is long, which is difficult to meet the real-time requirement. For example, in the multi-target segmentation scene of real-time video stream, there will be obvious lag and delay. At the same time, the large parameter model occupies too much memory, which not only makes the deployment difficult on the mobile terminal device with limited memory, but also may cause the device to run slowly or even crash. Therefore, the multi-task image segmentation method provided by the embodiments of the application is provided to solve the technical problems of long image segmentation inference time and low image segmentation efficiency of the prior art.

[0015] Figure 1 A flowchart of the multi-task image segmentation method provided by the embodiments of the application is given, and the multi-task image segmentation method provided by the embodiments of the application can be executed by a multi-task image segmentation device. The multi-task image segmentation device can be realized by hardware and / or software, and integrated in a multi-task image segmentation equipment.

[0016] The following describes the multi-task image segmentation method executed by the multi-task image segmentation device. Referring to Figure 1 , the multi-task image segmentation method comprises the following steps. S110: acquiring an image to be processed.

[0017] S120: input the to-be-processed image into the trained multi-task image segmentation model, perform feature extraction on the to-be-processed image by the backbone network to obtain image feature information, analyze and process the image feature information by the foreground-background segmentation head to obtain a foreground-background segmentation result, analyze and process the image feature information by the semantic segmentation head to obtain a semantic segmentation result, and analyze and process the image feature information by the face segmentation head to obtain a face image probability of the to-be-processed image belonging to a face image.

[0018] Exemplarily, a to-be-processed image requiring multi-task image segmentation is acquired, where the to-be-processed image can be a video frame in a video stream acquired in a video live broadcast scene, or a video frame in a to-be-edited video in a short video editing scene.

[0019] The multi-task image segmentation model provided in the application includes a backbone network, a foreground-background segmentation head, a semantic segmentation head, and a face segmentation head. The foreground-background segmentation head, the semantic segmentation head, and the face segmentation head are all connected with the backbone network, the backbone network is configured to perform feature extraction on an input image and output image feature information, the foreground-background segmentation head is configured to perform foreground-background analysis and processing on the image feature information to obtain a foreground-background segmentation result, the semantic segmentation head is configured to perform semantic analysis and processing on the image feature information to obtain a semantic segmentation result, and the face segmentation head is configured to perform analysis and processing on the image feature information to obtain a face image probability.

[0020] It needs to be explained that in multi-task image segmentation, the feature extraction stages of foreground-background segmentation, semantic segmentation, and face segmentation on an image have commonality, but the respective task targets are different. The application uses one backbone network shared by the foreground-background segmentation head, the semantic segmentation head, and the face segmentation head to realize parameter sharing of the backbone network and provide the same image feature extraction for each task. Meanwhile, the foreground-background segmentation, the semantic segmentation, and the face detection are respectively equipped with independent segmentation heads, which can perform targeted processing on the image feature information output by the backbone network according to the specific requirements of different tasks, thereby accurately realizing the target of the multi-task image segmentation model in each task.

[0021] In one embodiment, after the to-be-processed image is acquired, the to-be-processed image is input into the trained multi-task image segmentation model, so as to perform feature extraction on the to-be-processed image by the backbone network to obtain image feature information, analyze and process the image feature information by the foreground-background segmentation head to obtain a foreground-background segmentation result, analyze and process the image feature information by the semantic segmentation head to obtain a semantic segmentation result, and analyze and process the image feature information by the face segmentation head to obtain a face image probability of the to-be-processed image belonging to a face image. The face image can be understood as an image with a face.

[0022] The foreground-background segmentation can distinguish the foreground object from the background in an image or a video, for example, according to the differences in color, texture, position, etc. between the object and the background, and outline the approximate contour of the foreground object. The output can be a float value representing the probability of each pixel in the image belonging to the foreground. This output in the form of a float value gives the foreground-background segmentation result a unique "soft boundary" feature. Unlike traditional hard boundary segmentation, "soft boundary" means that the segmentation result presents a feathering effect. For example, when processing the segmentation of a person and the background, the edge of the person is no longer a harsh boundary, but there is a transition area, and the foreground probability value of the pixel gradually decreases from the center of the person to the edge, making the edge look more natural and soft. This feature is particularly advantageous in image synthesis, video special effect production, etc., and can make the synthesized or processed image or video look more realistic and smooth.

[0023] Semantic segmentation is widely used and detailed, and can be used to classify each pixel in an image and identify the semantic category to which each pixel belongs. The output of semantic segmentation can be an Int value, and each Int value corresponds to a specific segmentation category. For example, the semantic segmentation result can be used to reflect the categories (such as human-related categories such as face, skin, hair, and clothes, and various items frequently appearing in live streaming, such as musical instruments, microphones, headsets, and chairs) corresponding to each pixel in the image to be processed. Through semantic segmentation, each pixel in the image is accurately divided into the corresponding category, forming a labeled image containing rich semantic information.

[0024] Face detection is a relatively clear and focused task, i.e., locating the position of a face in an image or a video and marking the bounding box of the face. Face detection is widely used in security monitoring, face recognition access control systems, and photo beautification scenarios, and is a basic step for applications related to faces. Face detection generally uses features of a face, such as the relative positions of facial features and facial contour features, to accurately find the area where the face is located in a complex scene. Traditional face detection models output the number of faces and the four coordinate positions of the face box. However, existing face detection methods, whether traditional feature detectors or deep learning-based models that directly output face box coordinates, rely on complex non-maximum suppression algorithms, face box filtering, and other post-processing procedures, which undoubtedly increases the computational load and processing time of face detection.

[0025] The present application utilizes the commonality of foreground and background segmentation, semantic segmentation and face detection tasks in feature extraction, reduces repeated calculation of feature extraction, and improves computational efficiency. Moreover, each task branch can realize deep cooperation on the basis of image feature information output by the shared backbone network. For example, foreground and background segmentation can assist semantic segmentation of the foreground region, improving segmentation accuracy and efficiency, and the semantic segmentation result can provide semantic clues for face detection, helping to exclude similar object interference. Through the cooperative work of each branch, the efficiency and accuracy of multi-task processing are effectively improved.

[0026] S130: generating a face detection frame of the to-be-processed image according to the semantic segmentation result and the face image probability.

[0027] Illustratively, after obtaining the foreground and background segmentation result, the semantic segmentation result and the face image probability of the to-be-processed image, a face detection frame of the to-be-processed image can be generated according to the semantic segmentation result and the face image probability. For example, when the face image probability indicates that the to-be-processed image is an image with a face, a face detection frame of the to-be-processed image is generated using the face semantic result in the semantic segmentation result (for example, the pixel points in the to-be-processed image determined by the semantic segmentation result as corresponding to a face). Optionally, when there is one or more faces in the to-be-processed image, one or more face detection frames are correspondingly generated. Optionally, the face detection frame can be a frame surrounding or circumscribing a plurality of face pixel point sets, and the face detection frame can be represented by four corner coordinate points on the to-be-processed image.

[0028] As described above, by inputting the to-be-processed image into the trained multi-task image segmentation model, the backbone network is used to extract image features of the to-be-processed image to obtain image feature information, the foreground and background segmentation head is used to analyze and process the image feature information to obtain the foreground and background segmentation result, the semantic segmentation head is used to analyze and process the image feature information to obtain the semantic segmentation result, the face segmentation head is used to analyze and process the image feature information to obtain the face image probability of the to-be-processed image belonging to a face image, and a face detection frame of the to-be-processed image is generated according to the semantic segmentation result and the face image probability. Different segmentation heads can share the image feature information output by the backbone network, reducing the number of times of repeated extraction of image feature information of the to-be-processed image by different segmentation heads, and the face segmentation head does not need to output a face detection frame, but outputs a face image probability, and then generates a face detection frame of the to-be-processed image according to the semantic segmentation result and the face image probability, reducing the complex post-processing process of the face segmentation head, effectively shortening the image segmentation inference time, and improving the image segmentation efficiency.

[0029] On the basis of the above embodiments, Figure 2 A flowchart of another multi-task image segmentation method provided by an embodiment of the present application is given, which is a specific embodiment of the above multi-task image segmentation method. Reference is made toFigure 2 , the multi-task image segmentation method includes: S210: Acquire an image to be processed.

[0030] S220: Input the image to be processed into the trained multi-task image segmentation model, use the backbone network to extract features of the image to be processed through the multi-task image segmentation model to obtain image feature information, use the foreground and background segmentation head to analyze and process the image feature information to obtain the foreground and background segmentation result, use the semantic segmentation head to analyze and process the image feature information to obtain the semantic segmentation result, and use the face segmentation head to analyze and process the image feature information to obtain the face image probability that the image to be processed belongs to a face image.

[0031] like Figure 3 As shown in the schematic diagram of a split head structure provided, Figure 3 The foreground and background segmentation head, semantic segmentation head, and face segmentation head have the same structure. The foreground and background segmentation head, semantic segmentation head, and face segmentation head provided in this application include a preset number of first-type convolutional layers (the front-stage CONV in the figure, such as a convolutional layer with a convolution kernel size of 3×3), a batch normalization layer (BN, Batch Normalization in the figure), and the first-type activation function (RELU, i.e., rectified linear unit in the figure).

[0032] Each layer's convolutional layer, batch normalization layer, and activation function are connected in sequence. The first convolutional layer receives image feature information output by the backbone network, and the last activation function outputs feature map information. The convolutional layer is used to extract deep features from the feature map information or the output of the activation function of the previous layer to obtain deep feature information. The batch normalization layer and activation function are used to normalize the deep feature information and introduce nonlinear transformations to obtain feature map information. This can effectively accelerate the model convergence process and significantly improve the model's stability during training. The activation function can also introduce nonlinear factors into the model, breaking the limitations of linear models and greatly enhancing the model's ability to express complex data features.

[0033] The foreground-background segmentation head, the semantic segmentation head and the face segmentation head further comprise a squeeze-excitation module (SE), a plurality of up-sampling layers (up in the figure) and a second convolutional layer (CONV in the figure), the squeeze-excitation module comprising a global average pooling layer, two fully connected layers (fc in the figure) and a second activation function (sigmoid in the figure). In the squeeze-excitation module, the global average pooling layer performs a global average pooling operation on the feature map information output by the last layer of activation function, compresses the two-dimensional feature map information into a one-dimensional channel description vector, so that the information of each channel can be integrated and highlighted; the first fully connected layer performs channel compression processing on the global average pooling operation result to obtain a channel compression result, for example, compresses the channel number of the one-dimensional channel description vector to 1 / 16 of the original number, in this process, the model can automatically learn more compact and more representative feature relationships between channels, and the compressed vector further passes through a ReLU activation function to further enhance its non-linear expression ability; the second fully connected layer restores the channel compression result to the original channel number to obtain a channel restoration result, so that the model can retain key information while adapting to subsequent processing requirements; the second activation function analyzes and processes the channel restoration result to obtain channel weight coefficients of the feature map information, which reflect the importance of different channel features. The segmentation head can calibrate the feature map information according to the channel weight coefficients to obtain a calibration processing result (for example, the channel weight coefficients are multiplied by the first feature map information) to highlight important channel features and suppress redundant information, and perform up-sampling and convolution processing on the calibration processing result to obtain a probability feature map of the corresponding target. The up-sampling operation on the calibration processing result can restore the feature map to the same size as the input image. Alternatively, bilinear interpolation can be used for up-sampling, which can expand the size of the feature map while preserving as much detail information of the image as possible through linear interpolation of adjacent pixels. Alternatively, a convolutional layer with a size of 1x1 can be used to perform convolution processing on the up-sampling operation to adjust the channel number of the feature map to the number of classes required by the segmentation task, thereby outputting a segmentation result.

[0034] The foreground-background segmentation head, the semantic segmentation head and the face segmentation head differ in that the foreground-background segmentation head is further configured with a normalization layer after the second convolutional layer, which can be used to normalize the probability feature map to obtain a foreground-background segmentation result; the semantic segmentation head is further configured with a maximum value layer after the second convolutional layer, which can be used to perform maximum probability processing on the probability feature maps of a plurality of segmentation classes to obtain a semantic segmentation result; and the face segmentation head is further configured with a pooling layer after the second convolutional layer, which can be used to compress the probability feature map to obtain a face image probability.

[0035] In one possible implementation, the multi-task image segmentation method provided in the application uses a foreground-background segmentation head to analyze and process image feature information to obtain a foreground-background segmentation result, including: performing deep feature extraction on the image feature information through a preset number of first convolutional layers to obtain deep feature information, and performing normalization processing and introducing a nonlinear transformation on the deep feature information through a first batch normalization layer and a first activation function to obtain first feature map information; performing a global average pooling operation on the first feature map information, performing channel compression processing on the global average pooling operation result through a first fully connected layer to obtain a channel compression result, and restoring the channel compression result to an original channel number through a second fully connected layer to obtain a channel restoration result, and performing analysis and processing on the channel restoration result through a second activation function to obtain first channel weight coefficients of the first feature map information; performing calibration processing on the first feature map information according to the first channel weight coefficients to obtain a first calibration processing result, and performing up-sampling and convolution processing on the first calibration processing result to obtain a foreground probability feature map; performing normalization processing on the foreground probability feature map through a normalization layer to obtain the foreground-background segmentation result.

[0036] The foreground-background segmentation head outputs two segmentation categories (i.e., foreground and background), so the output channel number of the last 1x1 convolutional layer in the foreground-background segmentation head is set to 1, and the deep feature information obtained after the convolution can represent the probability that each pixel point in the image to be processed is predicted as foreground, and the pixel value of the foreground-background segmentation result can represent the probability that the corresponding pixel point is foreground, and the range of the pixel value is [0, 1].

[0037] The application analyzes and processes the image feature information through a preset number of first convolutional layers, a first batch normalization layer, a first activation function, a first fully connected layer, a second fully connected layer, and a second activation function to obtain first channel weight coefficients, performs calibration processing on the first feature map information according to the first channel weight coefficients to obtain a first calibration processing result, and performs up-sampling, convolution, and normalization processing on the first calibration processing result to obtain a foreground-background segmentation result, thereby realizing accurate foreground-background segmentation of the image to be processed.

[0038] In one possible implementation, the multi-task image segmentation method provided in the application utilizes a semantic segmentation head to analyze and process image feature information to obtain a semantic segmentation result, including: performing deep feature extraction on the image feature information through a preset number of third convolutional layers to obtain deep feature information, and performing normalization processing and introducing non-linear transformation on the deep feature information through a third batch normalization layer and a third activation function to obtain second feature map information; performing global average pooling operation on the second feature map information, performing channel compression processing on the global average pooling operation result through a third fully connected layer to obtain a channel compression result, and restoring the channel compression result to an original channel number through a fourth fully connected layer to obtain a channel restoration result, and performing analysis and processing on the channel restoration result through a fourth activation function to obtain third channel weight coefficients of the second feature map information; performing calibration processing on the second feature map information according to the third channel weight coefficients to obtain a third calibration processing result, and performing up-sampling and convolution processing on the third calibration processing result to obtain a plurality of segmentation class probability maps; and performing maximum probability processing on the plurality of segmentation class probability maps through a maximum value layer to obtain the semantic segmentation result.

[0039] The number of segmentation classes output by the semantic segmentation head can be a first number (for example, the first number is set to 9, and the segmentation classes include face, body skin, hair, clothes, musical instrument, microphone, headset, chair, and others), and correspondingly, the output channel number of the 1x1 convolution in the last layer of the semantic segmentation head is set to the first number, a first number of probability maps are output, and then a maximum value layer (argmax layer) is used to take the maximum probability value of each pixel in each probability map to obtain a semantic segmentation map, and the pixel class of each pixel is determined according to the segmentation class corresponding to the maximum probability value of each pixel in the semantic segmentation map, and the semantic segmentation result is generated according to the pixel class of each pixel.

[0040] The application analyzes and processes the image feature information through the third convolutional layer, the third batch normalization layer, the third activation function, the third fully connected layer, the fourth fully connected layer, and the fourth activation function to obtain the third channel weight coefficients, calibrates the second feature map information according to the third channel weight coefficients to obtain the third calibration processing result, performs up-sampling and convolution processing on the third calibration processing result to obtain a plurality of segmentation class probability maps, and performs maximum probability processing on the plurality of segmentation class probability maps to obtain the semantic segmentation result, thereby realizing accurate semantic segmentation of the image to be processed.

[0041] In one possible implementation, the multi-task image segmentation method provided in the application uses the face segmentation head to analyze and process image feature information to obtain a face image probability that the to-be-processed image belongs to a face image, including: performing deep feature extraction on the image feature information through a preset number of fifth convolutional layers to obtain deep feature information, and performing normalization processing and introducing a nonlinear transformation on the deep feature information through a fifth batch normalization layer and a fifth activation function to obtain third feature map information; performing a global average pooling operation on the third feature map information, performing channel compression processing on the global average pooling operation result through a fifth fully connected layer to obtain a channel compression result, and restoring the channel compression result to an original channel number through a sixth fully connected layer to obtain a channel restoration result, and performing analysis and processing on the channel restoration result through a sixth activation function to obtain a fifth channel weight coefficient of the third feature map information; performing calibration processing on the third feature map information according to the fifth channel weight coefficient to obtain a fifth calibration processing result, and performing up-sampling and convolution processing on the fifth calibration processing result to obtain a face image probability map; and performing compression processing on the face image probability map through a pooling layer to obtain a face image probability.

[0042] The value output by the face segmentation head can be a float type numerical value, the output channel number of the last 1x1 convolution in the face segmentation head is set to 1, and the feature map after convolution is a two-dimensional probability map, which represents the probability of each pixel being predicted as a face. Finally, a two-dimensional adaptive average pooling layer with an output size of (1, 1) is added to compress the two-dimensional probability map into a probability value (i.e., obtain the face image probability), which represents the probability of the to-be-processed image containing a face.

[0043] The application analyzes and processes the image feature information through the fifth convolutional layer, the fifth batch normalization layer, the fifth activation function, the fifth fully connected layer, the sixth fully connected layer, and the sixth activation function to obtain the fifth channel weight coefficient, calibrates the third feature map information according to the fifth channel weight coefficient to obtain the fifth calibration processing result, and performs up-sampling, convolution, and compression processing on the fifth calibration processing result to obtain the face image probability, thereby accurately predicting the probability of the to-be-processed image being a face image.

[0044] In a possible embodiment, considering that video storage and transmission in the live streaming field mostly use YUV data, the training data set composed of YUV data can be used to train the multi-task image segmentation model, which can reduce the conversion process from YUV image to RGB image, reduce the intermediate link of data processing, and improve the data processing efficiency. The multi-task image segmentation model can also directly learn the features of YUV data, which helps the multi-task image segmentation model better adapt to the characteristics of video data in the live streaming scenario and improve the model performance. For example, when collecting the training data set, YUV format training samples can be read from video files or real-time video streams. After reading the training samples, the size of the training samples can be adjusted according to the input requirements of the multi-task image segmentation model, and the length and width of the training samples can be accurately scaled to the range that can be processed by the multi-task image segmentation model, to ensure that the data fits the model expectation in the spatial dimension. Then, the pixel values of the training samples are normalized to the interval [0, 1] to eliminate the interference of different pixel value ranges on the model training, which can effectively accelerate the model convergence process. Then, the YUV format training samples are converted into tensor form suitable for model input. In the deep learning framework, tensors are a general form of data representation and can be efficiently processed by the model. During the model training, the multi-task image segmentation model can be repeatedly forward propagated and backward propagated to gradually and deeply learn the rich features of YUV data, so that the model can deeply fit the characteristics of YUV data in the live streaming and short video field, and the multi-task image segmentation model can more accurately and efficiently process YUV data, thereby improving the data processing efficiency of the multi-task image segmentation model in the live streaming field.

[0045] In a possible embodiment, the training process of the multi-task image segmentation model provided in the application can include: alternately training the multi-task image segmentation model using the foreground and background segmentation data set and the semantic segmentation data set; freezing the foreground and background segmentation head and the face segmentation head in the multi-task image segmentation model that completes the alternating training, and fine-tuning the semantic segmentation head.

[0046] Exemplarily, after the multi-task image segmentation model is built, the multi-task image segmentation model can be alternately trained by using the foreground and background segmentation dataset and the semantic segmentation dataset. For example, the foreground and background segmentation dataset and the semantic segmentation dataset are loaded respectively, and all preset training periods (epochs) are traversed. For the current preset training period, assuming that the current preset training period needs to be iterated N times, in one iteration, a batch of data of the foreground and background segmentation dataset is loaded, forward propagation is performed based on the batch of data, and the corresponding loss is calculated, and then the parameter update of the current iteration is completed through back propagation; after entering the next iteration, a batch of data of the semantic segmentation dataset is loaded, and the forward propagation, loss calculation and back propagation operations are also performed to complete the parameter update of the corresponding iteration, and the above operations are repeated to realize the alternate training of the multi-task image segmentation model in different preset training periods. Since the output forms of the foreground and background segmentation and the semantic segmentation are all segmentation maps, the application effectively promotes the information sharing between different tasks by using the foreground and background segmentation and the semantic segmentation alternately, and the model training effect can be effectively improved.

[0047] After the alternate training of the multi-task image segmentation model is completed, the foreground and background segmentation head and the face segmentation head in the multi-task image segmentation model that has completed the alternate training can be frozen, and the semantic segmentation head can be fine-tuned by using the semantic segmentation dataset. Optionally, during the fine-tuning training of the semantic segmentation head, the learning rate of 0.0001 can be set to update the gradient of the semantic segmentation head, so as to improve the accuracy of the semantic segmentation head in classifying multiple categories of pixels and improve the accuracy of the multi-task image segmentation model in different segmentation tasks.

[0048] The training process of the face probability value can reuse the semantic segmentation dataset, and the two branches of semantic segmentation and face probability value prediction can be jointly trained. For example, in the training process of semantic segmentation, the intersection over union (IoU) between the face image prediction result output by the face segmentation head and the face region ground truth in the semantic segmentation dataset is calculated, which can be used to quantitatively evaluate the accuracy of face image prediction, and the face image prediction can be used as the supervised ground truth of face image prediction.

[0049] In one possible embodiment, the training process of the multi-task image segmentation model provided by the application includes: performing preliminary training on the multi-task image segmentation model by using a training dataset; calculating the loss value of each training sample in the training dataset by using the multi-task image segmentation model that has completed the preliminary training; screening a plurality of difficult samples from the training dataset according to the loss value, and performing back propagation training on the multi-task image segmentation model that has completed the preliminary training by using the difficult samples.

[0050] The training data set can include a foreground-background segmentation data set and a semantic segmentation data set, and a plurality of training samples are recorded in the training data set. The multi-task image segmentation model is preliminarily trained by using the training data set. After the preliminary training of the multi-task image segmentation model is completed, each training sample in the training data set is predicted by using the multi-task image segmentation model after the preliminary training, and a loss value of each training sample in the training data set is calculated according to a prediction result of the training sample and a corresponding true value. Optionally, an online hard example mining cross-entropy loss (OhemCELoss) can be used as the loss value, so that the model can pay more attention to complex and difficult-to-distinguish sample features, avoid over-attention to simple samples in the training process, and thus improve the accuracy and generalization ability of semantic segmentation of the model.

[0051] In one embodiment, a plurality of difficult samples are screened out from the training data set according to the loss value determined above, for example, a preset proportion (which can be 40%-70%, for example, 60%) of training samples in the sorted loss value are used as the difficult samples. Further, the multi-task image segmentation model after the preliminary training can be trained by using the difficult samples determined above. Optionally, the gradient can be calculated by using the loss value of the difficult samples in the back propagation stage, and the network parameters of the multi-task image segmentation model are updated according to the gradient by using an optimizer (for example, an Adam optimizer), so that the multi-task image segmentation model can pay more attention to complex and difficult-to-distinguish sample features in the training process, and avoid being excessively affected by simple samples. Moreover, as the training progresses, the understanding and distinguishing ability of the multi-task image segmentation model for various samples is continuously enhanced, so as to gradually improve the accuracy of semantic segmentation, and the multi-task image segmentation model can more accurately segment different semantic categories.

[0052] The present application calculates the loss value of each training sample in the training data set by using the multi-task image segmentation model after the preliminary training, screens out a plurality of difficult samples from the training data set according to the loss value, and trains the multi-task image segmentation model by using the difficult samples, so as to improve the multi-task image segmentation accuracy of the multi-task image segmentation model.

[0053] In one possible embodiment, the training process of the multi-task image segmentation model provided by the present application includes: preliminarily training the multi-task image segmentation model by using a training data set; calculating L1 norms of a plurality of channels in the multi-task image segmentation model after the preliminary training, and performing pruning processing on the multi-task image segmentation model after the preliminary training according to the L1 norms.

[0054] Exemplarily, the multi-task image segmentation model is preliminarily trained by using the training data set, and after the preliminary training of the multi-task image segmentation model is completed, the L1 norm of each channel weight parameter in the multi-task image segmentation model after the preliminary training is calculated, and the channels with the L1 norm less than the preset L1 norm threshold are removed from the multi-task image segmentation model, so as to realize the pruning processing of the multi-task image segmentation model. The smaller the L1 norm is, the smaller the influence of the corresponding channel on the performance of the multi-task image segmentation model is likely to be.

[0055] For example, assuming that MobileNetV3 is used as the backbone network, in a certain layer of convolution of the backbone network, it is assumed that the layer originally has 256 channels, after the L1 norm of each channel weight parameter is calculated and compared and screened with the preset L1 norm threshold, 150 channels will be removed. After removing these channels, the structure of the multi-task image segmentation model is simplified. The L1 norm of the plurality of channels in the multi-task image segmentation model after the preliminary training is used to prune the multi-task image segmentation model, which effectively reduces the parameter quantity and the calculation quantity of the multi-task image segmentation model while ensuring the multi-task image segmentation accuracy of the multi-task image segmentation model, and improves the multi-task image segmentation efficiency.

[0056] Optionally, the pruned multi-task image segmentation model can be further fine-tuned, for example, the multi-task image segmentation model is retrained for a preset number of rounds by using the training data set, so that the remaining parameters in the multi-task image segmentation model can better adapt to the data features, while greatly reducing the parameter quantity and the calculation quantity of the multi-task image segmentation model, the performance of the multi-task image segmentation model is maintained as much as possible, the calculation burden of the multi-task image segmentation model when running on a mobile terminal can be effectively reduced, the device running speed is significantly improved, and the requirement of the mobile terminal for efficient operation of the model is met.

[0057] S230: determining whether the face image probability meets the preset probability threshold requirement.

[0058] S240: in the case that the face image probability meets the preset probability threshold requirement, generating a minimum bounding rectangle according to the face semantic result in the semantic segmentation result, and determining the minimum bounding rectangle as the face detection frame.

[0059] Exemplarily, when generating the face detection frame of the to-be-processed image according to the semantic segmentation result and the face image probability, it can be determined whether the face image probability meets a preset probability threshold requirement. Optionally, the preset probability threshold can be set in a range of 0.4-0.6, for example, the preset probability threshold is set to 0.5. When the face image probability is greater than or equal to the preset probability threshold, it can be considered that the face image probability meets the preset probability threshold requirement, and when the face image probability is less than the preset probability threshold, it can be considered that the face image probability does not meet the preset probability threshold requirement. In an embodiment, in the case that the face image probability does not meet the preset probability threshold requirement, it can be determined that there is no face in the to-be-processed image, and the face detection frame is not output.

[0060] In an embodiment, when it is determined that the face image probability meets the preset probability threshold requirement, then a minimum circumscribed rectangle is generated according to the face semantic result in the semantic segmentation result, and the minimum circumscribed rectangle is determined as the face detection frame, and the coordinates (coordinates of four corners) corresponding to the face detection frame are output. For example, one or more connected domains of multiple face pixels are determined according to the face semantic result in the semantic segmentation result, and a minimum circumscribed rectangle of the one or more connected domains is determined. One connected domain corresponds to one minimum circumscribed rectangle.

[0061] The present application reduces the case of false generation of the face detection frame when the face image probability is low by generating the minimum circumscribed rectangle according to the face semantic result in the semantic segmentation result when the face image probability meets the preset probability threshold requirement, and determining the minimum circumscribed rectangle as the face detection frame, thereby improving the generation accuracy of the face detection frame.

[0062] In a possible embodiment, the multi-task image segmentation method provided by the present application generates a face detection frame of a to-be-processed image according to a semantic segmentation result and a face image probability, which can be generating a candidate detection frame of the to-be-processed image according to the semantic segmentation result and the face image probability; determining whether the candidate detection frame meets a preset face area threshold requirement, and determining the candidate detection frame that meets the preset face area threshold requirement as the face detection frame.

[0063] Exemplarily, one or more connected domains of multiple face pixels are determined according to the face semantic result in the semantic segmentation result, and a minimum circumscribed rectangle of the one or more connected domains is determined, and the minimum circumscribed rectangle is determined as the candidate detection frame.

[0064] The area information of the one or more candidate detection boxes is determined, and it is determined whether the area information meets a preset face area threshold requirement. When the area information of the candidate detection box meets the preset face area threshold requirement, the candidate detection box that meets the preset face area threshold requirement can be retained as a face detection box, and when the area information of the candidate detection box does not meet the preset face area threshold requirement, the candidate detection box that does not meet the preset face area threshold requirement can be removed.

[0065] Optionally, the area information of the candidate detection box can be an area proportion of the candidate detection box in the image to be processed, and the preset face area threshold requirement can be represented by a preset area range, where the preset area range includes a preset upper limit of the area proportion and a preset lower limit of the area proportion. When the area information of the candidate detection box is within the preset area range, it can be considered that the candidate detection box meets the preset face area threshold requirement. According to the present application, the candidate detection box of the image to be processed is generated according to the semantic segmentation result and the face image probability, and the face detection box is determined according to whether the candidate detection box meets the preset face area threshold requirement, which reduces the situation of mistakenly generating a face detection box when the face occupies too small or too large a proportion in the image (where it is difficult to distinguish between a face and skin when the face occupies too large a proportion, unnecessary face detection boxes can be reduced, and data processing pressure can be reduced), improves the accuracy and reliability of face detection box generation, and effectively improves the robustness of face detection in complex scenes.

[0066] According to the above, by inputting the image to be processed into the trained multi-task image segmentation model, the image feature information of the image to be processed is extracted by the backbone network of the multi-task image segmentation model, the foreground and background segmentation result is obtained by analyzing and processing the image feature information by the foreground and background segmentation head, the semantic segmentation result is obtained by analyzing and processing the image feature information by the semantic segmentation head, the face image probability that the image to be processed belongs to a face image is obtained by analyzing and processing the image feature information by the face segmentation head, and the face detection box of the image to be processed is generated according to the semantic segmentation result and the face image probability. Different segmentation heads can share the image feature information output by the backbone network, reducing the number of times of repeatedly extracting image feature information of the image to be processed by different segmentation heads, and the face segmentation head does not need to output a face detection box, but outputs a face image probability, and then generates a face detection box of the image to be processed according to the semantic segmentation result and the face image probability, reducing the complex post-processing process of the face segmentation head, effectively shortening the image segmentation inference time, and improving the image segmentation efficiency. At the same time, when the face image probability meets the preset probability threshold requirement, the minimum bounding rectangle is generated according to the face semantic result in the semantic segmentation result, and the minimum bounding rectangle is determined as the face detection box, reducing the situation of mistakenly generating a face detection box when the face image probability is low, and improving the accuracy of face detection box generation.

[0067] Figure 4A structural schematic diagram of a multi-task image segmentation device provided by an embodiment of the present application is given. Referring to Figure 4 The multi-task image segmentation device comprises an image acquisition module 41, an image segmentation module 42 and a face detection module 43.

[0068] The image acquisition module 41 is configured to acquire a to-be-processed image; the image segmentation module 42 is configured to input the to-be-processed image into a trained multi-task image segmentation model, use a backbone network of the multi-task image segmentation model to extract features of the to-be-processed image to obtain image feature information, use a foreground-background segmentation head to analyze and process the image feature information to obtain a foreground-background segmentation result, use a semantic segmentation head to analyze and process the image feature information to obtain a semantic segmentation result, and use a face segmentation head to analyze and process the image feature information to obtain a face image probability that the to-be-processed image belongs to a face image; and the face detection module 43 is configured to generate a face detection frame of the to-be-processed image according to the semantic segmentation result and the face image probability.

[0069] Through the above, the to-be-processed image is input into the trained multi-task image segmentation model, the image feature information is extracted from the to-be-processed image by using the backbone network of the multi-task image segmentation model, the foreground-background segmentation result is obtained by analyzing and processing the image feature information by using the foreground-background segmentation head, the semantic segmentation result is obtained by analyzing and processing the image feature information by using the semantic segmentation head, the face image probability that the to-be-processed image belongs to the face image is obtained by analyzing and processing the image feature information by using the face segmentation head, and the face detection frame of the to-be-processed image is generated according to the semantic segmentation result and the face image probability. Different segmentation heads can share the image feature information output by the backbone network, the number of times of repeatedly extracting the image feature information of the to-be-processed image by different segmentation heads is reduced, the face detection frame is not output by the face segmentation head but the face image probability is output, the face detection frame of the to-be-processed image is generated according to the semantic segmentation result and the face image probability, the complex post-processing process of the face segmentation head is reduced, the image segmentation inference time is effectively shortened, and the image segmentation efficiency is improved.

[0070] In one possible embodiment, the face detection module 43 generates the face detection frame of the to-be-processed image according to the semantic segmentation result and the face image probability, and is configured to: determine whether the face image probability meets a preset probability threshold requirement; in a case where the face image probability meets the preset probability threshold requirement, generate a minimum bounding rectangle according to a face semantic result in the semantic segmentation result, and determine the minimum bounding rectangle as the face detection frame.

[0071] In one possible embodiment, the face detection module 43 generates the face detection frame of the to-be-processed image according to the semantic segmentation result and the face image probability, and is configured to: According to the semantic segmentation result and the face image probability, a candidate detection frame of the image to be processed is generated; It is determined whether the candidate detection frame meets a preset face area threshold requirement, and the candidate detection frame meeting the preset face area threshold requirement is determined as a face detection frame.

[0072] In one possible embodiment, the multi-task image segmentation model uses a foreground-background segmentation head to analyze and process image feature information to obtain a foreground-background segmentation result, which is configured as: The image feature information is subjected to deep feature extraction through a preset number of first convolutional layers to obtain deep feature information, and the deep feature information is subjected to normalization processing and introduction of a nonlinear transformation through a first batch normalization layer and a first activation function to obtain first feature map information; The first feature map information is subjected to a global average pooling operation, the global average pooling operation result is subjected to channel compression processing through a first fully connected layer to obtain a channel compression result, and the channel compression result is restored to the original channel number through a second fully connected layer to obtain a channel restoration result, and the channel restoration result is analyzed and processed through a second activation function to obtain first channel weight coefficients of the first feature map information; The first feature map information is calibrated according to the first channel weight coefficients to obtain a first calibration processing result, and the first calibration processing result is subjected to upsampling and convolution processing to obtain a foreground probability feature map; The foreground probability feature map is subjected to normalization processing through a normalization layer to obtain the foreground-background segmentation result.

[0073] In one possible embodiment, the multi-task image segmentation model uses a semantic segmentation head to analyze and process image feature information to obtain a semantic segmentation result, which is configured as: The image feature information is subjected to deep feature extraction through a preset number of third convolutional layers to obtain deep feature information, and the deep feature information is subjected to normalization processing and introduction of a nonlinear transformation through a third batch normalization layer and a third activation function to obtain second feature map information; The second feature map information is subjected to a global average pooling operation, the global average pooling operation result is subjected to channel compression processing through a third fully connected layer to obtain a channel compression result, and the channel compression result is restored to the original channel number through a fourth fully connected layer to obtain a channel restoration result, and the channel restoration result is analyzed and processed through a fourth activation function to obtain third channel weight coefficients of the second feature map information; The second feature map information is calibrated according to the third channel weight coefficients to obtain a third calibration processing result, and the third calibration processing result is subjected to upsampling and convolution processing to obtain a plurality of segmentation class probability maps; The semantic segmentation result is obtained by maximum probability processing of the plurality of segmentation class probability maps through the maximum value layer.

[0074] In one possible embodiment, the multi-task image segmentation model utilizes the face segmentation head to analyze and process the image feature information to obtain a face image probability that the to-be-processed image belongs to a face image, and is configured to: The image feature information is subjected to deep feature extraction through the preset number of fifth convolutional layers to obtain deep feature information, and the deep feature information is subjected to normalization processing and introduction of nonlinear transformation through the fifth batch normalization layer and the fifth activation function to obtain third feature map information; The third feature map information is subjected to global average pooling operation, the global average pooling operation result is subjected to channel compression processing through the fifth fully connected layer to obtain a channel compression result, and the channel compression result is restored to the original channel number through the sixth fully connected layer to obtain a channel restoration result, and the channel restoration result is analyzed and processed through the sixth activation function to obtain a fifth channel weight coefficient of the third feature map information; The third feature map information is calibrated according to the fifth channel weight coefficient to obtain a fifth calibration processing result, and the fifth calibration processing result is subjected to up-sampling and convolution processing to obtain a face image probability map; The face image probability map is compressed through the pooling layer to obtain a face image probability.

[0075] In one possible embodiment, the training process of the multi-task image segmentation model is configured to: The multi-task image segmentation model is alternately trained using the foreground and background segmentation dataset and the semantic segmentation dataset; The foreground and background segmentation head and the face segmentation head in the multi-task image segmentation model that completes the alternate training are frozen, and the semantic segmentation head is fine-tuned and trained.

[0076] In one possible embodiment, the training process of the multi-task image segmentation model is configured to: The multi-task image segmentation model is preliminarily trained using the training dataset; The loss values of each training sample in the training dataset are calculated using the multi-task image segmentation model that completes the preliminary training; A plurality of difficult samples are screened out from the training dataset according to the loss values, and the multi-task image segmentation model that completes the preliminary training is subjected to backpropagation training using the difficult samples.

[0077] In one possible embodiment, the training process of the multi-task image segmentation model is configured to: The multi-task image segmentation model is preliminarily trained using the training dataset; The L1 norm of the plurality of channels in the preliminarily trained multi-task image segmentation model is calculated, and the preliminarily trained multi-task image segmentation model is pruned according to the L1 norm.

[0078] It is worth noting that in the above-mentioned embodiments of the multi-task image segmentation device, each unit and module included is only divided according to functional logic, but is not limited to the above-mentioned division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for the convenience of mutual differentiation, and are not used to limit the protection scope of the embodiments of the present application.

[0079] The embodiments of the present application also provide a multi-task image segmentation device, which can integrate the multi-task image segmentation device provided by the embodiments of the present application. Figure 5 FIG. 1 is a structural schematic diagram of a multi-task image segmentation device provided by an embodiment of the present application. Referring to FIG. 1, Figure 5 The multi-task image segmentation device includes an input device 53, an output device 54, a memory 52, and one or more processors 51; the memory 52 is used to store one or more programs; when the one or more programs are executed by the one or more processors 51, the one or more processors 51 implement the multi-task image segmentation method provided by the above-mentioned embodiments. The multi-task image segmentation device, device, and computer provided above can be used to execute the multi-task image segmentation method provided by any of the above-mentioned embodiments, and have the corresponding functions and advantages.

[0080] The embodiments of the present application also provide a non-volatile storage medium storing computer executable instructions, which are used to execute the multi-task image segmentation method provided by the above-mentioned embodiments when executed by a computer processor. Of course, the non-volatile storage medium storing computer executable instructions provided by the embodiments of the present application has computer executable instructions which are not limited to the multi-task image segmentation method provided above, but can also execute the related operations in the multi-task image segmentation method provided by any of the embodiments of the present application. The multi-task image segmentation device, device, and storage medium provided in the above-mentioned embodiments can execute the multi-task image segmentation method provided by any of the embodiments of the present application, and the technical details not described in the above-mentioned embodiments can be referred to the multi-task image segmentation method provided by any of the embodiments of the present application.

[0081] On the basis of the above-mentioned embodiments, the embodiments of the present application also provide a computer program product, the technical solution of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, the computer program product is stored in a storage medium, and includes a plurality of instructions for making a computer device, a mobile terminal, or a processor therein execute all or part of the steps of the multi-task image segmentation method provided by the embodiments of the present application.

Claims

1. A multi-task image segmentation method, characterized in that: include: Get the image to be processed; Inputting the image to be processed into a trained multi-task image segmentation model, extracting features of the image to be processed using a backbone network through the multi-task image segmentation model to obtain image feature information, analyzing and processing the image feature information using a foreground-background segmentation head to obtain a foreground-background segmentation result, analyzing and processing the image feature information using a semantic segmentation head to obtain a semantic segmentation result, and analyzing and processing the image feature information using a face segmentation head to obtain a facial image probability that the image to be processed is a face image; A face detection frame of the image to be processed is generated according to the semantic segmentation result and the face image probability.

2. The multi-task image segmentation method according to claim 1, characterized in that: Generating a face detection frame of the image to be processed according to the semantic segmentation result and the face image probability includes: Determining whether the probability of the face image meets a preset probability threshold requirement; When the probability of the face image meets a preset probability threshold requirement, a minimum bounding rectangle is generated according to the face semantic result in the semantic segmentation result, and the minimum bounding rectangle is determined as a face detection frame.

3. The multi-task image segmentation method according to claim 1, characterized in that: Generating a face detection frame of the image to be processed according to the semantic segmentation result and the face image probability includes: Generating a candidate detection frame of the image to be processed according to the semantic segmentation result and the face image probability; Determine whether the candidate detection frame meets a preset face area threshold requirement, and determine the candidate detection frame that meets the preset face area threshold requirement as a face detection frame.

4. The multi-task image segmentation method according to claim 1, characterized in that: The foreground-background segmentation head is used to analyze and process the image feature information to obtain a foreground-background segmentation result, including: Performing depth feature extraction on the image feature information through a preset number of first convolutional layers to obtain depth feature information, and performing normalization processing on the depth feature information through a first batch of normalization layers and a first activation function and introducing a nonlinear transformation to obtain first feature map information; Performing a global average pooling operation on the first feature map information, performing channel compression processing on the global average pooling operation result through a first fully connected layer to obtain a channel compression result, restoring the channel compression result to the original number of channels through a second fully connected layer to obtain a channel recovery result, and analyzing the channel recovery result through a second activation function to obtain a first channel weight coefficient of the first feature map information; Calibrate the first feature map information according to the first channel weight coefficient to obtain a first calibration result, and upsample and convolve the first calibration result to obtain a foreground probability feature map; The foreground probability feature map is normalized by a normalization layer to obtain a foreground-background segmentation result.

5. The multi-task image segmentation method according to claim 1, characterized in that: The analyzing and processing the image feature information using the semantic segmentation head to obtain a semantic segmentation result includes: Performing depth feature extraction on the image feature information through a preset number of third convolutional layers to obtain depth feature information, and performing normalization processing on the depth feature information through a third batch normalization layer and a third activation function and introducing a nonlinear transformation to obtain second feature map information; performing a global average pooling operation on the second feature map information, performing channel compression processing on the global average pooling operation result through a third fully connected layer to obtain a channel compression result, restoring the channel compression result to the original number of channels through a fourth fully connected layer to obtain a channel recovery result, and analyzing the channel recovery result through a fourth activation function to obtain a third channel weight coefficient of the second feature map information; Calibrate the second feature map information according to the third channel weight coefficient to obtain a third calibration result, and upsample and convolve the third calibration result to obtain a plurality of segmentation category probability maps; The semantic segmentation result is obtained by performing maximum probability processing on the multiple segmentation category probability maps through the maximum layer.

6. The multi-task image segmentation method according to claim 1, characterized in that: The step of analyzing and processing the image feature information using a face segmentation head to obtain a facial image probability that the image to be processed belongs to a facial image includes: Performing depth feature extraction on the image feature information through a preset number of fifth convolutional layers to obtain depth feature information, and normalizing the depth feature information through a fifth batch normalization layer and a fifth activation function and introducing a nonlinear transformation to obtain third feature map information; performing a global average pooling operation on the third feature map information, performing channel compression processing on the global average pooling operation result through a fifth fully connected layer to obtain a channel compression result, restoring the channel compression result to the original number of channels through a sixth fully connected layer to obtain a channel recovery result, and analyzing the channel recovery result through a sixth activation function to obtain a fifth channel weight coefficient of the third feature map information; performing calibration processing on the third feature map information according to the fifth channel weight coefficient to obtain a fifth calibration processing result, and performing upsampling and convolution processing on the fifth calibration processing result to obtain a face image probability map; The face image probability map is compressed by a pooling layer to obtain the face image probability.

7. The multi-task image segmentation method according to claim 1, characterized in that: The training process of the multi-task image segmentation model includes: The multi-task image segmentation model is trained alternately using the foreground-background segmentation dataset and the semantic segmentation dataset; The foreground and background segmentation heads and the face segmentation heads in the multi-task image segmentation model that have completed alternating training are frozen, and the semantic segmentation head is fine-tuned.

8. The multi-task image segmentation method according to claim 1, characterized in that: The training process of the multi-task image segmentation model includes: Use the training dataset to perform preliminary training on the multi-task image segmentation model; Calculating the loss value of each training sample in the training data set using the multi-task image segmentation model that has completed preliminary training; A plurality of difficult samples are screened out from the training data set according to the loss value, and the difficult samples are used to perform back propagation training on the multi-task image segmentation model that has completed preliminary training.

9. The multi-task image segmentation method according to claim 1, characterized in that: The training process of the multi-task image segmentation model includes: Use the training dataset to perform preliminary training on the multi-task image segmentation model; Calculate the L1 norms of multiple channels in the multi-task image segmentation model that has completed preliminary training, and perform pruning processing on the multi-task image segmentation model that has completed preliminary training according to the L1 norm.

10. A multi-task image segmentation device, characterized in that: It includes image acquisition module, image segmentation module and face detection module, among which: The image acquisition module is configured to acquire an image to be processed; The image segmentation module is configured to input the image to be processed into a trained multi-task image segmentation model, extract features of the image to be processed using a backbone network through the multi-task image segmentation model to obtain image feature information, analyze and process the image feature information using a foreground and background segmentation head to obtain a foreground and background segmentation result, analyze and process the image feature information using a semantic segmentation head to obtain a semantic segmentation result, and analyze and process the image feature information using a face segmentation head to obtain a facial image probability that the image to be processed is a face image; The face detection module is configured to generate a face detection frame for the image to be processed based on the semantic segmentation result and the face image probability.

11. A multi-task image segmentation device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-task image segmentation method according to any one of claims 1 to 9.

12. A non-volatile storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the multi-task image segmentation method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-task image segmentation method according to any one of claims 1 to 9 is implemented.

Images