Image segmentation processing method, device, computer equipment and storage medium

Through distillation training technology, the error of the guiding model is used to adjust the image segmentation model parameters, which solves the problem of difficulty in taking into account both accuracy and efficiency in image segmentation technology, and achieves efficient and accurate image segmentation processing.

CN113538480BActive Publication Date: 2024-12-03TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202011477949.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-15
Publication Date
2024-12-03
Estimated Expiration
2040-12-15

AI Technical Summary

Technical Problem

The existing image segmentation technology is difficult to improve processing efficiency while ensuring accuracy, especially in neural network models with high computational complexity, which leads to lower efficiency on devices with smaller processing capabilities.

Method used

Using distillation training method, the sample image is segmented through at least two trained guidance models, the distillation error and edge error are calculated, and the parameters of the image segmentation model are adjusted according to these errors until the training stop condition is met, and the lightweight stage is obtained. But an accurate image segmentation model.

Benefits of technology

It effectively improves the efficiency and accuracy of image segmentation processing, and obtains a lightweight image segmentation model with fewer model parameters but higher performance, which is suitable for devices with smaller processing capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113538480B_ABST
    Figure CN113538480B_ABST
Patent Text Reader

Abstract

The present application relates to an image segmentation processing method, device, computer equipment and storage medium, including: obtaining a sample image; performing image segmentation processing on a target object in the sample image respectively through at least two trained guidance models to obtain a first segmentation result corresponding to each guidance model; performing image segmentation processing on the target object in the sample image through the image segmentation model to be trained to obtain a second segmentation result; determining a distillation error according to the difference between the second segmentation result and the first segmentation result; determining an edge error according to the difference between an edge prediction result in the second segmentation result and an edge prediction result in the first segmentation result; adjusting model parameters of the image segmentation model according to the distillation error and the edge error and continuing distillation training until a training stop condition is met to obtain a trained image segmentation model. The present method can effectively ensure the processing efficiency of image segmentation while improving accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image segmentation processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the research and progress of artificial intelligence technology, artificial intelligence technology has been widely used in the field of image processing. Image segmentation is a crucial preprocessing for image recognition and computer vision, and can be effectively used to assist image classification, object detection and other tasks in various scene images. With the development of deep learning, various neural network models trained by deep learning have emerged for image segmentation processing.

[0003] However, in current image processing applications such as image segmentation, the neural network models used usually rely on a large amount of labeled training data and a large number of model parameters, which is accompanied by huge computational complexity. It is difficult to ensure the accuracy of image segmentation while ensuring the processing efficiency of image segmentation, resulting in low processing efficiency on devices with relatively small processing capabilities. Summary of the invention

[0004] Based on this, it is necessary to provide an image segmentation processing method, device, computer equipment and storage medium that can effectively ensure the accuracy of image segmentation while effectively improving the processing efficiency of image segmentation in response to the above technical problems.

[0005] An image segmentation processing method, the method comprising:

[0006] Get a sample image;

[0007] In each round of distillation training, at least two trained guidance models are used to perform image segmentation processing on the target object in the sample image, respectively, to obtain a first segmentation result corresponding to each of the guidance models;

[0008] Performing image segmentation processing on the target object in the sample image by using the image segmentation model to be trained to obtain a second segmentation result; the model parameters of the image segmentation model are less than the model parameters of the guidance model;

[0009] determining a distillation error according to a difference between the second segmentation result and the first segmentation result;

[0010] Determining an edge error according to a difference between an edge prediction result in the second segmentation result and an edge prediction result in the first segmentation result;

[0011] According to the distillation error and the edge error, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model.

[0012] An image segmentation processing device, the device comprising:

[0013] A data acquisition module, used for acquiring sample images;

[0014] A first segmentation processing module, used to perform image segmentation processing on the target object in the sample image respectively through at least two trained guidance models in each round of distillation training, and obtain a first segmentation result corresponding to each of the guidance models;

[0015] A second segmentation processing module is used to perform image segmentation processing on the target object in the sample image through the image segmentation model to be trained to obtain a second segmentation result; the model parameters of the image segmentation model are less than the model parameters of the guidance model;

[0016] an error determination module, configured to determine a distillation error according to a difference between the second segmentation result and the first segmentation result; and determine an edge error according to a difference between an edge prediction result in the second segmentation result and an edge prediction result in the first segmentation result;

[0017] A parameter adjustment module is used to adjust the model parameters of the image segmentation model according to the distillation error and the edge error and continue the distillation training until the training stop condition is met to obtain a trained image segmentation model.

[0018] In one embodiment, the error determination module is also used to determine the pixel error based on the difference between the pixel information in the second segmentation result and the pixel information in the first segmentation result; the parameter adjustment module is also used to adjust the model parameters of the image segmentation model according to the distillation error, the pixel error and the edge error and continue the distillation training until the training stop condition is met to obtain a trained image segmentation model.

[0019] In one embodiment, the sample image includes an unlabeled sample image and a sample image carrying a labeled label; the error determination module is also used to execute, for the unlabeled sample image, the step of determining the edge error based on the difference between the edge prediction result in the second segmentation result and the edge prediction result in the first segmentation result, and the step of determining the pixel error based on the difference between the pixel information in the second segmentation result and the pixel information in the first segmentation result; for the labeled sample image, determining the edge error based on the difference between the edge prediction result in the second segmentation result and the edge label in the labeled label; for the labeled sample image, determining the pixel error based on the difference between the pixel information in the second segmentation result and the pixel information of the pixel labeled by the labeled label.

[0020] In one embodiment, the parameter adjustment module is also used to fuse the distillation error, the pixel error and the edge error to obtain a target error value; according to the target error value, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model.

[0021] In one embodiment, the guidance model includes a first guidance model and a second guidance model with different network structures; the first segmentation processing module is also used to perform image segmentation processing on the target object in the sample image through the trained first guidance model to obtain the segmentation result corresponding to the first guidance model; perform image segmentation processing on the target object in the sample image through the trained second guidance model to obtain the segmentation result corresponding to the second guidance model; and fuse the segmentation result corresponding to the first guidance model with the segmentation result corresponding to the second guidance model to obtain a first segmentation result.

[0022] In one embodiment, the image segmentation model includes a coding layer, a decoding layer and a fusion layer, and the coding layer and the decoding layer include a separable convolutional network; the second segmentation processing module is also used to input the sample image into the image segmentation model, and perform feature extraction and separable convolution processing on the sample image through the coding layer to obtain a first multi-scale feature map; through the decoding layer, deconvolution processing is performed on the first multi-scale feature map to obtain a second multi-scale feature; through the fusion layer, the first multi-scale feature map and the second multi-scale feature map are fused to obtain a sample image semantic feature, and the pixel points and contour edges of the target object are determined according to the sample image semantic features, and a second segmentation result is output according to the pixel points and the contour edges.

[0023] In one embodiment, the image segmentation processing device also includes a training module of a guidance model, which is used to obtain a sample image, wherein the sample image carries an annotation label; the sample image is input into the guidance model to be trained, and image segmentation processing is performed on the target object in the sample image through the guidance model to be trained to obtain a sample segmentation result; the pixel error is determined based on the difference between the pixel information in the sample segmentation result and the pixel information of the pixel annotated by the annotation label; the edge error is determined based on the difference between the edge prediction result in the sample segmentation result and the edge label of the annotation label; the model parameters of the guidance model are adjusted based on the pixel error and the edge error, and the iterative training is continued until the training stop condition is met to obtain a trained guidance model.

[0024] In one embodiment, the data acquisition module is further used to acquire an initial sample image; fill the initial sample image according to different types of image ratios; and normalize the filled sample image to obtain a preprocessed sample image.

[0025] In one embodiment, the image segmentation processing device also includes an image segmentation processing module, which is used to obtain an image to be segmented, and input the image to be segmented into a trained image segmentation model; through the encoding layer of the image segmentation model, feature extraction and separable convolution processing are performed on the image to be segmented to obtain a first multi-scale feature map; through the decoding layer of the image segmentation model, deconvolution processing is performed on the first multi-scale feature map to obtain a second multi-scale feature; through the fusion layer of the image segmentation model, the first multi-scale feature map and the second multi-scale feature map are fused to obtain image semantic features; the pixel points and contour edges of the target object are determined according to the image semantic features, and the segmentation result of the image to be segmented is obtained according to the pixel points and the contour edges.

[0026] In one embodiment, the image to be segmented is a continuous video frame, and the segmentation result of the image to be segmented includes an initial segmentation result corresponding to each of the video frames; the image segmentation processing module is also used to obtain the initial segmentation result of the current video frame, and the initial segmentation results corresponding to a preset number of video frames before the current video frame; according to the initial segmentation results corresponding to the preset number of video frames, the initial segmentation result of the current video frame is smoothed and filtered to obtain the target segmentation result corresponding to the current video frame.

[0027] In one embodiment, the segmentation result of the image to be segmented includes the area of ​​the target object and the background area; the image segmentation processing module is also used to obtain a virtual background; the background area in the image to be segmented is replaced with the virtual background image to generate an image after background replacement.

[0028] In one embodiment, the image to be segmented is a continuous video frame, and the target object is a person; the segmentation result of the image to be segmented includes the person area and the background area in each of the video frames; the image segmentation processing module is also used to replace the background area in each of the video frames with the virtual background image to generate continuous video frames after the background is replaced.

[0029] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0030] Get a sample image;

[0031] In each round of distillation training, at least two trained guidance models are used to perform image segmentation processing on the target object in the sample image, respectively, to obtain a first segmentation result corresponding to each of the guidance models;

[0032] Performing image segmentation processing on the target object in the sample image by using the image segmentation model to be trained to obtain a second segmentation result; the model parameters of the image segmentation model are less than the model parameters of the guidance model;

[0033] determining a distillation error according to a difference between the second segmentation result and the first segmentation result;

[0034] Determining an edge error according to a difference between an edge prediction result in the second segmentation result and an edge prediction result in the first segmentation result;

[0035] According to the distillation error and the edge error, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0037] Get a sample image;

[0038] In each round of distillation training, at least two trained guidance models are used to perform image segmentation processing on the target object in the sample image, respectively, to obtain a first segmentation result corresponding to each of the guidance models;

[0039] Performing image segmentation processing on the target object in the sample image by using the image segmentation model to be trained to obtain a second segmentation result; the model parameters of the image segmentation model are less than the model parameters of the guidance model;

[0040] determining a distillation error according to a difference between the second segmentation result and the first segmentation result;

[0041] Determining an edge error according to a difference between an edge prediction result in the second segmentation result and an edge prediction result in the first segmentation result;

[0042] According to the distillation error and the edge error, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model.

[0043] A computer program product or a computer program, wherein the computer program product or the computer program comprises computer instructions, wherein the computer instructions are stored in a computer-readable storage medium; a processor of the computer device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, the following steps are implemented:

[0044] Get a sample image;

[0045] In each round of distillation training, at least two trained guidance models are used to perform image segmentation processing on the target object in the sample image, respectively, to obtain a first segmentation result corresponding to each of the guidance models;

[0046] Performing image segmentation processing on the target object in the sample image by using the image segmentation model to be trained to obtain a second segmentation result; the model parameters of the image segmentation model are less than the model parameters of the guidance model;

[0047] determining a distillation error according to a difference between the second segmentation result and the first segmentation result;

[0048] Determining an edge error according to a difference between an edge prediction result in the second segmentation result and an edge prediction result in the first segmentation result;

[0049] According to the distillation error and the edge error, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model.

[0050] The above-mentioned image segmentation processing method, device, computer equipment and storage medium, after obtaining the sample image, in each round of distillation training, respectively perform image segmentation processing on the target object in the sample image through at least two trained guidance models to obtain the first segmentation result corresponding to each guidance model; perform image segmentation processing on the target object in the sample image through the image segmentation model to be trained to obtain the second segmentation result; then determine the distillation error according to the difference between the second segmentation result and the first segmentation result; determine the edge error according to the difference between the edge prediction result in the second segmentation result and the edge prediction result in the first segmentation result, thereby effectively introducing the edge error in the distillation training for supervised training. Then, according to the distillation error and the edge error, adjust the model parameters of the image segmentation model and continue to perform distillation training until the training stop condition is met to obtain a trained image segmentation model. Since the image segmentation model is a lightweight model with fewer model parameters than the guidance model, the sample image is subjected to image segmentation processing through multiple trained guidance models with high image segmentation accuracy, and then the lightweight image segmentation model to be trained is subjected to distillation training according to the distillation error and the edge error determined by the first segmentation result and the second segmentation result, which can effectively improve the accuracy of the lightweight image segmentation model. In this way, a lightweight image segmentation model with high processing efficiency and high accuracy can be effectively trained. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A diagram showing an application environment of an image segmentation processing method in an embodiment;

[0052] Figure 2 A schematic diagram of a flow chart of an image segmentation processing method in one embodiment;

[0053] Figure 3 is a flowchart of an image segmentation processing method in another embodiment;

[0054] Figure 4 A flowchart of performing distillation training using a multi-guidance model in one embodiment;

[0055] Figure 5 is a schematic diagram of the structure of an image segmentation model in one embodiment;

[0056] Figure 6 A schematic diagram of a flow chart of the training steps of a guidance model in one embodiment;

[0057] Figure 7 is a flow chart of another image segmentation processing method in one embodiment;

[0058] Figure 8 is a schematic diagram of an image to be segmented in one embodiment;

[0059] Fig. 9 is a schematic diagram of a segmentation result corresponding to an image to be segmented in one embodiment;

[0060] Fig.10 is a flowchart of yet another image segmentation processing method in an embodiment;

[0061] Fig.11 A schematic diagram of a virtual background setting interface in one embodiment;

[0062] Fig.12 is a structural block diagram of an image segmentation processing device in one embodiment;

[0063] Fig.13 is a structural block diagram of an image segmentation processing device in another embodiment;

[0064] Fig.14 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] The image segmentation method provided in the present application can be applied to a computer device. The computer device can be a terminal or a server. It is understandable that the image segmentation processing method provided in the present application can be applied to a terminal, can also be applied to a server, can also be applied to a system including a terminal and a server, and can be implemented through the interaction between the terminal and the server.

[0067] The image segmentation processing method provided in this application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. Among them, the terminal 102 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 102 and the server 104 can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0068] Among them, cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called "cloud". As a basic capability provider for cloud computing, a cloud computing resource pool (referred to as cloud platform, generally referred to as IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose and use.

[0069] Specifically, after the terminal 102 collects the sample image, it uploads the sample image to the server 104. After the server 104 obtains the sample image, in each round of distillation training, at least two trained guidance models are used to perform image segmentation processing on the target object in the sample image, respectively, to obtain the first segmentation result corresponding to each guidance model; the image segmentation model to be trained is used to perform image segmentation processing on the target object in the sample image, to obtain the second segmentation result; the distillation error is determined according to the difference between the second segmentation result and the first segmentation result; the edge error is determined according to the difference between the edge prediction result in the second segmentation result and the edge prediction result in the first segmentation result. Then, according to the distillation error and the edge error, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model.

[0070] It can be understood that the image segmentation method in each embodiment of the present application adopts computer vision technology and machine learning technology in artificial intelligence technology, etc., which can effectively realize automatic detection and segmentation of target objects in three-dimensional images. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0071] Computer Vision (CV) is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. It can be understood that this application uses computer vision technology to detect and segment target objects from images.

[0072] Machine Learning (ML) is a multi-disciplinary cross-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning. It can be understood that the semantic segmentation network and instance segmentation network used in some embodiments of the present application are obtained by training using machine learning technology. Based on the guidance model and image segmentation model obtained by training using the machine learning technology, the segmentation results corresponding to the target object can be segmented more accurately from the image.

[0073] In one embodiment, Figure 2 As shown, an image segmentation processing method is provided, and the method is applied to a computer device as an example for explanation. The computer device can be specifically Figure 1 It is understood that the method can also be applied to a system including a terminal and a server and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0074] S202, obtaining a sample image.

[0075] In the field of computer vision, image segmentation refers to the technology and process of dividing an image into several specific regions with unique properties and proposing targets of interest. In other words, it is the process of subdividing a digital image into multiple image sub-regions, that is, a collection of pixels, also known as superpixels.

[0076] The target object refers to the object to be segmented in the image, that is, the image data is segmented for the target object to locate the object and boundary corresponding to the target object in the image, which can be represented by lines, curves, etc. For example, the target object can be a person, face, vehicle in the image, or a medical detection object in a medical image, etc., which is not limited here.

[0077] It can be understood that the sample image is an image used as sample data for model training, and the sample image includes unlabeled sample images, that is, there is no pre-labeled label for supervised training. The sample image can specifically be an image including a target object. The sample image can be directly obtained from the Internet or read from a memory, or an image captured by a shooting device can be used as a sample image. In one embodiment, the sample image includes both manually collected images and images crawled from public data sets or authorized websites.

[0078] The computer device first obtains unlabeled sample images, and then uses the obtained sample images to train the image segmentation model to be trained.

[0079] S204, in each round of distillation training, performing image segmentation processing on the target object in the sample image by using at least two trained guidance models, and obtaining a first segmentation result corresponding to each guidance model.

[0080] It can be understood that distillation training, also known as model distillation, refers to transferring the knowledge learned by a well-trained complex model with high accuracy to a network model with a simpler structure, or learning the knowledge in a complex model through a simple network model. For example, a complex model is usually called a teacher model, and a simple model is usually called a student model, so that the teacher model can be used to guide the training of the student model.

[0081] Among them, the guidance model is a machine learning model that has been trained with high image segmentation accuracy, and the guidance model is used to guide the training of the image segmentation model to be trained. Specifically, the guidance model can also be an image segmentation model with image segmentation processing functions. For example, the guidance model can be a machine learning model based on deep convolutional networks such as HRNet-OCR (HighResolutionNet+Object Contextual Representation), EPSPanopticDeep-Lab (Efficient Panoptic Segmentation Panoptic DeepLab), and MaskRCNN (Mask-based Convolutional Region Neural Network).

[0082] After the computer device obtains the sample image, it obtains at least two trained guidance models to perform distillation training on the image segmentation model to be trained using the at least two trained guidance models. The at least two trained guidance models are multiple guidance models with different structures. It can be understood that multiple refers to two or more.

[0083] In each round of distillation training, the computer device first inputs the sample images into at least two trained guidance models respectively, and performs image segmentation processing on the target objects in the sample images through at least two trained guidance models to obtain the first segmentation results corresponding to each guidance model.

[0084] Specifically, feature extraction and convolution processing are performed on the sample image through each network layer in each guidance model to extract the sample image semantic features of the sample image, determine the pixel points and contour edges belonging to the target object according to the sample image semantic features, and output the first segmentation result based on the pixel points and contour edges of the target object.

[0085] It can be understood that the first segmentation result includes not only the prediction result for the target object, but also the edge prediction result for the contour edge of the target object. The segmentation result can be a mask image corresponding to the sample image after image segmentation processing, specifically a binary mask image. It includes the pixel area of ​​the target object, that is, the foreground area, and the background area other than the pixel area of ​​the target object.

[0086] In one embodiment, the segmentation result may also include a mask image obtained by segmenting the contour edge of the target object.

[0087] It can be understood that the first segmentation result corresponding to each guiding model may include the segmentation result corresponding to each guiding model respectively, and may also be generated by fusing the segmentation results corresponding to each guiding model.

[0088] S206, performing image segmentation processing on the target object in the sample image by using the image segmentation model to be trained to obtain a second segmentation result; the model parameters of the image segmentation model are less than the model parameters of the guidance model.

[0089] The model parameters of the image segmentation model to be trained are less than those of the guidance model, that is, the image segmentation model to be trained is a lightweight model with fewer channels and less model depth than the guidance model. This enables the image segmentation model to process image segmentation tasks more efficiently on mobile devices with lower processing performance, and can effectively meet the real-time requirements for image segmentation processing.

[0090] For example, the image segmentation model may be a machine learning model based on neural networks such as Unet (U-shaped structure network), ResNet (residual network), RPN (Region Proposal Network), and FasterRCNN (Faster Region Convolutional Neural Networks, deep learning target detection network).

[0091] In each round of distillation training, the computer device also inputs the sample image into the image segmentation model to be trained, and performs image segmentation processing on the target object in the sample image through the image segmentation model to be trained to obtain a second segmentation result.

[0092] Specifically, the sample image can be subjected to feature extraction and convolution processing through each network layer in the image segmentation model to extract the sample image semantic features of the sample image, determine the pixel points and contour edges belonging to the target object according to the sample image semantic features, and output the second segmentation result according to the pixel points and contour edges of the target object. Similarly, the second segmentation result also includes not only the prediction result for the target object, but also the edge prediction result for the contour edge of the target object.

[0093] S208: Determine a distillation error according to a difference between the second segmentation result and the first segmentation result.

[0094] The distillation error refers to the error between the predicted probability of the pixel points in the sample image belonging to the target object in the second segmentation result and the predicted probability of the pixel points in the sample image belonging to the target object in the first segmentation result.

[0095] The computer device performs image segmentation processing on the sample image through each guidance model to obtain a corresponding first segmentation result, and performs image segmentation processing on the sample image through the image segmentation model to be trained to obtain a corresponding second segmentation result. The computer device determines the distillation error between the guidance model and the image segmentation model to be trained according to the difference between the predicted probability of the target object in the second segmentation result and the first segmentation result.

[0096] S210: Determine an edge error according to a difference between an edge prediction result in the second segmentation result and an edge prediction result in the first segmentation result.

[0097] The edge error is an error between an edge prediction result for the target object in the second segmentation result and an edge prediction result for the target object in the first segmentation result.

[0098] After the computer device obtains the first segmentation result and the second segmentation result corresponding to each guidance model, it also determines the edge error between the guidance model and the image segmentation model to be trained based on the difference between the edge prediction result for the target object in the second segmentation result and the edge prediction result for the target object in the first segmentation result.

[0099] S212, adjusting the model parameters of the image segmentation model according to the distillation error and the edge error and continuing the distillation training until the training stop condition is met to obtain a trained image segmentation model.

[0100] It can be understood that in the process of training the image segmentation model, it is necessary to perform multiple rounds of iterative distillation training on the image segmentation model. Each round of iterative distillation training is to gradually converge the model by adjusting the model parameters of the initial image segmentation model of this round to obtain the final image segmentation model. Among them, this round is the current round of model training.

[0101] The training stop condition refers to the end condition of the model training being satisfied, for example, the training stop condition may be reaching a preset number of iterations, or the image segmentation processing performance index of the image segmentation model after adjusting the parameters reaches a preset index. For example, the preset index may include the overall pixel accuracy, the intersection-and-union ratio of the pixel area of ​​the target object, the edge pixel accuracy, and the intersection-and-union ratio of the edge pixel area.

[0102] Specifically, after the computer device determines the distillation error and edge error between each guiding model and the image segmentation model to be trained in this round, the model parameters of the image segmentation model are adjusted according to the distillation error and the edge error, and the distillation training is continued. The image segmentation model to be trained continuously adjusts the model parameters of the image segmentation model in the direction of reducing the distillation error and the edge error according to the distillation error and the edge error. The image segmentation model continuously learns from the guiding model, so that the image segmentation processing capability and segmentation accuracy of the image segmentation model are close to those of the guiding model.

[0103] When the training stop condition is not met in this round, the next round of distillation training is entered, and the sample image is segmented through at least two guidance models and the image segmentation model to be trained. Then, according to the distillation error and edge error obtained in the next round, the model parameters of the image segmentation model are adjusted to perform iterative distillation training. When the training stop condition is met, the iterative training is stopped to obtain a trained image segmentation model.

[0104] In one of the embodiments, the image segmentation model can also be tested during the distillation training process or after the training is completed. Specifically, the computer device obtains a test image, performs image segmentation processing on the target object in the test image through the current image segmentation model, and obtains a test segmentation result. The test segmentation result is a binary mask image. The computer device tests the performance of the current image segmentation model according to preset indicators, such as overall pixel accuracy, intersection-over-union ratio of the pixel area of ​​the target object, edge pixel accuracy, and intersection-over-union ratio of the edge pixel area, so as to continuously optimize and iteratively train the structure of the image segmentation model according to the test results to improve the performance and accuracy of the image segmentation model.

[0105] It can be understood that the trained image segmentation model is a machine learning model that has the ability to perform image segmentation on images, and can be applied to perform image segmentation processing on images to segment the target object and background in the image.

[0106] In the above-mentioned image segmentation processing method, after obtaining the sample image, in each round of distillation training, at least two trained guidance models are used to perform image segmentation processing on the target object in the sample image, respectively, to obtain the first segmentation result corresponding to each guidance model; and the image segmentation model to be trained is used to perform image segmentation processing on the target object in the sample image, to obtain the second segmentation result. Then, the distillation error is determined according to the difference between the second segmentation result and the first segmentation result; and the edge error is determined according to the difference between the edge prediction result in the second segmentation result and the edge prediction result in the first segmentation result, thereby effectively introducing the edge error in the distillation training for supervised training. Then, according to the distillation error and the edge error, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model. Since the image segmentation model is a lightweight model with fewer model parameters than the guidance model, the sample image is subjected to image segmentation processing by using multiple trained guidance models with high image segmentation accuracy, and then the lightweight image segmentation model to be trained is subjected to distillation training according to the distillation error and the edge error determined by the first segmentation result and the second segmentation result, which can effectively improve the accuracy of the lightweight image segmentation model. In this way, it is possible to effectively train a lightweight image segmentation method that can ensure the accuracy of image segmentation while effectively improving the processing efficiency of image segmentation.

[0107] In one embodiment, the image segmentation processing method further includes: determining a pixel error according to a difference between pixel information in the second segmentation result and pixel information in the first segmentation result;

[0108] According to the distillation error and the edge error, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model, including: according to the distillation error, the pixel error and the edge error, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model.

[0109] It can be understood that the pixel error refers to the pixel difference between the pixel information in each image. The pixel error in this embodiment refers to the pixel difference between the pixel information in the second segmentation result and the pixel information in the first segmentation result.

[0110] Among them, the pixel information may include a binary pixel value and a pixel feature corresponding to each pixel point or pixel area. Therefore, the pixel error may include a binary pixel value error of each pixel point, and a pixel feature error. Specifically, the distillation error can be obtained by calculating the classification loss using the KLLoss (Kullback-Leibler divergence Loss, relative entropy loss) function, that is, comparing the predicted category probability of each pixel in the sample image in the second prediction result with the predicted category probability of each pixel in the sample image in the first prediction result, and determining the distillation error between the image segmentation model to be trained and the guidance model.

[0111] Among them, the pixel error may include the binary pixel value error of each pixel point, and the pixel feature error corresponding to each pixel point or pixel area. Specifically, the binary pixel value error can be calculated using the CrossEntropyLoss (cross entropy loss) function, that is, the binary pixel value of each pixel point in the pixel information in the second prediction result is compared with the difference between the binary pixel value of each pixel point in the pixel information in the second segmentation result, or with the difference between the binary pixel value of each pixel point in the pixel information annotated by the annotation label, to determine the binary pixel value error.

[0112] The pixel feature error can be calculated using an Affinity Loss function, that is, by comparing the pixel features corresponding to each pixel point or pixel area in the pixel information in the second prediction result with the difference between the pixel features corresponding to each pixel point or pixel area in the pixel information in the second segmentation result, or with the difference between the pixel features corresponding to the pixel information or pixel area marked by the annotation label, to determine the pixel error.

[0113] Furthermore, the edge error can be calculated using an Edge Loss (affinity loss) function, that is, the edge error is determined by comparing the edge prediction result for the target object in the second prediction result with the difference between the edge prediction result for the target object in the first segmentation result, or with the difference between the edge labels in the annotation label.

[0114] Specifically, after obtaining the first segmentation result and the second segmentation result corresponding to each guidance model, the computer device determines the distillation error according to the difference between the prediction probability of the target object in the second segmentation result and the first segmentation result; and determines the edge error according to the difference between the edge prediction result of the target object in the second segmentation result and the edge prediction result of the target object in the first segmentation result. At the same time, the computer device also determines the pixel error between the guidance model and the image segmentation model to be trained according to the difference between the pixel information in the second segmentation result and the pixel information in the first segmentation result.

[0115] The computer device then adjusts the model parameters of the image segmentation model according to the determined distillation error, pixel error, and edge error and continues the distillation training until the training stop condition is met to obtain a trained image segmentation model.

[0116] Specifically, the distillation error can be obtained by calculating the classification loss using the KLLoss (Kullback-Leibler divergence Loss, relative entropy loss) function, that is, comparing the predicted category probability of each pixel in the sample image in the second prediction result with the predicted category probability of each pixel in the sample image in the first prediction result, and determining the distillation error between the image segmentation model to be trained and the guidance model.

[0117] Among them, the pixel error may include the binary pixel value error of each pixel point, and the pixel feature error corresponding to each pixel point or pixel area. Specifically, the binary pixel value error can be calculated using the CrossEntropyLoss (cross entropy loss) function, that is, the binary pixel value of each pixel point in the pixel information in the second prediction result is compared with the difference between the binary pixel value of each pixel point in the pixel information in the second segmentation result, or with the difference between the binary pixel value of each pixel point in the pixel information annotated by the annotation label, to determine the binary pixel value error.

[0118] The pixel feature error can be calculated using an Affinity Loss function, that is, by comparing the pixel features corresponding to each pixel point or pixel area in the pixel information in the second prediction result with the difference between the pixel features corresponding to each pixel point or pixel area in the pixel information in the second segmentation result, or with the difference between the pixel features corresponding to the pixel information or pixel area marked by the annotation label, to determine the pixel error.

[0119] Furthermore, the edge error can be calculated using an Edge Loss (affinity loss) function, that is, the edge error is determined by comparing the edge prediction result for the target object in the second prediction result with the difference between the edge prediction result for the target object in the first segmentation result, or with the difference between the edge labels in the annotation label.

[0120] In this embodiment, during the distillation training process, the distillation error, pixel error and edge error between the guidance model and the image segmentation model to be trained are respectively determined, so that the image segmentation model to be trained continuously learns the knowledge in the guidance model based on the distillation error, pixel error and edge error, thereby effectively improving the accuracy of the image segmentation model.

[0121] In one embodiment, Figure 3 As shown, another image segmentation method is provided, comprising the following steps:

[0122] S302, obtaining sample images; the sample images include unlabeled sample images and sample images carrying labeled labels.

[0123] S304, in each round of distillation training, performing image segmentation processing on the target object in the sample image by using at least two trained guidance models, and obtaining a first segmentation result corresponding to each guidance model.

[0124] S306, performing image segmentation processing on the target object in the sample image by using the image segmentation model to be trained to obtain a second segmentation result; the model parameters of the image segmentation model are less than the model parameters of the guidance model.

[0125] S308: Determine a distillation error according to a difference between the second segmentation result and the first segmentation result.

[0126] S310, for an unlabeled sample image, determining an edge error according to a difference between an edge prediction result in a second segmentation result and an edge prediction result in a first segmentation result; determining a pixel error according to a difference between pixel information in the second segmentation result and pixel information in the first segmentation result.

[0127] S312, for the labeled sample image, determine the edge error according to the difference between the edge prediction result in the second segmentation result and the edge label in the labeled label; determine the pixel error according to the difference between the pixel information in the second segmentation result and the pixel information of the pixel labeled by the labeled label.

[0128] S314, adjusting the model parameters of the image segmentation model according to the distillation error, the pixel error and the edge error, and continuing the distillation training until the training stop condition is met, thereby obtaining a trained image segmentation model.

[0129] It can be understood that the sample images include a portion of unlabeled sample images and a portion of sample images with labeled labels. The labeled labels are labels marked for target objects in the sample images, which are used to compare the differences with the output results of the image segmentation model, so as to adjust the parameters of the image segmentation model. The labeled labels can be generated by manual labeling.

[0130] It can be understood that the annotation label may include not only pixel information annotated for the target object, but also edge labels for the contour edges of the target object.

[0131] In one embodiment, the sample images include unlabeled sample images of a preset ratio and labeled sample images carrying labeled labels. The number of unlabeled sample images is greater than the number of labeled sample images. For example, the preset ratio may be 5:1, such as the number of unlabeled sample images is 500,000 and the number of labeled sample images is 100,000. This can effectively reduce the resource consumption of labeling sample images, and at the same time, adding some labeled sample images can effectively improve the accuracy of the training image segmentation model.

[0132] After the computer device obtains the sample image, in each round of distillation training, at least two trained guidance models are used to perform image segmentation processing on the target object in the sample image, respectively, to obtain the first segmentation result corresponding to each guidance model; the image segmentation model to be trained is used to perform image segmentation processing on the target object in the sample image, to obtain the second segmentation result. Then, according to the difference between the predicted probability of the target object in the second segmentation result and the first segmentation result, the distillation error between the guidance model and the image segmentation model to be trained is determined.

[0133] Furthermore, for an unlabeled sample image, an edge error is determined based on a difference between an edge prediction result in the second segmentation result and an edge prediction result in the first segmentation result; and a pixel error is determined based on a difference between pixel information in the second segmentation result and pixel information in the first segmentation result.

[0134] For the labeled sample image, the edge error is determined based on the difference between the edge prediction result in the second segmentation result and the edge label in the labeled label; the pixel error is determined based on the difference between the pixel information in the second segmentation result and the pixel information of the pixel labeled by the labeled label.

[0135] The computer device then adjusts the model parameters of the image segmentation model according to the determined distillation error, pixel error, and edge error and continues the distillation training until the training stop condition is met to obtain a trained image segmentation model.

[0136] In this embodiment, by obtaining a larger portion of unlabeled sample images and a smaller portion of labeled sample images, the resource consumption of labeling sample images is effectively reduced, and the accuracy of the training image segmentation model can be effectively improved. In addition, during the distillation training process, the corresponding pixel errors and edge errors are determined for the unlabeled sample images and the labeled sample images, respectively, so that the image segmentation model to be trained continuously learns the knowledge in the guidance model according to the determined distillation error, pixel error, and edge error, thereby effectively improving the accuracy of the image segmentation model.

[0137] In one embodiment, according to the distillation error, pixel error and edge error, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model, including: fusing the distillation error, pixel error and edge error to obtain a target error value; according to the target error value, adjusting the model parameters of the image segmentation model and continuing the distillation training until the training stop condition is met to obtain a trained image segmentation model.

[0138] It can be understood that after the computer device determines the distillation error, pixel error and edge error respectively, it can also fuse the distillation error, pixel error and edge error. Specifically, the distillation error, pixel error and edge error can be directly accumulated to obtain a target error value. The distillation error, pixel error and edge error can also be weighted and summed to generate a target error value. The computer device then adjusts the model parameters of the image segmentation model according to the target error value and continues to perform distillation training until the training stop condition is met to obtain a trained image segmentation model.

[0139] In one embodiment, the guidance model includes a first guidance model and a second guidance model with different network structures; image segmentation processing is performed on the target object in the sample image through at least two trained guidance models, respectively, to obtain a first segmentation result corresponding to each guidance model, including: performing image segmentation processing on the target object in the sample image through the trained first guidance model to obtain a segmentation result corresponding to the first guidance model; performing image segmentation processing on the target object in the sample image through the trained second guidance model to obtain a segmentation result corresponding to the second guidance model; and fusing the segmentation result corresponding to the first guidance model with the segmentation result corresponding to the second guidance model to obtain a first segmentation result.

[0140] The guidance model includes a first guidance model and a second guidance model with different network structures. For example, the first guidance model may be a machine learning model based on the HRNet-OCR network; the second guidance model may be a machine learning model based on the EPSPanoptic Deep-Lab network.

[0141] After the computer device obtains the sample image, in each round of distillation training, the sample image is input into the trained first guiding model and the second guiding model respectively, and then the first guiding model is used to perform image segmentation processing on the target object in the sample image to obtain the segmentation result corresponding to the first guiding model. The second guiding model is used to perform image segmentation processing on the target object in the sample image to obtain the segmentation result corresponding to the second guiding model.

[0142] The computer device then fuses the segmentation result corresponding to the first guidance model with the segmentation result corresponding to the second guidance model. Specifically, the segmentation results corresponding to the first guidance model and the second guidance model are averaged and fused, such as weighted averaging the segmentation results of the two guidance models, to obtain a fused segmentation result. The fused segmentation result is the first segmentation result.

[0143] The computer device also performs image segmentation processing on the target object in the sample image through the image segmentation model to be trained to obtain a second segmentation result. The distillation error is determined based on the difference between the second segmentation result and the first segmentation result; the edge error is determined based on the difference between the edge prediction result in the second segmentation result and the edge prediction result in the first segmentation result. Then, based on the distillation error and the edge error, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model.

[0144] It can be understood that the number of guidance models in this embodiment is two. In some other embodiments, the number of guidance models can be more than two, and all of them are guidance models with different network structures.

[0145] For example, Figure 4 FIG. 1 is a flowchart of using a multi-guidance model to perform distillation training in one embodiment. Figure 4 , the guidance model 41 includes multiple, for example, more than three, including guidance model 1, guidance model 2...guidance model N. The image segmentation model to be trained is 42. Taking the target object to be segmented as a person as an example, in each round of distillation training, the sample image 40 is input into the multiple trained guidance models and the image segmentation model to be trained. Then each guidance model performs image segmentation processing on the target object in the sample image 40, and obtains the segmentation results corresponding to each guidance model. That is Figure 4 , the segmentation result output by the guidance model 1 is 4101, the segmentation result output by the guidance model 2 is 4102, and the segmentation result output by the guidance model N is 4103. Then, the segmentation result 4101, the segmentation result 4102, and the segmentation result 4103 are fused to obtain a fused first segmentation result 4104. In addition, the image segmentation model 42 to be trained is used to perform image segmentation processing on the target object in the sample image 40 to obtain a second segmentation result 4201.

[0146] The computer device determines the distillation error based on the difference between the second segmentation result 4201 and the fused first segmentation result 4104. The edge error is determined based on the difference between the edge prediction result in the second segmentation result 4201 and the edge prediction result in the first segmentation result 4104; the pixel error is determined based on the difference between the pixel information in the second segmentation result 4201 and the pixel information in the first segmentation result 4104. For the labeled sample image, the edge error is determined based on the difference between the edge prediction result in the second segmentation result 4201 and the edge label in the labeled label 401; the pixel error is determined based on the difference between the pixel information in the second segmentation result 4201 and the pixel information of the pixel labeled by the labeled label 401.

[0147] The computer device then adjusts the model parameters of the image segmentation model according to the distillation error and the edge error and continues the distillation training until the training stop condition is met to obtain a trained image segmentation model. Since the various guidance models have differences in network structure, the segmentation results of the various guidance models can be complemented, and the fusion result is greater than that of a single guidance model, thereby more effectively training a more accurate image segmentation model.

[0148] In one embodiment, an image segmentation model includes a coding layer, a decoding layer and a fusion layer, the coding layer and the decoding layer include a separable convolutional network; image segmentation processing is performed on a target object in a sample image through the image segmentation model to be trained to obtain a second segmentation result, including: inputting the sample image into the image segmentation model, performing feature extraction and separable convolution processing on the sample image through the coding layer to obtain a first multi-scale feature map; performing deconvolution processing on the first multi-scale feature map through the decoding layer to obtain a second multi-scale feature; fusing the first multi-scale feature map and the second multi-scale feature map through the fusion layer to obtain a semantic feature of the sample image, determining the pixel points and contour edges of the target object according to the semantic feature of the sample image, and outputting the second segmentation result according to the pixel points and contour edges.

[0149] It can be understood that the image segmentation model adopts a network structure including a coding layer, a decoding layer and a fusion layer. Figure 5 As shown, it is a structural diagram of an image segmentation model in an embodiment. The image segmentation model includes a coding layer 502, a decoding layer 504 and a fusion layer 506. Among them, the coding layer 502 and the decoding layer 504 include a separable convolutional network, that is, the image segmentation model adopts a machine learning model based on a separable convolutional network. Specifically, Depth-wise SeparableConvolutions (deep separable convolutional network) can be used. Compared with the traditional convolutional network, the use of a separable convolutional network can effectively reduce the number of channels and the model depth of the image segmentation model, thereby obtaining a lightweight image segmentation model with fewer model parameters. Among them, the sample image 50 is the input image of the image segmentation model. Taking the target object to be segmented in the sample image 50 as a person as an example, after the sample image 50 is input into the image segmentation model, the person in the sample image is subjected to image segmentation processing by the coding layer 502, the decoding layer 504 and the fusion layer 506 of the image segmentation model, and the edge segmentation result 5001 for the person and the target object segmentation result 5002 are output respectively. Among them, the edge segmentation result 5001 includes an edge segmentation result mask image for the edge contour area of ​​the person, and the target object segmentation result 5002 includes a segmentation result mask image for the pixel area belonging to the person.

[0150] Specifically, the encoding layer 502 of the image segmentation model may include multiple depth-separable convolutional networks for performing convolution processing, pooling processing, and downsampling processing on the input sample image to extract features of multiple scales of the sample image.

[0151] The decoding layer 504 of the image segmentation model also includes multiple depth-separable convolutional networks for deconvolution, pooling, and upsampling of the input. By adopting an encoding-decoding network structure, image semantic features at the image level can be accurately and effectively extracted.

[0152] The fusion layer 506 of the image segmentation model is used to fuse the results output by the encoding layer and the decoding layer to obtain the final sample image semantic features of the sample image and output the final segmentation result.

[0153] Specifically, in the process of performing image segmentation processing on the sample image by the image segmentation model to be trained, the computer device first inputs the sample image into the image segmentation model, and performs downsampling processing on the sample image through each depth-separable convolution network in the encoding layer to obtain a corresponding feature map. Then, feature extraction and separable convolution processing are performed on the corresponding feature augmentation map to obtain high-level local features, and feature maps of multiple scales and corresponding local features are obtained, that is, a first multi-scale feature map is generated.

[0154] The first multi-scale feature map output by the encoding layer and the corresponding local features are further input into the decoding layer, wherein the down-sampled pooling index is used as a guide for up-sampling, and the features of the encoding layer are jump-connected to the decoding layer, and then the first multi-scale feature map and the corresponding local features are deconvolved through the decoding layer to obtain the second multi-scale features.

[0155] Then, through the fusion layer of the image segmentation model, the first multi-scale feature map and the second multi-scale feature map are fused to obtain the semantic features of the sample image, thereby effectively determining the pixel points and contour edges of the target object based on the semantic features of the sample image, and outputting the second segmentation result based on the pixel points and contour edges of the target object.

[0156] In this embodiment, by adopting the encoding-decoding network structure of the separable convolutional network, the number of model channels and model parameters of the image segmentation model can be effectively reduced. Therefore, during the distillation training process, a lightweight image segmentation model with high segmentation accuracy can be effectively trained.

[0157] In one embodiment, Figure 6 As shown, the training steps of the guidance model include:

[0158] S602, obtaining a sample image, where the sample image carries a label.

[0159] S604, inputting the sample image into the guidance model to be trained, performing image segmentation processing on the target object in the sample image through the guidance model to be trained, and obtaining a sample segmentation result.

[0160] S606: Determine a pixel error according to the difference between the pixel information in the sample segmentation result and the pixel information of the pixel annotated by the annotation label.

[0161] S608, determining an edge error according to a difference between an edge prediction result in the sample segmentation result and an edge label of the annotated label.

[0162] S610, adjusting the model parameters of the guidance model according to the pixel error and the edge error and continuing the iterative training until the training stop condition is met to obtain a trained guidance model.

[0163] The sample images are training sample data for training guidance models, and the training sample data are all sample images with annotated labels, and the annotated labels are labels marked for target objects in the sample images. The annotated labels include pixel information of the annotated pixels and edge labels.

[0164] Before the distillation training of the image segmentation model, at least two guidance models are pre-trained. The training process of the guidance model can be the model training of the first stage, and the distillation training process of the image segmentation model is the model training of the second stage. It can be understood that in the model training process of the first stage, at least two guidance models can be trained simultaneously. For example, the same sample image can be used to train multiple guidance models simultaneously. In one embodiment, a multi-GPU (Graphics Processing Unit) distributed method can also be used to efficiently train multiple guidance models simultaneously.

[0165] Specifically, after the computer acquires the sample image, the sample image is input into the guidance model to be trained, and the sample image is feature extracted through each network layer in the guidance model to be trained, so as to extract the sample image semantic features corresponding to the sample image, and to determine the pixel points and contour edges belonging to the target object in the sample image according to the sample image semantic features, and output the sample segmentation result according to the pixel points and contour edges of the target object. The sample segmentation result includes the pixel information obtained based on the feature extraction and the edge prediction result of the target object.

[0166] After the computer device obtains the sample segmentation result, it determines the pixel error based on the difference between the pixel information in the sample segmentation result and the pixel information of the pixel annotated by the annotated label. At the same time, it also determines the edge error based on the difference between the edge prediction result in the sample segmentation result and the edge label of the annotated label.

[0167] The computer device then adjusts the model parameters of the guidance model according to the pixel error and the edge error and continues to perform iterative training until the training stop condition is met to obtain a trained guidance model. Among them, the difference between the sample segmentation result and the annotation label can also be measured by a loss function. For example, functions such as the cross entropy loss function, the mean absolute value loss function, the smoothed mean absolute error, etc. can be selected as loss functions. For example, the model parameters of the guidance model can also be updated by the Adam (Adaptive moment estimation) algorithm, and the iterative training is continued until the model converges to obtain the trained large model parameters. The training condition is the condition for ending the network model training. The training stop condition can be reaching a preset number of iterations, or the image segmentation performance index of the guidance model after adjusting the parameters reaches a preset index.

[0168] In this embodiment, the pixels and contour edges corresponding to the target object in the sample image are predicted and trained, and according to the difference between the sample segmentation result and the annotated label, the pixel error is calculated, and the edge error is introduced for supervised training, and the parameters of the instance segmentation network are gradually adjusted. Therefore, in the parameter adjustment process, after multiple feature extractions and feature enhancements, the guidance model can effectively obtain deep-level image features, thereby training a guidance model with high segmentation accuracy for the target object.

[0169] In one embodiment, obtaining a sample image includes: obtaining an initial sample image; filling the initial sample image according to different types of image ratios; and normalizing the filled sample image to obtain a preprocessed sample image.

[0170] It is understandable that before using the sample images for model distillation training, the acquired initial sample images may also be preprocessed. Among them, different types of image ratios may be inconsistent aspect ratios of the images. For example, different types of image ratios may include image ratios of a landscape type and image ratios of a portrait type, that is, a landscape ratio and a portrait ratio. For example, the landscape ratio is usually 16:9, and the portrait ratio is usually 9:16.

[0171] The computer device first fills the initial sample image according to different types of image ratios. Specifically, the sample image is expanded to the corresponding image ratio, and the blank area after expansion is filled with a preset pixel value, such as a pixel value of 0. Furthermore, after the initial sample image is filled, the filled initial sample image is also reduced to a preset image size. For example, the sample image for the horizontal screen ratio is reduced to 288*160; the sample image for the vertical screen ratio is reduced to 160*288. In this way, the image aspect ratio can be effectively maintained consistent, thereby effectively preventing interference caused by image width and height disturbance.

[0172] The computer device further performs normalization processing on the scaled-down sample image, for example, normalizing the image pixel values ​​of the sample image to [-1, 1], thereby obtaining a preprocessed sample image.

[0173] In one of the embodiments, the image segmentation model to be trained may also include a model channel for the horizontal screen ratio and a model channel for the vertical screen ratio, so that the image segmentation model can adapt to different types of image ratios and can accurately perform image segmentation processing.

[0174] In one embodiment, after obtaining the trained image segmentation model, the trained image segmentation model can be applied. Specifically, the computer can obtain the image to be segmented, input the image to be segmented into the trained image segmentation model, and perform image segmentation processing on the target object in the image to be segmented through the image segmentation model to obtain the segmentation result of the target object in the image to be segmented.

[0175] In one embodiment, Figure 7 As shown, after obtaining the trained image segmentation model, another image segmentation processing method is provided, including the following steps:

[0176] S702, obtaining an image to be segmented, and inputting the image to be segmented into a trained image segmentation model.

[0177] S704, performing feature extraction and separable convolution processing on the image to be segmented through the encoding layer of the image segmentation model to obtain a first multi-scale feature map.

[0178] S706: Perform deconvolution processing on the first multi-scale feature map through a decoding layer of the image segmentation model to obtain a second multi-scale feature.

[0179] S708, fusing the first multi-scale feature map and the second multi-scale feature map through a fusion layer of the image segmentation model to obtain image semantic features.

[0180] S710, determining the pixel points and the contour edge of the target object according to the image semantic features, and obtaining the segmentation result of the image to be segmented according to the pixel points and the contour edge.

[0181] It can be understood that the image to be segmented refers to an unlabeled image that needs to be segmented. The image to be segmented may include but is not limited to an image captured in real time by a camera device, an image transmitted in network communication, and an image obtained from the network.

[0182] The computer device uses sample images to iteratively distill the image segmentation model to obtain a trained image segmentation model with high image segmentation accuracy. The pre-trained image segmentation model can then be used directly to perform image segmentation processing on the image to be segmented.

[0183] Among them, the image segmentation model is a machine learning model based on a separation convolutional network, and the image segmentation model includes an encoding layer, a decoding layer, and a fusion layer.

[0184] Specifically, after the computer device obtains the image to be segmented, it inputs the image to be segmented into a trained image segmentation model. First, the image to be segmented is downsampled through each depth-separable convolutional network in the encoding layer of the image segmentation model to obtain a corresponding feature map. Then, feature extraction and separable convolution processing are performed on the corresponding feature augmentation map to obtain high-level local features, obtain feature maps of multiple scales and corresponding local features, and obtain a first multi-scale feature map.

[0185] The first multi-scale feature map output by the encoding layer and the corresponding local features are further input into the decoding layer, wherein the down-sampled pooling index is used as a guide for up-sampling, and the features of the encoding layer are jump-connected to the decoding layer, and then the first multi-scale feature map and the corresponding local features are deconvolved through the decoding layer to obtain the second multi-scale features.

[0186] Then, the first multi-scale feature map and the second multi-scale feature map are fused through the fusion layer of the image segmentation model to obtain image semantic features, and then the pixel points and contour edges of the target object are determined according to the image semantic features, and the segmentation result of the image to be segmented is output according to the pixel points and contour edges of the target object. The segmentation result can be a mask map corresponding to the sample image after image segmentation processing, and specifically can be a binary mask map.

[0187] The segmentation result includes the pixel area of ​​the target object, i.e., the foreground area, and the background area other than the pixel area of ​​the target object. For example, the obtained segmentation result mask image can use binary pixel values ​​0 and 1 to mark the segmentation result of the target object, i.e., the pixel value of the pixel point in the pixel area of ​​the target object is marked as 1, and the pixel value of the pixel point in the background area is marked as 0.

[0188] For example, Figure 8 FIG. 1 is a schematic diagram of an image to be segmented in an embodiment. It can be seen that Figure 8 The target object is a person in the image. The trained image segmentation model is used to segment the Figure 8 After performing image segmentation on the person in the image, the segmentation result is as follows: Fig. 9 As shown, Fig. 9 That is a schematic diagram of the segmentation result corresponding to the image to be segmented. Fig. 9 The segmentation result shown is a segmentation result mask image in which the person area is separated from the background area after pixel binarization processing.

[0189] In one embodiment, the segmentation result may also include a mask image after the contour edge of the target object is segmented. That is, the segmentation result may include two segmentation result mask images, namely, a segmentation result mask image for separating the pixel area of ​​the target object from the background area, and a segmentation result mask image for the contour edge of the target object. In the segmentation result mask image of the contour edge, the pixel values ​​of the pixel points in the contour edge area of ​​the target object are marked as 1, and the pixel values ​​of the pixel points in the area other than the contour edge area are marked as 0.

[0190] Furthermore, the computer device restores the segmentation result mask output by the image segmentation model to the original image size, that is, the original image size of the image to be segmented. Then, the connected domain processing is performed on the area of ​​the target object in the segmentation result mask. For example, if there are multiple connected domains of smaller areas in the segmentation result mask, they are considered as outliers and removed to obtain the final segmentation result of the image to be segmented.

[0191] In this embodiment, since the trained image segmentation model is a lightweight model with a small number of model channels and model parameters and a high image segmentation accuracy, it can efficiently perform image segmentation processing on the image to be segmented, and at the same time can effectively ensure the image segmentation accuracy.

[0192] In one embodiment, Fig.10 As shown, after obtaining the trained image segmentation model, another image segmentation processing method is provided, comprising the following steps:

[0193] S1002, obtaining continuous video frames, and inputting the continuous video frames into a trained image segmentation model.

[0194] S1004, performing feature extraction and separable convolution processing on each video frame through the encoding layer of the image segmentation model to obtain a first multi-scale feature map.

[0195] S1006, performing deconvolution processing on the first multi-scale feature map through a decoding layer of the image segmentation model to obtain a second multi-scale feature.

[0196] S1008, fusing the first multi-scale feature map and the second multi-scale feature map through a fusion layer of the image segmentation model to obtain image semantic features.

[0197] S1010, determining the pixel points and contour edges of the target object according to the image semantic features, and obtaining the initial segmentation results corresponding to each video frame according to the pixel points and contour edges.

[0198] S1012, obtaining an initial segmentation result of the current video frame and initial segmentation results corresponding to a preset number of video frames before the current video frame.

[0199] S1014, performing smoothing filtering on the initial segmentation result of the current video frame according to the initial segmentation results corresponding to a preset number of video frames, to obtain a target segmentation result corresponding to the current video frame.

[0200] The image to be segmented may also be a continuous video frame, which refers to a continuous picture image collected at a certain frame rate. The continuous video frames may be obtained from a video, and the video includes a series of continuous picture images, i.e., video frames. It can be understood that the video can also be segmented by a trained image segmentation model.

[0201] After the computer device acquires continuous video frames, the continuous video frames are respectively input into the trained image segmentation model according to the inter-frame sequence. Then, the continuous video frames are respectively subjected to image segmentation processing by the image segmentation model. Specifically, through the encoding layer of the image segmentation model, feature extraction and separable convolution processing are performed on each video frame to obtain a first multi-scale feature map. Then, through the decoding layer of the image segmentation model, the first multi-scale feature map is deconvolved to obtain a second multi-scale feature. Finally, through the fusion layer of the image segmentation model, the first multi-scale feature map and the second multi-scale feature map are fused to obtain image semantic features. Then, the pixel points and contour edges of the target object are determined according to the image semantic features, and the initial segmentation results corresponding to each video frame are output according to the pixel points and contour edges.

[0202] The current video frame may be any frame in the series of continuous video frames currently being processed. In one embodiment, the current video frame may be any frame except the first frame in the continuous video frames. It can be understood that the preset number of video frames before the current video frame is the adjacent preset number of video frames before the current video frame.

[0203] The initial segmentation result corresponding to each video frame includes a mask image corresponding to each video frame after image segmentation is performed on the target object.

[0204] The computer device performs image segmentation processing on the continuous video frames through the trained image segmentation model, and outputs the initial segmentation results corresponding to each video frame. It also obtains the initial segmentation results of the current video frame and the initial segmentation results corresponding to the preset number of video frames before the current video frame for each current video frame being processed.

[0205] Among them, the initial segmentation result corresponding to the video frame is a segmentation result mask image after image segmentation processing, which can be specifically a binary mask image, including the pixel area of ​​the target object, that is, the foreground area, and the background area other than the pixel area of ​​the target object.

[0206] The computer device then performs smoothing filtering on the segmentation result mask image of the current video frame according to the segmentation result mask images corresponding to the preset number of video frames, and obtains the target segmentation result corresponding to the current video frame. For example, the segmentation result mask images corresponding to the previous 4 video frames of the current video and the segmentation result mask image of the current video frame can be used to perform smoothing filtering together, so as to obtain the target segmentation result of the current video frame.

[0207] In this embodiment, by combining the segmentation result information of the previous frame to perform smoothing filtering on the subsequent current video frame, the segmentation result of the previous frame can be effectively combined as a reference, thereby effectively improving the accuracy of image segmentation processing on continuous video frames.

[0208] In one embodiment, the segmentation result of the image to be segmented includes the area of ​​the target object and the background area; the above-mentioned image segmentation processing method also includes: obtaining a virtual background; replacing the background area in the image to be segmented with a virtual background image to generate an image after background replacement.

[0209] The virtual background refers to a background image that can be used for editing, that is, a background layer used to set off the foreground. It is an independent picture that is not connected to the foreground picture or image. It can be understood that the virtual background is different from the image to be segmented and does not belong to the background image of the pixels in the image to be segmented.

[0210] The computer device performs image segmentation processing on the image to be segmented through the trained image segmentation model, and obtains a segmentation result in which the foreground area and the background area corresponding to the target object are separated. The computer device can further replace the background area.

[0211] Specifically, the computer device obtains a virtual background, which may be pre-set or selected by a user. The computer device then replaces the background area in the image to be segmented with the virtual background image, thereby obtaining an image after background replacement.

[0212] In this embodiment, the foreground area and the background area corresponding to the target object in the original image can be accurately segmented through the trained image segmentation model, and the background area can be replaced more accurately.

[0213] In one embodiment, the image to be segmented is a continuous video frame, and the target object is a person; the segmentation result of the image to be segmented includes the person area and the background area in each video frame; the background area in the image to be segmented is replaced with a virtual background image to generate an image after the background is replaced, including: replacing the background area in each video frame with a virtual background image to generate continuous video frames after the background is replaced.

[0214] The image to be segmented may be a continuous video frame, and the target object to be segmented may be a person. It can be understood that the segmentation result of the image to be segmented includes a mask image that separates the foreground area and the background area corresponding to the target object, that is, the segmentation result of each video frame includes a mask image that separates the person area and the background area in each video frame.

[0215] After the computer device obtains the segmentation results of each video frame, it can also replace the background of the continuous video frames. Specifically, the computer device obtains the virtual background, and then replaces the background area in each video frame with the virtual background image, thereby obtaining continuous video frames after the background is replaced.

[0216] In this embodiment, since the trained image segmentation model is a lightweight model with a small number of model channels and model parameters and a high image segmentation accuracy, it can efficiently perform image segmentation processing on continuous video frames, while ensuring the image segmentation accuracy and effectively ensuring the image segmentation processing efficiency.

[0217] In a specific application scenario, a trained image segmentation model is deployed in a mobile terminal. Specifically, an application with an image processing function may be installed in the mobile terminal, and the trained image segmentation model is deployed in the application.

[0218] The mobile terminal can perform image segmentation processing on the image to be segmented and realize other image processing functions by running the application with image processing function in the mobile terminal. Specifically, the user first obtains the image to be segmented through the mobile terminal. For example, the image to be segmented can be acquired in real time by a camera device of the mobile terminal, can be read locally from the mobile terminal, or can be acquired in real time through a network, such as an image transmitted in a communication network.

[0219] The mobile terminal then inputs the image to be segmented into a trained image segmentation model, and performs image segmentation processing on the target object in the image to be segmented through the image segmentation model to obtain a segmentation result of the target object in the image to be segmented. For example, the target object may be a person or a face. In this embodiment, the target object is a person, and the image segmentation model is used to perform image segmentation processing on the image to be segmented, identify the complete outline of the human body in the image, and segment it from the background area, and output a segmentation result mask image that separates the person area from the background area. For example, the segmentation result mask image may include a foreground person image and a grayscale background area image of the person area.

[0220] Among them, the application of the mobile terminal includes various virtual background materials. The user can also select the virtual background material through the mobile terminal to replace the background area after the image segmentation. Furthermore, the user can also obtain a customized virtual background, and replace the background area after the image segmentation through the mobile terminal, so as to synthesize the character area in the original image with the selected virtual background.

[0221] In other embodiments, the target object may also be a human face. The image to be segmented is processed by the image segmentation model, the contours of the facial features in the human face in the image are identified, and the facial features are segmented from the background area. A segmentation result mask image is obtained in which the human face area is separated from the background area. The user terminal can then replace the background area according to the background characteristics selected by the user, so that a synthesized expression image can be generated to generate a customized expression image.

[0222] In another specific application scenario, the above-mentioned image segmentation processing method can also be applied to the scenario of real-time video streaming in a mobile terminal. An application with image processing and communication functions can be installed in the mobile terminal, and a trained image segmentation model can be deployed in the application, or an SDK (Software Development Kit) with a trained image segmentation model can be called offline.

[0223] When a mobile terminal runs an application with image processing function, it can collect video streams in real time and identify the portrait area and background area in each video frame through a trained image segmentation model.

[0224] Specifically, the application program may be an online conference application, i.e., video conferencing software, etc. The continuous video frames are video streams captured in real time during the video conference. The video conferencing software of the mobile terminal includes a virtual background setting function. The user can choose to turn on or off the virtual background function in the virtual background setting interface, and can also select the material of the virtual background, such as a preset virtual filter such as background blur, or a virtual background image. Furthermore, the user can also select a customized virtual background image. The mobile terminal inputs the continuous video frames captured in real time into a trained image segmentation model according to the virtual background selected by the user. Through the image segmentation model, the portraits in the continuous video frames are subjected to image segmentation processing, and a segmentation result mask map is outputted to separate the portrait area from the background area in each video frame, and then the background area is replaced with a virtual background. The image frame after the background replacement is synthesized and then output and displayed.

[0225] like Fig.11 As shown, it is a schematic diagram of the setting interface of the virtual background in one embodiment. The setting interface includes a setting bar area 1101, which includes options such as "General Settings", "Video", "Audio" and "Background Replacement". When the background replacement option 1101a is selected, the background replacement preview area 1102 and the virtual background selection area 1103 are displayed. Among them, by selecting the background replacement option 1101a in the setting bar area 1101 of the setting interface, the virtual background setting options and virtual background materials are displayed in the virtual background selection area 1103. Among them, the virtual background setting options include "Video Filter Effect" and "I Have Green Cloth", and the virtual background materials include background blur, virtual background 1-virtual background 6, etc. The user can then set the virtual background setting options and select virtual background materials in the virtual background selection area 1103. When the user selects the virtual background, the effect of background replacement of the currently captured image or video frame is displayed in the background replacement preview area 1102. Compared to Figure 8 The original image to be segmented has a real background. Fig.11 The image or video frame after the background is replaced with the virtual background hides the real background and improves security.

[0226] Furthermore, for scenes with a fixed green screen background, a fixed green background is pre-arranged at the shooting scene before capturing video frames. The mobile terminal can use a pre-established green screen segmentation model to perform image segmentation processing on the real-time captured video through the green screen segmentation model, thereby obtaining more accurate and precise image segmentation results. The virtual background setting interface can also include a green screen selection option, namely "I have a green screen". When a green screen background is arranged at the scene of the video captured by the user, the virtual background setting option corresponding to "I have a green screen" can be selected to use different segmentation models to perform image segmentation processing on the video frames to achieve a more accurate segmentation effect.

[0227] In other embodiments, the above-mentioned image segmentation processing method can also be applied to an application with a live video broadcast function and a short video editing function in a mobile terminal, so as to perform image segmentation processing on the portraits in continuous video frames in the live video broadcast or continuous video frames in a video segment through a trained image segmentation model deployed or called in the application, so as to identify the portrait area and the background area in each video frame, and replace the background area with a virtual background.

[0228] In another embodiment, a trained image segmentation model is used to generate corresponding application software packages for different application scenarios and deployed to a set cloud service platform. For example, application software packages corresponding to application scenarios such as portrait segmentation scenarios, custom portrait segmentation scenarios, portrait segmentation SDK scenarios, and video portrait segmentation scenarios can be deployed in the cloud service platform. The user can call the application software package corresponding to the corresponding scenario from the cloud service platform through the corresponding mobile terminal, so that the mobile terminal performs image segmentation processing through the image segmentation model in the application software package. Thereby, it is possible to use the application software package corresponding to the trained image segmentation model to perform image segmentation processing in different application scenarios, thereby effectively ensuring the processing efficiency of image segmentation while improving the accuracy.

[0229] Among them, the application software package corresponding to the portrait segmentation scene can provide the function of identifying the complete outline of the person in the picture and segmenting it from the background. After the portrait segmentation processing is performed through the image segmentation model in the software package, the grayscale image and the foreground portrait image are returned. The grayscale image and foreground portrait image obtained after the segmentation processing can be further applied to scenes such as photo synthesis.

[0230] The image segmentation model in the application software package corresponding to the custom portrait segmentation scene can separate the facial features of the face from the background, and can also segment the hair, facial features, clothing, etc. in a single portrait picture. Among them, the facial features obtained after segmentation can be further applied to scenes such as making fun expressions and portrait beauty processing.

[0231] The image segmentation model in the application software package corresponding to the portrait segmentation SDK scene can call the portrait segmentation SDK offline in the real-time video stream scene to separate the portrait from the background in the video stream. The segmented portrait and background can be used for background replacement in short videos, live video broadcasts, online meetings and other scenes.

[0232] The image segmentation model in the application software package corresponding to the video portrait segmentation scene can separate the portrait from the background in the video uploaded by the mobile terminal to the cloud service platform, obtain the video after portrait segmentation and return it to the mobile terminal to achieve the segmentation of the portrait in the video. Among them, the video obtained after portrait segmentation can be further used in scenes such as video editing and film and television background replacement. By using the trained image segmentation model for image segmentation processing in different application scenarios, the processing efficiency of image segmentation can be effectively guaranteed while improving the accuracy.

[0233] It should be understood that although Figure 2 , 3 The steps in the flowcharts of , 6, 7, and 10 are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , 3 At least part of the steps in , 6, 7, and 10 may include multiple steps or multiple stages. These steps or stages do not necessarily have to be performed at the same time, but can be performed at different times. The execution order of these steps or stages does not necessarily have to be sequentially, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0234] In one embodiment, Fig.12 As shown, an image segmentation processing device 1200 is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: a data acquisition module 1202, a first segmentation processing module 1204, a second segmentation processing module 1206, an error determination module 1208 and a parameter adjustment module 1210, wherein:

[0235] The data acquisition module 1202 is used to acquire a sample image.

[0236] The first segmentation processing module 1204 is used to perform image segmentation processing on the target object in the sample image through at least two trained guidance models in each round of distillation training, and obtain the first segmentation result corresponding to each guidance model.

[0237] The second segmentation processing module 1206 is used to perform image segmentation processing on the target object in the sample image through the image segmentation model to be trained to obtain a second segmentation result; the model parameters of the image segmentation model are less than the model parameters of the guidance model.

[0238] The error determination module 1208 is used to determine the distillation error according to the difference between the second segmentation result and the first segmentation result; and determine the edge error according to the difference between the edge prediction result in the second segmentation result and the edge prediction result in the first segmentation result.

[0239] The parameter adjustment module 1210 is used to adjust the model parameters of the image segmentation model according to the distillation error and the edge error and continue the distillation training until the training stop condition is met to obtain a trained image segmentation model.

[0240] In one embodiment, the error determination module 1208 is also used to determine the pixel error based on the difference between the pixel information in the second segmentation result and the pixel information in the first segmentation result; the parameter adjustment module is also used to adjust the model parameters of the image segmentation model based on the distillation error, pixel error and edge error and continue the distillation training until the training stop condition is met to obtain a trained image segmentation model.

[0241] In one embodiment, the sample image also includes a sample image carrying an annotated label; the error determination module 1208 is also used to execute, for the unannotated sample image, a step of determining an edge error based on a difference between an edge prediction result in the second segmentation result and an edge prediction result in the first segmentation result, and a step of determining a pixel error based on a difference between pixel information in the second segmentation result and pixel information in the first segmentation result; for the annotated sample image, determine the edge error based on a difference between an edge prediction result in the second segmentation result and an edge label in the annotated label; for the annotated sample image, determine the pixel error based on a difference between pixel information in the second segmentation result and pixel information of pixels annotated by the annotated label.

[0242] In one embodiment, the parameter adjustment module 1210 is also used to fuse the distillation error, pixel error and edge error to obtain a target error value; according to the target error value, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model.

[0243] In one embodiment, the guidance model includes a first guidance model and a second guidance model with different network structures; the first segmentation processing module 1204 is also used to perform image segmentation processing on the target object in the sample image through the trained first guidance model to obtain the segmentation result corresponding to the first guidance model; perform image segmentation processing on the target object in the sample image through the trained second guidance model to obtain the segmentation result corresponding to the second guidance model; and fuse the segmentation result corresponding to the first guidance model with the segmentation result corresponding to the second guidance model to obtain the first segmentation result.

[0244] In one embodiment, the image segmentation model includes a coding layer, a decoding layer and a fusion layer, and the coding layer and the decoding layer include a separable convolutional network; the second segmentation processing module 1206 is also used to input the sample image into the image segmentation model, perform feature extraction and separable convolution processing on the sample image through the coding layer to obtain a first multi-scale feature map; perform deconvolution processing on the first multi-scale feature map through the decoding layer to obtain a second multi-scale feature; through the fusion layer, the first multi-scale feature map and the second multi-scale feature map are fused to obtain the semantic features of the sample image, determine the pixel points and contour edges of the target object according to the semantic features of the sample image, and output the second segmentation result according to the pixel points and contour edges.

[0245] In one embodiment, the above-mentioned image segmentation processing device also includes a guidance model training module 1201, which is used to obtain a sample image, and the sample image carries an annotation label; the sample image is input into the guidance model to be trained, and the target object in the sample image is subjected to image segmentation processing through the guidance model to be trained to obtain a sample segmentation result; the pixel error is determined according to the difference between the pixel information in the sample segmentation result and the pixel information of the pixel annotated by the annotation label; the edge error is determined according to the difference between the edge prediction result in the sample segmentation result and the edge label of the annotation label; according to the pixel error and the edge error, the model parameters of the guidance model are adjusted and iterative training is continued until the training stop condition is met to obtain a trained guidance model.

[0246] In one embodiment, the data acquisition module 1202 is further used to acquire an initial sample image; fill the initial sample image according to different types of image ratios; and normalize the filled sample image to obtain a pre-processed sample image.

[0247] In one embodiment, Fig.13 As shown, the above-mentioned image segmentation processing device 1200 also includes an image segmentation processing module 1212, which is used to obtain the image to be segmented, and input the image to be segmented into a trained image segmentation model; through the encoding layer of the image segmentation model, feature extraction and separable convolution processing are performed on the image to be segmented to obtain a first multi-scale feature map; through the decoding layer of the image segmentation model, deconvolution processing is performed on the first multi-scale feature map to obtain a second multi-scale feature; through the fusion layer of the image segmentation model, the first multi-scale feature map and the second multi-scale feature map are fused to obtain image semantic features; according to the image semantic features, the pixel points and contour edges of the target object are determined, and according to the pixel points and contour edges, the segmentation result of the image to be segmented is obtained.

[0248] In one embodiment, the image to be segmented is a continuous video frame, and the segmentation result of the image to be segmented includes the initial segmentation result corresponding to each video frame; the image segmentation processing module 1212 is also used to obtain the initial segmentation result of the current video frame, and the initial segmentation results corresponding to a preset number of video frames before the current video frame; according to the initial segmentation results corresponding to the preset number of video frames, the initial segmentation result of the current video frame is smoothed and filtered to obtain the target segmentation result corresponding to the current video frame.

[0249] In one embodiment, the segmentation result of the image to be segmented includes the area of ​​the target object and the background area; the image segmentation processing module 1212 is also used to obtain a virtual background; the background area in the image to be segmented is replaced with a virtual background image to generate an image after background replacement.

[0250] In one embodiment, the image to be segmented is a continuous video frame, and the target object is a person; the segmentation result of the image to be segmented includes the person area and the background area in each video frame; the image segmentation processing module 1212 is also used to replace the background area in each video frame with a virtual background image to generate continuous video frames after the background is replaced.

[0251] For the specific definition of the image segmentation processing device, please refer to the definition of the image segmentation processing method above, which will not be repeated here. Each module in the above-mentioned image segmentation processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0252] In one embodiment, a computer device is provided. The computer device may be a terminal or a server. The internal structure diagram thereof may be as follows: Fig.14 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image segmentation processing method is implemented.

[0253] Those skilled in the art will understand that Fig.14The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0254] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0255] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0256] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0257] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0258] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0259] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image segmentation processing method, characterized in that: The method comprises: Get a sample image; In each round of distillation training, at least two trained guidance models are used to perform image segmentation processing on the target object in the sample image, and the segmentation results corresponding to each of the guidance models are fused to obtain the first segmentation results corresponding to each of the guidance models; Performing image segmentation processing on the target object in the sample image by using the image segmentation model to be trained to obtain a second segmentation result; the model parameters of the image segmentation model are less than the model parameters of the guidance model; determining a distillation error according to a difference between a predicted probability of a pixel point in the sample image belonging to the target object in the second segmentation result and a predicted probability of a pixel point in the sample image belonging to the target object in the first segmentation result; determining an edge error according to a difference between an edge prediction result in the second segmentation result and an edge prediction result in the first segmentation result, wherein the edge prediction result in the second segmentation result includes a mask image obtained by segmenting the contour edge of the target object using the image segmentation model to be trained, and the edge prediction result in the first segmentation result includes a mask image obtained by segmenting the contour edge of the target object using the guidance model; Determine a pixel error according to a difference between pixel information in the second segmentation result and pixel information in the first segmentation result, wherein the pixel error includes a binary pixel value error of each pixel point and a pixel feature error corresponding to each pixel point; According to the distillation error, the edge error and the pixel error, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model.

2. The method according to claim 1, characterized in that The binary pixel value error is calculated based on the cross entropy loss, and the pixel feature error is calculated based on the affinity loss.

3. The method according to claim 1, characterized in that The sample images include unlabeled sample images and sample images carrying labeled labels; the method further includes: For an unlabeled sample image, performing the step of determining an edge error according to a difference between an edge prediction result in the second segmentation result and an edge prediction result in the first segmentation result, and the step of determining a pixel error according to a difference between pixel information in the second segmentation result and pixel information in the first segmentation result; For the labeled sample image, determining an edge error according to a difference between an edge prediction result in the second segmentation result and an edge label in the labeled label; For the labeled sample image, a pixel error is determined according to a difference between pixel information in the second segmentation result and pixel information of pixels labeled by the label.

4. The method according to claim 1, characterized in that: The step of adjusting the model parameters of the image segmentation model according to the distillation error, the edge error, and the pixel error and continuing the distillation training until a training stop condition is met to obtain a trained image segmentation model includes: The distillation error, the pixel error and the edge error are merged to obtain a target error value; According to the target error value, the model parameters of the image segmentation model are adjusted and the distillation training is continued until the training stop condition is met to obtain a trained image segmentation model.

5. The method according to claim 1, characterized in that The guiding model comprises a first guiding model and a second guiding model with different network structures; the image segmentation processing is performed on the target object in the sample image by using at least two trained guiding models respectively, and the segmentation results corresponding to each of the guiding models are merged to obtain the first segmentation results corresponding to each of the guiding models, including: Performing image segmentation processing on the target object in the sample image by using the trained first guiding model to obtain a segmentation result corresponding to the first guiding model; Performing image segmentation processing on the target object in the sample image by using the trained second guiding model to obtain a segmentation result corresponding to the second guiding model; The segmentation result corresponding to the first guidance model is fused with the segmentation result corresponding to the second guidance model to obtain a first segmentation result.

6. The method according to claim 1, characterized in that The image segmentation model includes a coding layer, a decoding layer and a fusion layer, and the coding layer and the decoding layer include a separable convolutional network; the image segmentation model to be trained is used to perform image segmentation processing on the target object in the sample image to obtain a second segmentation result, including: Inputting the sample image into the image segmentation model, performing feature extraction and separable convolution processing on the sample image through the encoding layer, and obtaining a first multi-scale feature map; Performing deconvolution processing on the first multi-scale feature map through the decoding layer to obtain a second multi-scale feature map; The first multi-scale feature map and the second multi-scale feature map are fused through the fusion layer to obtain a sample image semantic feature, the pixel points and the contour edge of the target object are determined according to the sample image semantic feature, and a second segmentation result is output according to the pixel points and the contour edge.

7. The method according to claim 1, characterized in that The training steps of the guidance model include: Acquire a sample image, wherein the sample image carries a label; Inputting the sample image into the guidance model to be trained, performing image segmentation processing on the target object in the sample image through the guidance model to be trained, and obtaining a sample segmentation result; Determine a pixel error according to a difference between pixel information in the sample segmentation result and pixel information of the pixel annotated by the annotated label; Determining an edge error according to a difference between an edge prediction result in the sample segmentation result and an edge label of the annotated label; According to the pixel error and the edge error, the model parameters of the guidance model are adjusted and iterative training is continued until a training stop condition is met to obtain a trained guidance model.

8. The method according to claim 1, characterized in that The obtaining of the sample image comprises: Get an initial sample image; Filling the initial sample image according to different types of image ratios; The padded sample image is normalized to obtain a preprocessed sample image.

9. The method according to any one of claims 1 to 8, characterized in that: After obtaining the trained image segmentation model, the method further includes: Acquire an image to be segmented, and input the image to be segmented into a trained image segmentation model; Performing feature extraction and separable convolution processing on the image to be segmented through the encoding layer of the image segmentation model to obtain a first multi-scale feature map; Performing deconvolution processing on the first multi-scale feature map through a decoding layer of the image segmentation model to obtain a second multi-scale feature map; fusing the first multi-scale feature map and the second multi-scale feature map through a fusion layer of the image segmentation model to obtain an image semantic feature; The pixel points and the contour edge of the target object are determined according to the image semantic features, and the segmentation result of the image to be segmented is obtained according to the pixel points and the contour edge.

10. The method according to claim 9, characterized in that The image to be segmented is a continuous video frame, and the segmentation result of the image to be segmented includes an initial segmentation result corresponding to each of the video frames; The method further comprises: Obtaining an initial segmentation result of a current video frame and initial segmentation results corresponding to a preset number of video frames before the current video frame; According to the initial segmentation results corresponding to the preset number of video frames, a smoothing filtering process is performed on the initial segmentation result of the current video frame to obtain a target segmentation result corresponding to the current video frame.

11. The method according to claim 9, characterized in that The segmentation result of the image to be segmented includes the area of ​​the target object and the background area; the method further includes: Get a virtual background; The background area in the image to be segmented is replaced with the virtual background image to generate an image after background replacement.

12. The method according to claim 11, characterized in that The image to be segmented is a continuous video frame, and the target object is a person; the segmentation result of the image to be segmented includes a person area and a background area in each of the video frames; The step of replacing the background area in the image to be segmented with the virtual background image to generate an image after background replacement includes: The background area in each of the video frames is replaced with the virtual background image to generate continuous video frames after background replacement.

13. An image segmentation processing device, characterized in that: The device comprises: A data acquisition module, used for acquiring sample images; A first segmentation processing module is used to perform image segmentation processing on the target object in the sample image respectively through at least two trained guidance models in each round of distillation training, and fuse the segmentation results corresponding to each of the guidance models to obtain a first segmentation result corresponding to each of the guidance models; A second segmentation processing module is used to perform image segmentation processing on the target object in the sample image through the image segmentation model to be trained to obtain a second segmentation result; the model parameters of the image segmentation model are less than the model parameters of the guidance model; an error determination module, configured to determine a distillation error according to a difference between a predicted probability of a pixel point in the sample image belonging to the target object in the second segmentation result and a predicted probability of a pixel point in the sample image belonging to the target object in the first segmentation result; determine an edge error according to a difference between an edge prediction result in the second segmentation result and an edge prediction result in the first segmentation result, wherein the edge prediction result in the second segmentation result includes a mask image obtained by segmenting a contour edge of the target object using the image segmentation model to be trained, and the edge prediction result in the first segmentation result includes a mask image obtained by segmenting a contour edge of the target object using the guidance model; determine a pixel error according to a difference between pixel information in the second segmentation result and pixel information in the first segmentation result, wherein the pixel error includes a binarized pixel value error of each pixel point and a pixel feature error corresponding to each pixel point; A parameter adjustment module is used to adjust the model parameters of the image segmentation model according to the distillation error, the edge error and the pixel error and continue the distillation training until the training stop condition is met to obtain a trained image segmentation model.

14. The device according to claim 13, characterized in that The binary pixel value error is calculated based on the cross entropy loss, and the pixel feature error is calculated based on the affinity loss.

15. The device according to claim 13, characterized in that The sample images include unlabeled sample images and sample images carrying labeled labels; The error determination module is further configured to, for an unlabeled sample image, perform the step of determining the edge error according to the difference between the edge prediction result in the second segmentation result and the edge prediction result in the first segmentation result, and the step of determining the pixel error according to the difference between the pixel information in the second segmentation result and the pixel information in the first segmentation result; For the labeled sample image, the edge error is determined based on the difference between the edge prediction result in the second segmentation result and the edge label in the labeled label; for the labeled sample image, the pixel error is determined based on the difference between the pixel information in the second segmentation result and the pixel information of the pixel labeled by the labeled label.

16. The device according to claim 13, characterized in that The parameter adjustment module is also used to fuse the distillation error, the pixel error and the edge error to obtain a target error value; according to the target error value, adjust the model parameters of the image segmentation model and continue distillation training until the training stop condition is met to obtain a trained image segmentation model.

17. The device according to claim 13, characterized in that The guidance model includes a first guidance model and a second guidance model having different network structures; The first segmentation processing module is also used to perform image segmentation processing on the target object in the sample image through the trained first guidance model to obtain the segmentation result corresponding to the first guidance model; to perform image segmentation processing on the target object in the sample image through the trained second guidance model to obtain the segmentation result corresponding to the second guidance model; and to fuse the segmentation result corresponding to the first guidance model with the segmentation result corresponding to the second guidance model to obtain the first segmentation result.

18. The device according to claim 13, characterized in that The image segmentation model includes a coding layer, a decoding layer and a fusion layer, and the coding layer and the decoding layer include a separable convolutional network; The second segmentation processing module is also used to input the sample image into the image segmentation model, perform feature extraction and separable convolution processing on the sample image through the encoding layer to obtain a first multi-scale feature map; perform deconvolution processing on the first multi-scale feature map through the decoding layer to obtain a second multi-scale feature map; and fuse the first multi-scale feature map and the second multi-scale feature map through the fusion layer to obtain semantic features of the sample image, determine the pixel points and contour edges of the target object according to the semantic features of the sample image, and output a second segmentation result according to the pixel points and the contour edges.

19. The device according to claim 13, characterized in that The device also includes: The guidance model training module is used to obtain a sample image, wherein the sample image carries an annotation label; input the sample image into the guidance model to be trained, and perform image segmentation processing on the target object in the sample image through the guidance model to be trained to obtain a sample segmentation result; determine the pixel error according to the difference between the pixel information in the sample segmentation result and the pixel information of the pixel annotated by the annotation label; determine the edge error according to the difference between the edge prediction result in the sample segmentation result and the edge label of the annotation label; adjust the model parameters of the guidance model according to the pixel error and the edge error and continue to perform iterative training until the training stop condition is met to obtain a trained guidance model.

20. The device according to claim 13, characterized in that The data acquisition module is further used to acquire an initial sample image; fill the initial sample image according to different types of image ratios; and normalize the filled sample image to obtain a preprocessed sample image.

21. The device according to any one of claims 13 to 20, characterized in that The device also includes: The image segmentation processing module is used to obtain an image to be segmented, input the image to be segmented into a trained image segmentation model; perform feature extraction and separable convolution processing on the image to be segmented through the encoding layer of the image segmentation model to obtain a first multi-scale feature map; perform deconvolution processing on the first multi-scale feature map through the decoding layer of the image segmentation model to obtain a second multi-scale feature map; fuse the first multi-scale feature map and the second multi-scale feature map through the fusion layer of the image segmentation model to obtain image semantic features; determine the pixel points and contour edges of the target object according to the image semantic features, and obtain the segmentation result of the image to be segmented according to the pixel points and the contour edges.

22. The device according to claim 21, characterized in that The image to be segmented is a continuous video frame, and the segmentation result of the image to be segmented includes an initial segmentation result corresponding to each of the video frames; The image segmentation processing module is also used to obtain the initial segmentation result of the current video frame and the initial segmentation results corresponding to a preset number of video frames before the current video frame; based on the initial segmentation results corresponding to the preset number of video frames, the initial segmentation result of the current video frame is smoothed and filtered to obtain the target segmentation result corresponding to the current video frame.

23. The device according to claim 21, characterized in that The segmentation result of the image to be segmented includes the area of ​​the target object and the background area; The image segmentation processing module is also used to obtain a virtual background; replace the background area in the image to be segmented with the virtual background image to generate an image after the background is replaced.

24. The device according to claim 23, characterized in that The image to be segmented is a continuous video frame, and the target object is a person; the segmentation result of the image to be segmented includes a person area and a background area in each of the video frames; The image segmentation processing module is further used to replace the background area in each of the video frames with the virtual background image to generate continuous video frames after the background is replaced.

25. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

26. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

27. A computer program product, comprising computer instructions, wherein the computer instructions are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, the steps of the method described in any one of claims 1 to 12 are implemented.

Citation Information

Patent Citations

  • Character image automatic segmentation method based on deep learning and information data processing terminal

    CN110008832A

  • Deep learning semantic segmentation model compression method for embedded mobile terminal

    CN110059740A

  • Real-time semantic segmentation method based on sequence knowledge distillation

    CN111723812A