Model training method, image segmentation method and related device

Abstract

The invention provides a model training method, an image segmentation method and a related device. The method comprises the following steps: carrying out image segmentation on a sample image through a model to obtain a first mask image; performing difference calculation on the first mask image and an actual mask image of the sample image to obtain a first loss function; performing key point detection on the first mask pattern to obtain a first key point; performing difference calculation on the first key point and an actual key point of the sample image to obtain a second loss function; and training the model based on the first loss function and the second loss function. According to the image segmentation method and device, the key points in the to-be-segmented image can be detected, so that the image segmentation accuracy of the model is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, an image segmentation method, and related devices. Background Art

[0002] With the advancement of computer technology, image segmentation has become increasingly widely used. Image segmentation is the process of separating a portion of an image or video, with the goal of accurately distinguishing a target object or person from the background at the pixel level. Related technologies use image segmentation models to separate an image into foreground and background, then remove the background to retain the foreground. However, when dealing with complex scenes and rapidly changing image edges, the blurry boundary between the target and background can affect the image segmentation performance and accuracy.

[0003] Therefore, how to efficiently and accurately segment objects in segmented images has become one of the hot research issues in the field of computer technology. Summary of the Invention

[0004] The embodiments of the present application provide a model training method, an image segmentation method and related devices, which can improve the accuracy of image segmentation performed by the model.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides a model training method, which includes:

[0007] Performing image segmentation on the sample image through the model to obtain a first mask image;

[0008] performing a difference calculation between the first mask image and an actual mask image of the sample image to obtain a first loss function;

[0009] Performing key point detection on the first mask image to obtain a first key point;

[0010] performing a difference calculation on the first key point and the actual key point of the sample image to obtain a second loss function;

[0011] The model is trained based on the first loss function and the second loss function.

[0012] The present invention provides an image segmentation method, which includes:

[0013] Performing image segmentation on the image to be segmented using a model to obtain a second mask image corresponding to the target object in the image to be segmented, wherein the model is trained based on the model training method provided in an embodiment of the present application;

[0014] A preset operation is performed on the second mask image and the image to be segmented to obtain an image of the target object in the image to be segmented.

[0015] The present invention provides a model training device, comprising:

[0016] A processing module, configured to perform image segmentation on the sample image using an image segmentation model to obtain a first mask image;

[0017] A determination module is used to calculate the difference between the first mask image and the actual mask image of the sample image to obtain a first loss function; perform key point detection on the first mask image to obtain a first key point; calculate the difference between the first key point and the actual key point of the sample image to obtain a second loss function; and train the image segmentation model based on the first loss function and the second loss function.

[0018] The present invention provides an image segmentation device, comprising:

[0019] A segmentation module is used to perform image segmentation on the image to be segmented through a model to obtain a second mask image corresponding to the target object in the image to be segmented, wherein the model is trained based on the model training method provided in an embodiment of the present application; the second mask image is subjected to a preset operation with the image to be segmented to obtain an image of the target object in the image to be segmented.

[0020] An embodiment of the present application provides an electronic device, comprising:

[0021] a memory for storing computer-executable instructions or computer programs;

[0022] The processor is used to implement the model training method or image segmentation method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0023] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which is used to implement the model training method or image segmentation method provided in the embodiment of the present application when executed by a processor.

[0024] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the model training method or image segmentation method provided in the embodiment of the present application is implemented.

[0025] The embodiments of the present application have the following beneficial effects:

[0026] The model is used to perform image segmentation on the sample image to obtain a first mask image, thereby achieving segmentation of the target area in the sample image. The difference between the first mask image and the actual mask image of the sample image is calculated to obtain the image difference. The first loss function is determined based on the image difference. The first loss function can measure the accuracy of image segmentation. Key point detection is performed on the first mask image to obtain the first predicted key point. The difference between the first predicted key point and the actual key point of the sample image is calculated to obtain the second loss function. The second loss function can measure the completeness of the model's segmentation of the target area in the sample image. Training the model based on the second loss function can enable the model to better obtain the location information of the key points of the target area. The model is trained based on the first loss function and the second loss function. The training of the model combines multiple loss functions. The trained model can achieve more accurate segmentation of the sample image by detecting the key points of the image to be segmented. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of an application mode of the model training method provided in an embodiment of the present application;

[0028] Figure 2 This is a schematic diagram of the structure of the server provided in the embodiment of the present application;

[0029] Figure 3A This is a first flow chart of the model training method provided in an embodiment of the present application;

[0030] Figure 3B This is a second flow chart of the model training method provided in an embodiment of the present application;

[0031] Figure 3C This is a third flow chart of the model training method provided in an embodiment of the present application;

[0032] Figure 3D This is a schematic diagram of the first process of the image segmentation method provided in an embodiment of the present application;

[0033] Figure 4 This is a fourth flow chart of the model training method provided in an embodiment of the present application;

[0034] Figure 5 It is a structural diagram of the image segmentation model provided in the embodiment of the present application.

[0035] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of distinction between the advantages and disadvantages of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0037] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0038] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0039] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0040] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0041] In the embodiments of this application, the collection and processing of relevant data (for example, sample images) should be carried out in strict accordance with the requirements of relevant laws and regulations when applied in practice, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0042] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0043] 1) Image segmentation: It is a technology and process that divides an image into several specific regions with unique properties and proposes objects of interest. These regions have similar characteristics such as color, brightness, texture, etc., while there are significant differences between regions.

[0044] 2) Portrait Segmentation: This refers to the process of separating a person’s portrait from their background using computer vision technology. It is widely used in image editing, virtual reality, augmented reality, video editing, and social media.

[0045] 3) Convolutional neural network: It is a network structure for image feature extraction. It uses convolution layers to automatically extract local features in images and performs feature dimensionality reduction through pooling layers. In the embodiment of the present application, a convolutional neural network is used as a feature extraction network for image features, and the image feature extraction operation is implemented through MobileNetV3. MobileNetV3 is a lightweight convolutional neural network that uses depth-separable convolution to reduce the number of parameters and computational complexity. It usually includes a downsampling stage to reduce the spatial size of the image while increasing the depth (number of channels) of the feature. The output is a set of feature maps that contain hierarchical features learned from the input image.

[0046] 4) Segmentation network: It is a model that classifies each pixel in the image into different categories to distinguish different objects or areas. The segmentation network usually includes an encoder and a decoder. The encoder is used to extract image features, and the decoder is used to map these features back to the original image space to generate a segmentation map. In the embodiment of the present application, a U-net network (hereinafter referred to as the U-net network) is used as the image segmentation network. The U-net network is a convolutional neural network structure. Downsampling is a method of reducing the image size, which can reduce computational complexity and extract higher-level features, which helps to capture the contextual information of the image. The second half is mainly responsible for upsampling and feature fusion, and finally outputs the segmentation map.

[0047] 5) Upsampling: This is the process of increasing the image resolution or expanding the spatial size of the feature map. The purpose of upsampling is to restore the image resolution and generate a segmentation map with the same resolution as the input image. Upsampling is used in the decoder to map the feature map back to the spatial size of the original image, thereby generating accurate segmentation results.

[0048] 6) Downsampling: refers to the operation of reducing the resolution of an image or reducing the spatial size of a feature map during image processing. The purpose of downsampling is to reduce the amount of computation, extract high-level features of the image, and help reduce overfitting. It is usually used in the encoder part to extract abstract features of the image.

[0049] 7) Single Stage Headless Face Detector (SSH): It is a deep learning-based face detection algorithm that uses a single-stage detection framework to directly predict the bounding box and category probability of the face on the feature map.

[0050] 8) 68-point facial key point detection: It is a technology in computer vision that uses algorithms to determine the precise location of specific feature points in facial images. These key points usually include a series of specific points on parts such as the eyes, nose, mouth, and facial contours.

[0051] 9) Mask: A mask image is a binary image. Each pixel of the mask image contains only two possible values, usually 0 and 1, or in some cases black and white. The mask image is used to control the visibility of the foreground and background during image synthesis. A pixel value of 1 (or white) represents the foreground area, and the pixels in the foreground area will be retained during the synthesis process; a pixel value of 0 (or black) represents the background area, and the pixels in the foreground area will be ignored or set to transparent during the synthesis process. In an embodiment of the present application, the mask can be used to multiply the original image to obtain a portrait image.

[0052] In related technologies, an image segmentation model is used to segment an image, and a segmentation map is obtained by removing the background and retaining the foreground. However, in complex scenes and when image edges change rapidly, the boundary between the foreground and the background is blurred, and accurate image segmentation cannot be achieved, resulting in low image segmentation accuracy.

[0053] The embodiments of the present application provide a model training method, an image segmentation method, a model training device, an image segmentation device, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of image segmentation performed by the model by detecting key points in the image to be segmented.

[0054] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. The devices provided in the embodiments of this application can be implemented as various types of terminals, such as laptops, tablet computers, desktop computers, set-top boxes, smartphones, smart speakers, smart watches, smart TVs, and in-vehicle terminals. They can also be implemented as servers. The following describes exemplary applications when the devices are implemented as terminals or servers.

[0055] See also Figure 1 , Figure 1 This is a schematic diagram of the application mode of the model training method provided in the embodiment of the present application, which is used to support a model training application. Figure 1The server 200, network 300, terminal device 400 and database 500 are involved. The terminal device 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0056] In some embodiments, the user may be a technician in this field, the server 200 is a server for performing sample image segmentation, the terminal device 400 is a terminal operated by the user, an application capable of displaying the segmented image is installed in the terminal device 400, and the database 500 stores the sample image and the segmented image.

[0057] For example, the terminal device 400 is used to receive sample images preconfigured by the user. The terminal device 400 sends the sample images to the server 200 via the network 300. The server 200 randomly selects an image from the sample images as the input of the image segmentation model to be trained. The image segmentation model is used to perform image segmentation on the input sample image to obtain a segmented mask image. The segmented mask image is detected for key points to obtain predicted key points. The loss function of the image segmentation model is determined based on the difference between the segmented mask image and the predicted key points and the actual mask image and the actual key points. The image segmentation model is trained using the loss function to obtain a trained image segmentation model. The trained image segmentation model is stored in the server 200 or the terminal device 400 and is used to segment the image to be segmented to obtain a facial image in the image to be segmented. The server 200 or the terminal device 400 can execute related applications based on the segmented facial image.

[0058] In some embodiments, the model training method of the embodiments of the present application can also be applied in the following application scenarios:

[0059] 1. Application scenarios of video communication: The image segmentation model obtained through the model training method of the embodiment of this application is used in the scenario of video communication, specifically for performing image segmentation processing on video images, extracting specific character images from video images, realizing character and background segmentation, and then performing operations such as blocking and replacing the background image.

[0060] 2. Special effects production scenarios for movies or games: The image segmentation model obtained in this application is used for image segmentation in the special effects production of movies or games. For example, the image segmentation model of this application is used to segment multiple frames of movie or game images, segment the target object and the background, and perform special effects transformation and other processing based on the segmentation results to complete different special effects scenes.

[0061] 3. Scenario of video playback occlusion setting: The trained image segmentation model obtained through the model training method of the embodiment of the present application is used in the scenario of video playback occlusion setting, and the image edges of the video playback are quickly and accurately segmented to improve the user's viewing experience.

[0062] The embodiments of the present application can be implemented using database technology. A database, in short, can be considered an electronic filing cabinet that stores electronic files, allowing users to add, query, update, and delete data in these files. A "database" is a collection of data that is stored together in a specific manner, can be shared by multiple users, has minimal redundancy, and is independent of applications.

[0063] A database management system (DBMS) is a computer software system designed for managing databases, typically providing basic functions such as storage, retrieval, security, and backup. DBMSs can be categorized by the database model they support, such as relational or XML (Extensible Markup Language); by the type of computer they support, such as server clusters or mobile phones; by the query language they use, such as SQL or XQuery; by performance priorities, such as maximum scale or maximum speed; or by other classification methods. Regardless of the classification method used, some DBMSs are cross-category, for example, supporting multiple query languages ​​simultaneously.

[0064] See also Figure 2 , Figure 2 is a schematic diagram of the structure of the server provided in the embodiment of the present application, Figure 2 The server 200 shown includes: at least one processor 410, a memory 450 and at least one network interface 420. The various components in the server 200 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0065] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0066] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0067] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0068] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0069] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0070] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wireless LAN (Wi-Fi), and Universal Serial Bus (USB).

[0071] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 The model training device 455 stored in the memory 450 is shown. The model training device 455 can be software in the form of a program or plug-in, and includes the following software modules: a processing module 4551, a determination module 4552, and a segmentation module 4553. These modules are logical and can be arbitrarily combined or further divided according to the functions implemented. This does not exclude implementations that only include the segmentation module 4553 or only include the processing module 4551 and the determination module 4552. The functions of each module will be described below.

[0072] In some embodiments, the terminal or server can implement the model training method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in an operating system; it can be a local (Native) application (APPlication, APP); it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0073] The training method of the image segmentation model provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the electronic device provided in the embodiment of the present application.

[0074] The following describes the training method for the image segmentation model provided in the embodiments of the present application. As previously mentioned, the electronic device implementing the device management method of the embodiments of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0075] See also Figure 3A , Figure 3A This is a first flow chart of the model training method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained, Figure 3A The executive body is Figure 1 Server 200 in.

[0076] In step 301, the sample image is segmented using a model to obtain a first mask image.

[0077] For example, the above model can be an image segmentation model, which is used to segment the foreground object in the image to obtain a segmented image of the foreground object. The sample image is input into the model and image segmentation is performed through the segmentation network, and the output image segmentation result is used as the first mask image. The image segmentation model can be a convolutional neural network or a generative adversarial network. The model trained in the embodiments of the present application is illustrated using a convolutional neural network as an example. The model training method provided in the embodiments of the present application is also applicable to the training of other types of image segmentation models.

[0078] The mask image is a binary image. Each pixel of the mask image contains only two possible values, usually 0 and 1, or in some cases black and white. The mask image is used to control the visibility of the foreground and background during image synthesis. A pixel value of 1 (or white) represents the foreground area, and the pixels in the foreground area will be retained during the synthesis process; a pixel value of 0 (or black) represents the background area, and the pixels in the foreground area will be ignored or set to transparent during the synthesis process. Each pixel value of the first mask image is multiplied by the pixel value of the same position in the foreground image to determine the pixels that should be included in the final portrait image, resulting in a new image that only contains the desired foreground content, that is, a new image in which the background is completely removed and only contains the portrait of the person.

[0079] In some embodiments, before performing image segmentation on the sample image through the model to obtain the first mask image, the sample image is obtained by at least one of the following processing: obtaining a preconfigured original image as a sample image; adding noise to the original image to obtain a sample image; erasing at least part of the original image to obtain a sample image; obtaining an image with the same background color as the original image as the sample image.

[0080] For example, the sample images in the model are randomly input from the sample dataset. Through the above-mentioned multiple processing methods, the sample dataset contains four types of sample images, which are explained below:

[0081] The first category of sample images are pre-configured original images, which are pre-collected from public datasets for image segmentation.

[0082] The second type of sample image is an image with random noise added around the original image using a modular component. A modular component is used to create and manipulate masks during image processing. A mask is a pixel-level map that indicates which parts of an image should be modified or retained. The second type of sample image can be obtained by, for example, adding a random texture to the original image or adding a preset shape of random size (for example, a solid or hollow circle).

[0083] The third type of sample image is obtained by erasing the original image. For example, the original image is divided into 9*9 blocks, and 20%-30% of the small blocks are randomly erased (to prevent the erased part from having no image), and the missing image is obtained as the third type of sample image.

[0084] The fourth type of sample image is an image with the same background color as the original image, for example, an image with the same background color as the clothes of the portrait in the original image is added.

[0085] In the embodiments of the present application, sample images are obtained in a variety of ways, which increases the diversity of the data set, helps the model learn different backgrounds and scenes, and helps the model learn to perform accurate segmentation even when some information is missing, making the trained model more suitable for actual scenarios.

[0086] In some embodiments, see Figure 3B , Figure 3B This is a second flow chart of the model training method provided in an embodiment of the present application; Figure 3A Step 301 can be performed by Figure 3B Steps 3011 to 3012 in the embodiment are implemented as described below.

[0087] In step 3011, feature extraction is performed on the sample image through the model to obtain sample image features, down-sampling processing is performed on the sample image features to obtain down-sampled features, and up-sampling processing is performed on the down-sampled features to obtain up-sampled features.

[0088] For example, feature extraction is performed on the sample image through the feature extraction network in the model to obtain sample image features. In the embodiment of the present application, the feature extraction network is a lightweight neural network (MobileNetV3 network). The MobileNetV3 network uses deep separable convolution to reduce the number of parameters and computational complexity. The feature extraction network includes a downsampling stage. The output of the feature extraction network is a set of feature maps. The output sample image features include hierarchical features learned from the input image.

[0089] In some embodiments, see Figure 5 , Figure 5 5 is a schematic diagram of the structure of the image segmentation model provided in an embodiment of the present application; the image segmentation model includes a feature extraction network 502 and a segmentation network 503. The segmentation network 503 includes a downsampling module 5031 and an upsampling module 5032. A sample image 501 is input into the image segmentation model, and the feature extraction network 502 extracts features from the sample image to obtain sample image features.

[0090] For example, the sample image features are segmented by the segmentation network in the model. In an embodiment of the present application, the segmentation network classifies each pixel in the image into different categories to distinguish different objects or areas, and usually includes an encoder and a decoder. In an embodiment of the present application, a U-Net network is used as a segmentation network. The U-Net network is a convolutional neural network structure, including a downsampling module and an upsampling module. The downsampling processing performed by the downsampling module in the U-Net network refers to the operation of reducing the resolution of the image or reducing the spatial size of the feature map during the image processing process. The downsampling processing can reduce the computational complexity and extract higher-level features, which helps to capture the contextual information of the image. The sample image features are downsampled by the segmentation network to obtain downsampled features.

[0091] See also Figure 5 , the sample image features are used as the input of the segmentation network 503, and the downsampling process is performed through the downsampling module 5031. The downsampling module 5031 includes four downsampling convolution layers: downsampling layer 50311, downsampling layer 50312, downsampling layer 50313 and downsampling layer 50314. The number of downsampling can be set according to the size of the mask image so that the downsampling mask label is consistent with the size of the mask image. In the embodiment of the present application, upsampling and downsampling each use 4 layers of convolution and activation function. The convolution of the downsampling module 5031 is 3*3 convolution with a step size of 2. As a downsampling layer, the convolution kernel size is 64, and the downsampled features are obtained after convolution. The step size is the stride size of the convolution kernel sliding on the input feature map, which determines the distance the convolution kernel slides each time. In the embodiment of the present application, the downsampling module 5031 includes 4 downsampling layers, which is only used as an example. In the specific implementation, the number of downsampling layers can be set according to the requirements of the actual application scenario, and the embodiment of the present application is not limited to this.

[0092] For example, the upsampling process performed by the upsampling module is the process of increasing the image resolution or expanding the spatial size of the feature map. The purpose of the upsampling process is to restore the image resolution and generate a segmentation map with the same resolution as the input image. The downsampled features are upsampled by the segmentation network in the image segmentation model. During the upsampling process, the size of the feature map is increased so that it can be fused with the corresponding feature map during the downsampling process to obtain the upsampled features.

[0093] The downsampled features obtained by the downsampling process are input into the upsampling module 5032 for upsampling. The upsampling module 5032 includes four upsampling convolution layers: upsampling layer 50321, upsampling layer 50322, upsampling layer 50323, and upsampling layer 50324. The convolutions in the upsampling layers are all 3*3 convolutions with a step size of 1 / 2 and a convolution kernel size of 64. The upsampling module 5032 is used to restore the original size of the image and fuse it with the downsampled feature map to obtain the upsampled features. In the embodiment of the present application, the upsampling module 5032 includes four upsampling layers, which is only used as an example. In the specific implementation, the number of upsampling layers can be set according to the requirements of the actual application scenario, and the embodiment of the present application is not limited to this.

[0094] Continue to see Figure 3B In step 3012, a nonlinear transformation is performed on the upsampled features, and the results of the nonlinear transformation are classified to obtain a first mask image.

[0095] For example, a nonlinear transformation is a mathematical function or process in which the ratio of the change in each output value to the change in its corresponding input value is not a constant, and a change in the input value causes the output value to change at a different rate. The nonlinear transformation of the upsampled features is achieved by a nonlinear function in the classification network, and a convolutional layer is used to classify the upsampled and nonlinearly transformed features of the result of the nonlinear transformation, and a first mask image with the same size as the input sample image is output. In the embodiment of the present application, the nonlinear function is an activation function (Parametric Rectified Linear Unit, PReLU), hereinafter referred to as PreLU activation function. The PreLU activation function is an activation function improved based on the adaptive activation function. The slope of the negative value is controlled by a learnable parameter in the PreLU activation function, so that the model can adaptively adjust the response of the negative value part during the training process.

[0096] In the embodiment of the present application, by performing feature extraction on the sample image, the image features obtained help to improve the segmentation accuracy. Through upsampling and downsampling processing, the image features can be effectively extracted and restored, the context information of the image can be obtained, and an accurate mask map can be generated.

[0097] Continue to refer Figure 3A In step 302, a difference calculation is performed between the first mask image and the actual mask image of the sample image to obtain a first loss function.

[0098] For example, a difference calculation is performed between the first mask image and the actual mask image of the sample image to obtain a cross entropy loss between the mask images. The first loss function is determined based on the cross entropy loss, which can be implemented by formula (1), as described in detail below.

[0099] Loss1=BCE(unet(x),mask) (1)

[0100] Here, Loss1 represents the first loss function, BCE represents binary cross-entropy, unet represents the segmentation network, x represents the input sample image, unet(x) represents the first mask output by the segmentation network, and mask represents the actual sample mask. The binary cross-entropy (BCE) loss function is a common method for evaluating the performance of binary classification models, used to measure the degree of inconsistency between the model's predictions and the true labels.

[0101] In step 303, key point detection is performed on the first mask image to obtain a first key point.

[0102] For example, a facial recognition network is introduced to evaluate the model's output. The first mask image is used as input to the facial recognition network to detect facial key points and obtain the first key points. Facial key point detection is to describe the facial structure and features of a person in detail by identifying and locating 68 specific points in the face image. The locations of the key points include the main facial features: eyes, nose, mouth, and facial contours. The label of each first predicted key point is a score that is used to indicate whether a facial portrait exists in a certain area in the first mask image, or whether the facial portrait segmentation output is correct. During the training process of the image segmentation model, the facial recognition network can exist independently of the structure of the image segmentation model.

[0103] See also Figure 5 , the key point detection in the first mask image output by the upsampling layer 50324 is realized through the facial recognition network 505. The facial recognition network 505 is realized by adding a single-point headless face detector. It is a network with fixed pre-trained parameters. The facial recognition network 505 directly predicts the position and key points of the face from the first mask image, and outputs the score of each key point in the first mask image. The score represents the category of the key point, for example: the key point is the key point located at the left eye, right eye, and nose.

[0104] In the embodiments of the present application, facial key point detection can accurately identify and locate facial features, verify whether the output of image segmentation is accurate, evaluate the effectiveness of the image segmentation model, and improve its applicability in practical applications.

[0105] In step 304, a difference calculation is performed on the first key point and the actual key point of the sample image to obtain a second loss function.

[0106] For example, the position difference between the first key point and the actual key point in the sample image is calculated, the preconfigured key point type is 17 edge key points of the face, and the second loss function is determined based on the position difference of the preconfigured key point type.

[0107] In some embodiments, step 304 can be implemented by: determining the type of the first key point; when the type of the first key point is a preconfigured type, performing a difference calculation on the first key point and the actual key point to obtain a first difference; based on the number of the first key points, performing a preset operation on the first difference to obtain a second difference, and determining a second loss function based on the second difference.

[0108] For example, the first keypoint type includes edge keypoints and non-edge keypoints, and the preconfigured type is edge keypoints. Edge keypoints are keypoints located on the contour lines of the face, for example, keypoints located at the highest point of the cheekbone, the edge of the jawline, and the edge of the hairline. Conversely, non-edge keypoints are keypoints located within facial features, for example, keypoints located at the center of the eyes, the tip and bottom of the nose, and the center and corners of the mouth.

[0109] In the case where the type of the first key point is a preconfigured type, that is, the difference between the edge key points and the actual key points in the first key point is calculated to obtain the position difference. According to the number of preconfigured type key points, the position difference is averaged to obtain the average position difference value, and the second loss function is determined based on the average difference value. This can be achieved by formula (2), which is described in detail below.

[0110] Lmds=1if(lmd(i)-label[i])>μ,i∈[eyes, mouth, nose]

[0111] Lmds=1 / N(lmd(i)-label[i]),i∈others(2)

[0112] Among them, Lmds represents the second loss function, 1if is used to determine whether the condition is met, lmd(i) represents the set of facial key points, i represents the region, each i includes multiple key points, when i∈[eyes, mouth, nose], the value of i is 6 eye key points, 5 nose key points and 12 mouth key points, when i∈other, the value of i is the remaining key points of the face except eyes, mouth and nose, a total of 45, and a total of 68 facial key points, μ represents the threshold of the preset value, which is 0.8, N is the format of the preconfigured type key point, which represents the total number of key points in other areas, which is 17, and label[i] represents the combination of actual key points.

[0113] In this embodiment, a second loss function is determined by the facial recognition network to improve the trained model's accuracy in locating facial key points and enhance the quality of facial image segmentation. The average position difference value is selectively calculated based on edge key points, reducing computational overhead and improving the efficiency of the loss function calculation.

[0114] In some embodiments, key point detection is implemented through a facial recognition network. After determining the second loss function, the facial recognition network is also trained by the following method: determining the predicted label value corresponding to the first key point; training the facial recognition network through the second loss function, and when there is a predicted label value greater than the label value threshold, deleting the actual key point corresponding to the second key point greater than the label value threshold from the sample image; when there are remaining first key points in the sample image, continuing to train the facial recognition network based on the remaining first key points, the second key point refers to the first key point with a predicted label value greater than the label value threshold; when all actual key points in the sample image are deleted, it is determined that the training of the facial recognition network is completed.

[0115] For example, the predicted label value corresponding to each first predicted key point obtained by key point detection is determined. Each label value can reflect whether there is a facial image in the area where the predicted key point is located in the first mask image or whether the facial mask image output is accurate. The facial recognition network is iteratively trained using the second loss function. When the predicted label value is greater than the label value threshold, the actual key point corresponding to the first predicted key point with a predicted label value greater than the label value threshold is removed from the sample image. The actual key point is at the same position as the first predicted key point in the sample image. When all the actual key points in the sample image are removed, a trained facial recognition network is obtained. Each area of ​​the mask image contains multiple key points, and a threshold for the key point is set. When the key point is any of the key points of the eyes, nose, and mouth, if the threshold of the first predicted key point is greater than the preset threshold of 0.8, the training of the key point is stopped, and the value is assigned to 1, and the training of other preconfigured key points is continued.

[0116] In an embodiment of the present application, by removing the actual key points corresponding to the first predicted key points whose predicted label values ​​are greater than the label value threshold according to the predicted label value, redundancy in training can be reduced, training speed can be accelerated and efficiency can be improved, thereby improving the training effect of the facial recognition network.

[0117] In some embodiments, the training of the image segmentation model is performed iteratively. During the iterative process of the image segmentation model, the facial recognition network is also synchronously trained iteratively, thereby improving the training effect of the image segmentation model so that the trained image segmentation model can obtain more accurate segmentation results.

[0118] In step 305 , the model is trained based on the first loss function and the second loss function.

[0119] For example, the weighted sum of the first loss function and the second loss function can be used as the total loss function to train the model to obtain a trained model.

[0120] In some embodiments, step 305 can be implemented by performing classification processing on the upsampled features to obtain a predicted category for the sample image, performing a calculation based on the predicted category and the actual category to obtain a classification loss function. The first loss function, the second loss function, and the classification loss function are combined to obtain a third loss function. The model is trained using the third loss function.

[0121] For example, in addition to the first loss function and the second loss function, a sub-loss function can be determined for other data output by the model, and the weighted sum of multiple loss functions is used as the total loss function to train the image segmentation model. The upsampled features output by the feature extraction network of the model are classified through two classification networks to obtain the predicted category of the sample image, and the corresponding sub-loss function is obtained based on the calculation of the predicted category and the actual category. The third loss function of the model is determined based on the combination of the first loss function, the second loss function and the classification sub-loss function of the model. The model is trained by the third loss function to obtain a trained model for performing segmentation operations on the sample image.

[0122] For example, the first loss function, the second loss function, the first classification loss function and the second classification loss function are linearly added to obtain the total loss function of the model as the third loss function. The determined total loss function is sent to each layer of the model network, and the image segmentation model is trained to obtain the trained image segmentation model. The determination of the total loss function can be achieved by formula (3), which is explained in detail below.

[0123] Total Loss=5Loss1+Loss2+Loss3+2Lmds (3)

[0124] Among them, total Loss is the third loss function of the image segmentation model, Loss1 is the first loss function, Loss2 and Loss3 are the first classification loss function and the second classification loss function, and Lmds is the second loss function.

[0125] In some embodiments, before executing step 305, the predicted category of the sample image and the predicted category of each pixel point are also determined by the following method: the upsampled features are classified to obtain the predicted category of the sample image; based on the predicted category of the sample image and the upsampled features, the pixels in the sample image are classified to obtain the predicted category of each pixel point corresponding to each pixel point in the sample image.

[0126] For example, for the upsampling features in the upsampling process, a first classification network is introduced to classify the features output by the first two layers of upsampling, and the output features of the two layers of upsampling are used as the input of the first classification network to obtain the predicted category of the sample image output by the first classification network. The predicted category of the sample image is the probability of the four sample image categories corresponding to the output of the last layer of the first classification network, and the sample image category corresponding to the highest probability value is used as the category of the upsampling feature. A second classification network is introduced, and the predicted category of the sample image is used as the input of the second classification network for classification processing to obtain the predicted category of each pixel in the sample image. The predicted category of each pixel is used as a weight and weighted to the parameters of the penultimate layer of upsampling to fine-tune the parameters of the segmentation network. During the training process of the image segmentation model, the first classification network and the second classification network can exist independently of the structure of the image segmentation model.

[0127] In some embodiments, see Figure 3C , Figure 3C This is a third flow chart of the model training method provided in the embodiment of the present application; the classification loss function includes: a first classification loss function and a second classification loss function, which can be performed by Figure 3C Steps 3051 to 3053 in are implemented as described below.

[0128] In step 3051, a difference calculation is performed based on the predicted category of the sample image and the actual category of the sample image feature to obtain a first classification loss function.

[0129] Here, the predicted category of the sample image represents the image type corresponding to the sample image.

[0130] For example, the image type is used to distinguish images obtained through different processing methods. Each image type corresponds to an image acquisition method. The embodiment of the present application involves four types of sample images. The cross entropy loss between the predicted category of the sample image and the actual category of the sample image feature is calculated to obtain the first classification loss function, which can be represented by Loss2=BCE(score1), where Loss2 is the first classification loss function of the model classification processing, BCE represents the binary cross entropy, and score1 represents the predicted category of the sample image output by the first classification network.

[0131] In step 3052, based on the predicted category of the sample image and the up-sampled features, the pixels in the sample image are classified to obtain the predicted category of each pixel in the sample image.

[0132] For example, based on the predicted category of the sample image and the upsampling features, the pixels in the sample image are classified to obtain the predicted category of each pixel in the sample image. The categories corresponding to the pixels include: the pixels belong to the sample image features and do not belong to the sample image features.

[0133] In step 3053, a difference calculation is performed based on the predicted category of each pixel and the actual category of the corresponding pixel to obtain a second classification loss function.

[0134] Here, the predicted category of each pixel represents the type of pixel.

[0135] For example, the cross entropy loss between the predicted category of each pixel and the actual category of the pixel in the sample image feature is calculated to obtain the second classification loss function, which can be represented by Loss3=BCE(score2), where Loss3 is the second classification loss function of the model classification processing, BCE represents the binary cross entropy, and score2 represents the predicted category of each pixel output by the second classification network.

[0136] In some embodiments, the present application also provides an image segmentation method, see Figure 3D , Figure 3D This is a first flow chart of the image segmentation method provided in the embodiment of the present application, which will be combined with Figure 3D Steps 306 to 307 are shown for explanation. Figure 3D The executive body is Figure 1 The server 200 or terminal device 400 in.

[0137] In step 306 , the image to be segmented is segmented using the model to obtain a second mask image corresponding to the target object in the image to be segmented.

[0138] Here, the model is trained based on the above model training method.

[0139] For example, after obtaining the trained image segmentation model, the image segmentation model is stored in a server or terminal device serving as the execution subject, the image to be segmented is input into the image segmentation model, the features of the image to be segmented are extracted, and the features of the image to be segmented are segmented to obtain a second mask image corresponding to the target object in the image to be segmented. The target object in the image to be segmented may be a face, and the second mask image is represented as a binary image.

[0140] In step 307 , a preset operation is performed on the second mask image and the image to be segmented to obtain an image of the target object in the image to be segmented.

[0141] For example, each pixel in the second mask image contains only two possible values. The second mask image is multiplied by the image to be segmented to obtain a target object image that only contains the image to be segmented. The target object image can be a facial portrait and does not contain the background, thereby realizing facial image segmentation of the sample image.

[0142] In an embodiment of the present application, feature extraction is performed on a sample image to obtain sample image features while maintaining low computational complexity. The sample image features are segmented through a model, and context information is retained during downsampling and upsampling to improve segmentation accuracy. Introducing a facial recognition network to perform key point detection on the first mask image obtained by segmentation can help the model locate facial features more accurately. Removing the actual key points corresponding to the first predicted key points whose predicted label values ​​are greater than the label value threshold based on the predicted label value can reduce the burden in training, speed up training, improve efficiency, and improve the training effect of the facial recognition network. Two classification networks are introduced to perform near-classification processing on the upsampled features to obtain the first and second scores of the image category and the pixel category, and further determine two classification loss functions. The third loss function of the model is determined by weighted summation of the first loss function determined in the image segmentation process, the second loss function determined in the key point detection process, and the two sub-loss functions in the classification process. The model is trained using the third loss function and optimized from multiple angles, thereby improving the accuracy of facial segmentation of the sample image by the trained image segmentation model.

[0143] Below, an exemplary application of the model training method provided in an embodiment of the present application in an actual application scenario will be described.

[0144] Image segmentation is the process of dividing an image into multiple segments, distinguishing between the target and the background at the pixel level, and has a wide range of applications. Portrait segmentation, which segments a half-length portrait and distinguishes it from the background, is a classic image segmentation task. The development of deep learning technology has seen rapid breakthroughs in both the effectiveness and efficiency of portrait segmentation. Image segmentation models are used to obtain segmented images and distinguish between masks for the foreground and background. The resulting segmented images can be applied in various image processing scenarios. For example, in a video call or video conferencing scenario, the image to be segmented can be a real-time portrait of a user participating in the video call or video conferencing. The output segmented portrait image can be used to replace the video background of the video call or video conferencing.

[0145] Related technologies typically train image segmentation models directly through supervised training based on labeled datasets, enabling them to achieve segmentation capabilities. However, for rapidly changing, complex portraits, the accuracy of the model's image segmentation cannot be guaranteed. Segmentation results are prone to artifacts along edges, and the segmentation effect on fine textures is poor, resulting in a lack of high-resolution image segmentation.

[0146] In an embodiment of the present application, an image segmentation model is used to extract features of an input sample image, and a segmentation network is used to implement image segmentation based on the extracted image features to obtain an output mask image. The first loss function of the image segmentation model is determined by calculating the cross entropy loss between the output mask image and the actual mask image. A classification network is added to optimize the upsampling process in the segmentation network. The second loss function of the image segmentation model is determined by calculating the cross entropy loss between the predicted category score and the actual category. Key points of the output mask image are predicted through a facial recognition network. The third loss function of the image segmentation model is determined based on the positional difference between the predicted key points and the actual key points. Multiple loss functions are linearly added to determine the total loss function of the image segmentation model. The image segmentation model is trained using the total loss function, and the trained image segmentation model is capable of achieving more accurate segmentation of the sample image.

[0147] The following is a description with reference to the accompanying drawings. Figure 4 , Figure 4 This is a fourth flow chart of the model training method provided in the embodiment of the present application. The execution subject can be a terminal device, a server, or a combination of the two. The embodiment of the present application takes the server as the execution subject as an example. Figure 4 The steps shown are explained in detail.

[0148] In step 401, feature extraction is performed on a sample image to obtain sample image features.

[0149] For example, the sample image is an image including the foreground and background to be segmented. The foreground object can be a living object, such as a human object or other living things. When it is a human object, it can specifically be the whole body, upper body or head of the human object. The foreground object can also be an inanimate object, such as a vehicle, furniture or building. When the foreground object is the head of a human object, the sample image is a portrait. The image segmentation model is an artificial intelligence model that is used to perform image segmentation on the foreground object in the image to obtain a segmented image of the foreground object. The input data of the image segmentation model is the sample image, and the final output data is the image segmentation result. The image segmentation model includes a feature extraction network and a segmentation network. In the process of training the image segmentation model, a classification network and a facial recognition network are also introduced to evaluate the mask map output by the segmentation network. The sample image is input into the image segmentation model, and the feature extraction network is used to extract features from the sample image to obtain the sample image features. In an embodiment of the present application, the feature extraction network is a lightweight neural network (MobileNetV3 network). The MobileNetV3 network uses depth-separable convolution to reduce the number of parameters and computational complexity. The feature extraction network includes a downsampling stage. The output of the feature extraction network is a set of feature maps. The output sample image features include hierarchical features learned from the input image.

[0150] For example, the sample images in the image segmentation model are randomly extracted from the sample dataset, which contains four types of portrait samples. The first type of portrait samples are original portrait samples, which are pre-collected from public datasets for image segmentation; the second type of portrait samples are obtained by using module components to attach random images (for example, solid or hollow circles) around the original portraits. The module components are components used to create and manipulate masks during image processing. The mask is a pixel-level mapping used to indicate which parts of the image should be modified or retained; the third type of portrait samples are obtained by erasing the original images, for example: dividing the original portrait into 9*9 blocks, randomly erasing 20%-30% of small blocks of images (to prevent the erased parts from having no portraits), and obtaining the missing images as the third type of portrait samples; the fourth type of samples are images with a background of the same color as the portrait clothes.

[0151] In step 402, segmentation processing is performed through a segmentation network based on the sample image features to obtain a mask image.

[0152] For example, the segmentation network classifies each pixel in the image into different categories to distinguish different objects or areas. The segmentation network usually includes an encoder and a decoder. In an embodiment of the present application, a U-Net network is used as a segmentation network. The U-Net network is a convolutional neural network structure. The structure of the U-Net network includes a downsampling module and an upsampling module. The downsampling processing of the U-Net network downsampling module is a method of reducing the image size, which can reduce the computational complexity and extract higher-level features, and help capture the contextual information of the image. The second half of the U-Net network structure is an upsampling module. The upsampling module is mainly responsible for upsampling processing and feature fusion, and finally outputs a segmentation map. Based on the sample image features, downsampling and upsampling are performed through the segmentation network to obtain a mask map after segmenting the sample image (equivalent to the above-mentioned first mask map).

[0153] For example, the mask image is a binary image. Each pixel of the mask image contains only two possible values, usually 0 and 1, or black and white. The mask image is used to control the visibility of the foreground and background during image synthesis. A pixel value of 1 (or white) represents the foreground area, and the pixels in the foreground area will be retained during the synthesis process; a pixel value of 0 (or black) represents the background area, and the pixels in the foreground area will be ignored or set to transparent during the synthesis process. Multiplying each pixel value in the mask image with the pixel value at the same position in the foreground image determines the pixels to be included in the final portrait image, and obtains a new image that only contains the desired foreground content, that is, a new image in which the background is completely removed and only contains the portrait of the person.

[0154] In some embodiments, see Figure 5 , Figure 5 Schematic diagram of the structure of an image segmentation model provided in an embodiment of the present application; the image segmentation model includes a feature extraction network 502, a segmentation network 503, the segmentation network 503 including a downsampling module 5031 and an upsampling module 5032, a classification network 504 including a first classification network 5041 and a second classification network 5042, and a facial recognition network 505. A sample image 501 is input into the image segmentation model, and the feature extraction network 502 extracts features from the sample image to obtain sample image features. The sample image features are used as input to the segmentation network, and downsampling processing is performed by the downsampling module 5031.

[0155] For example, the downsampling module 5031 includes four downsampling convolution layers: downsampling layer 50311, downsampling layer 50312, downsampling layer 50313 and downsampling layer 50314. The number of downsampling can be set according to the size of the mask image so that the downsampling mask label is consistent with the size of the mask image. In the embodiment of the present application, upsampling and downsampling use 4 layers of convolution and activation functions respectively. The convolution of the downsampling module 5031 is 3*3 convolution with a step size of 2. As a downsampling layer, the convolution kernel size is 64, and the downsampling feature is obtained after convolution. The step size is the stride size of the convolution kernel sliding on the input feature map, which determines the distance the convolution kernel slides each time.

[0156] In the segmentation network, the convolution layer is used to extract image features. The 3*3 convolution representation segmentation network extracts a 3*3 local feature area from the input image each time. The step size is 2 in downsampling, which means that after the convolution operation, the size of the feature map will be reduced by half. Downsampling is a method to reduce the image size, which can reduce computational complexity and extract higher-level features, which helps to capture the contextual information of the image and learn more abstract features.

[0157] For example, the down-sampled features obtained by down-sampling are input into the up-sampling module 5032 for up-sampling. The up-sampling module 5032 includes four up-sampling convolution layers: up-sampling layer 50321, up-sampling layer 50322, up-sampling layer 50323 and up-sampling layer 50324. The convolutions in the up-sampling layers are all 3*3 convolutions with a step size of 1 / 2 and a convolution kernel size of 64. Then, a 3*3 convolution layer 506 is used for feature smoothing. The feature smoothing process is used to reduce noise, perform feature integration, keep edges clear and share parameters. The last layer is a 1*1*3 convolution layer 507, which outputs the segmented mask image 508.

[0158] The upsampling module 5032 is used to restore the original size of the image. During the upsampling process, the size of the feature map will increase in order to be fused with the corresponding feature map during the downsampling process. The upsampled feature map may contain some unnecessary details or noise. Therefore, a 3*3 convolution layer 506 is usually used after upsampling to smooth the feature map. Smoothing refers to the process of reducing noise and unnecessary details in the image, making the main features in the image more prominent while reducing interference information. This convolution layer usually has a stride of 1 and does not change the size of the feature map, but reduces the number of channels, thereby reducing the complexity of the feature map and helping to remove noise. The 1*1 convolution layer 507 means that the features from the upsampling and convolution layers 506 are processed independently at each pixel.

[0159] Continue to see Figure 4In step 403, a difference calculation is performed between the mask image and the actual mask image, and a first loss function is determined according to the image difference.

[0160] For example, after the segmentation network performs segmentation processing on the sample image features, a mask image (equivalent to the first mask image mentioned above) is obtained, and the cross entropy loss between the mask image and the actual mask image of the sample image is calculated to determine the first loss function, which can be implemented by formula (1), as described in detail below.

[0161] Loss1=BCE(unet(x),mask) (1)

[0162] Here, Loss1 represents the first loss function (equivalent to the first loss function described above), BCE represents the binary cross-entropy, unet represents the segmentation network, x represents the input sample image, unet(x) represents the mask output by the segmentation network (equivalent to the first mask described above), and mask represents the actual sample mask. The binary cross-entropy (BCE) loss function is a common method for evaluating the performance of binary classification models. It is used to measure the degree of inconsistency between the model's predictions and the true labels, assisting in adjusting the segmentation network parameters and reducing inconsistencies.

[0163] In step 404, the data obtained in the upsampling process of the segmentation network is classified to obtain category scores, and a classification loss function is determined based on the category scores.

[0164] For example, in the upsampling process of the segmentation network, two classification networks are introduced to classify the upsampled features. The output features of the two layers of upsampling are used as the input of the first classification network, and the classification prediction results of the first classification network are obtained as the output category score (equivalent to the predicted category of the above-mentioned sample image). The last layer of the first classification network outputs the probabilities corresponding to the four sample image categories, and the sample image category corresponding to the highest probability value is used as the category of the upsampled feature.

[0165] The classification prediction results output by the first classification network are weighted and fed into the second classification network. This network then uses the results as input for classification prediction, obtaining the scores output by the second classification network (equivalent to the predicted category for each pixel mentioned above). The scores output by the second classification network are used as weights to represent the category of the pixel. The scores of the second classification network are weighted and fed to the penultimate layer of the upsampling process, and the parameters of the segmentation network are fine-tuned. The scores output by the classification network represent the confidence level that the input image belongs to different categories.

[0166] The difference between the classification prediction result output by the first classification network and the actual category is calculated to obtain the classification difference of the first classification network. Based on the classification difference, the cross entropy loss of the first classification network is determined as the first sub-loss function, which can be represented by Loss2 = BCE(score1), where Loss2 is the first sub-loss function, BCE represents the binary cross entropy, and score1 represents the predicted score output by the first classification network. Based on the same classification process as the first classification network, the second sub-loss function is determined, which can be represented by Loss3 = BCE(score2), where Loss3 is the second sub-loss function, BCE represents the binary cross entropy, and score2 represents the predicted score output by the second classification network. The classification loss function of the image segmentation model includes the first sub-loss function Loss2 and the second sub-loss function Loss3.

[0167] In some embodiments, see Figure 5 The first classification network 5041 classifies the upsampling features in the upsampling layer 50321 and the upsampling layer 50322 in the upsampling module 5032, weights the output score result to the second classification network 5042, outputs a second score, weights it to the upsampling layer 50323, and optimizes the upsampling module 5032. The convolution of the first classification network 5041 and the second classification network 5042 are both 3*3 convolutions with a stride of 1 / 2. As the upsampling layer, the convolution kernel size is 64, and the channel dimension is compressed through a 1*1*1 convolution. Then, a 1*1 vector is obtained through a global maximum pooling layer. The 1*1 vector represents the category, and the obtained vector is used as the category score.

[0168] Continue to see Figure 4 In step 405, key point detection is performed based on the mask image through the facial recognition network to obtain predicted key points.

[0169] For example, facial landmark detection is performed by identifying and locating 68 specific points in a face image to detail the facial structure and features, including key facial features: eyes, nose, mouth, and facial contours. The mask output by the upsampling layer is fed into the facial recognition network for facial landmark detection, resulting in predicted keypoints. Each keypoint's label represents a score, not a coordinate. The score indicates whether a facial image exists within a certain area of ​​the mask image or whether the facial image segmentation output is correct.

[0170] See also Figure 5, the key point detection in the mask image output by the upsampling layer 50324 is realized through the facial recognition network 505. The facial recognition network 505 is realized by adding a single-point headless face detector. It is a network with fixed pre-trained parameters. The facial recognition network 505 directly predicts the position and key points of the face from the mask image, and outputs the score of each key point in the mask image. The score represents the category of the key point, for example: the key point is the key point located at the left eye, right eye, and nose.

[0171] Continue to see Figure 4 In step 406, the difference between the predicted key points and the actual key points is calculated, and the second loss function is determined according to the position difference.

[0172] For example, the position difference between the predicted key points output by the facial recognition network and the actual key points in the sample image is calculated. Based on the position difference of the pre-configured key points, the third loss function is determined. The pre-configured key points are the 17 edge key points of the face. Each area of ​​the mask image contains multiple key points. The threshold of the key points is set. When the key point is any key point of the eyes, nose and mouth, if the corresponding threshold is greater than the pre-set threshold of 0.8, the training of the key point is stopped and the value is assigned to 1. The key point assigned to 1 indicates that it has been accurately identified and located, and no further training is required. Continue training for other pre-configured key points. The third loss function can be implemented by formula (2), which is explained in detail below.

[0173] Lmds=1if(lmd(i)-label[i])>μ,i∈[eyes, mouth, nose]

[0174] Lmds=1 / N(lmd(i)-label[i]),i∈others(2)

[0175] Among them, Lmds represents the second loss function, 1if is used to determine whether the condition is met, lmd(i) represents the set of facial key points, i represents the region, each i includes multiple key points, when i∈[eyes, mouth, nose], the value of i is 6 eye key points, 5 nose key points and 12 mouth key points, when i∈other, the value of i is the remaining key points of the face except eyes, mouth and nose, a total of 45, and a total of 68 facial key points, μ represents the threshold of the preset value, which is 0.8, N is the format of the facial edge key points, which represents the total number of key points in other areas, which is 17, and label[i] represents the combination of actual key points.

[0176] In step 407, based on the first loss function, the second loss function and the classification loss function, a third loss function of the image segmentation model is determined, and the image segmentation model is trained using the third loss function to obtain a trained image segmentation model.

[0177] For example, based on the first loss function Loss1, the classification loss function (including the first sub-loss function Loss2 and the second sub-loss function Loss3), and the second loss function Lmds, multiple loss functions are linearly added to determine the third loss function of the image segmentation model. The determined third loss function is sent to each layer of the model, and the image segmentation model is trained to obtain the trained image segmentation model. The determination of the total loss function can be achieved by formula (3), which is described in detail below.

[0178] Total Loss=5Loss1+Loss2+Loss3+2Lmds (3)

[0179] Among them, total Loss is the third loss function of the image segmentation model, Loss1 is the first loss function, Loss2 and Loss3 are the two sub-losses contained in the classification loss function, and Lmds is the second loss function.

[0180] For example, after obtaining the trained image segmentation model, the image to be segmented is input into the image segmentation model, and feature extraction processing is performed through the feature extraction network. Based on the extracted image features, the trained segmentation network is used to segment the facial portrait, and the segmented mask image is output.

[0181] The model training method provided in the embodiment of the present application has the following beneficial effects:

[0182] During the training process of the image segmentation model constructed in the present application, the feature extraction network is used to extract features from the input sample image to obtain image features, and the image features are downsampled and upsampled by the segmentation network to obtain a segmented mask map. Two classification networks are introduced for classification processing in the upsampling process of the segmentation network. The scores output by the classification processing are used to fine-tune the upsampling structure to assist the image segmentation model in outputting more accurate images. A facial recognition network is added to the mask map output by the segmentation network to identify facial key points and evaluate the positions of facial key points in the segmented image, thereby improving the accuracy of the image segmentation network in locating facial key points. The training process of the facial recognition network is based on the types of pre-configured key points, which improves the efficiency of the facial recognition network and saves time costs. The first loss function, the second loss function and the third loss function are determined by difference calculation, and the multiple loss functions are linearly added to determine the total loss function of the image segmentation model. The total loss function is sent to each layer of the network to train the image segmentation model, and the trained image segmentation model is obtained to achieve accurate image segmentation.

[0183] The following continues to describe the exemplary structure of the model training device 455 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the model training device 455 of the memory 450 may include: a processing module 4551, used to perform image segmentation on the sample image through an image segmentation model to obtain a first mask image; a determination module 4552, used to perform difference calculation between the first mask image and the actual mask image of the sample image to obtain a first loss function; perform key point detection on the first mask image to obtain a first key point; perform difference calculation between the first key point and the actual key point of the sample image to obtain a second loss function; and train the image segmentation model based on the first loss function and the second loss function.

[0184] In some embodiments, the processing module 4551 is also used to extract features of the sample image through the model to obtain sample image features, and to downsample the sample image features to obtain downsampled features, and to upsample the downsampled features to obtain upsampled features; perform nonlinear transformation on the upsampled features, and classify the results of the nonlinear transformation to obtain a first mask image.

[0185] In some embodiments, before performing image segmentation on the sample image through the model to obtain the first mask image, the processing module 4551 is also used to obtain the sample image through at least one of the following processing: obtaining a preconfigured original image as a sample image; adding noise to the original image to obtain a sample image; erasing at least part of the original image to obtain a sample image; obtaining an image with the same background color as the original image as the sample image.

[0186] In some embodiments, the determination module 4552 is also used to classify the upsampled features to obtain the predicted category of the sample image, and perform operations based on the predicted category and the actual category to obtain a classification loss function; combine the first loss function, the second loss function, and the classification loss function to obtain a third loss function; and train the model through the third loss function.

[0187] In some embodiments, the determination module 4552 is further used to perform a difference calculation based on the predicted category of the sample image and the actual category of the sample image features to obtain a first classification loss function; based on the predicted category of the sample image and the upsampling features, the pixels in the sample image are classified to obtain a predicted category for each pixel in the sample image; and based on the difference calculation between the predicted category of each pixel and the actual category of the corresponding pixel, a second classification loss function is obtained.

[0188] In some embodiments, the loss determination module 4552 is also used to determine the type of the first key point; when the type of the first key point is a preconfigured type, the difference between the first key point and the actual key point is calculated to obtain a first difference; based on the number of the first key points, a preset operation is performed on the first difference to obtain a second difference, and a second loss function is determined based on the second difference.

[0189] In some embodiments, key point detection is implemented through a facial recognition network. Key point detection is used to determine key points and predicted label values ​​corresponding to key points. The determination module 4552 is also used to determine the predicted label value corresponding to the first key point; the facial recognition network is trained through a second loss function. When there is a predicted label value greater than a label value threshold, the actual key point corresponding to the second key point greater than the label value threshold is deleted from the sample image. When there are remaining first key points in the sample image, the facial recognition network continues to be trained based on the remaining first key points. The second key point refers to the first key point with a predicted label value greater than the label value threshold; when all actual key points in the sample image are deleted, it is determined that the training of the facial recognition network is completed.

[0190] In some embodiments, the segmentation module 4553 is used to perform image segmentation on the image to be segmented through a model to obtain a second mask image corresponding to the target object in the image to be segmented. The model is trained based on the model training method provided in the embodiment of the present application; the second mask image is subjected to a preset operation with the image to be segmented to obtain the target object image in the image to be segmented.

[0191] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the model training method and image segmentation method described in the present invention.

[0192] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the model training method and image segmentation method provided in the embodiment of the present application, for example, Figure 3A The model training method shown or Figure 3D The image segmentation method is shown.

[0193] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0194] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0195] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0196] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0197] To summarize, the embodiment of the present application segments the sample image to obtain a first mask image, performs key point detection on the first mask image to obtain first predicted key points, determines the first loss function and the second loss function through the difference between the first mask image and the first predicted key points and the actual mask image and the key points, and trains the image segmentation model through the first loss function and the second loss function. In the process of determining the loss function, more factors are referred to so that the trained image segmentation model can better recognize key point information and achieve more accurate image segmentation.

[0198] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Performing image segmentation on the sample image through the model to obtain a first mask image; performing a difference calculation between the first mask image and an actual mask image of the sample image to obtain a first loss function; Performing key point detection on the first mask image to obtain a first key point; performing a difference calculation on the first key point and the actual key point of the sample image to obtain a second loss function; The model is trained based on the first loss function and the second loss function.

2. The method according to claim 1, characterized in that The method of performing image segmentation on the sample image by using the model to obtain a first mask image includes: Extracting features from the sample image using the model to obtain sample image features, performing downsampling processing on the sample image features to obtain downsampled features, and performing upsampling processing on the downsampled features to obtain upsampled features; Nonlinear transformation is performed on the up-sampled features, and classification processing is performed on the results of the nonlinear transformation to obtain a first mask image.

3. The method according to claim 2, characterized in that The training of the model based on the first loss function and the second loss function includes: Classify the upsampled features to obtain a predicted category of the sample image, and perform calculations based on the predicted category and the actual category to obtain a classification loss function; Combining the first loss function, the second loss function, and the classification loss function to obtain a third loss function; The model is trained using the third loss function.

4. The method according to claim 3, characterized in that The classification loss function includes: a first classification loss function and a second classification loss function; the classification loss function is obtained by performing calculation based on the predicted category and the actual category, including: performing a difference calculation based on the predicted category of the sample image and the actual category of the sample image feature to obtain the first classification loss function; Based on the predicted category of the sample image and the up-sampled features, classify the pixels in the sample image to obtain the predicted category of each pixel in the sample image; The second classification loss function is obtained by calculating the difference between the predicted category of each pixel and the actual category of the corresponding pixel.

5. The method according to claim 1, wherein The performing difference calculation on the first key point and the actual key point of the sample image to obtain a second loss function includes: determining the type of the first key point; When the type of the first key point is a preconfigured type, performing a difference calculation on the first key point and the actual key point to obtain a first difference; Based on the number of the first key points, a preset operation is performed on the first difference to obtain a second difference, and a second loss function is determined based on the second difference.

6. The method according to claim 5, characterized in that The key point detection is implemented by a facial recognition network, and the key point detection is used to determine the key points and the predicted label values ​​corresponding to the key points; After performing a preset operation on the first difference based on the number of the first key points to obtain a second difference, and determining a second loss function based on the second difference, the method further includes: Determining a predicted label value corresponding to the first key point; Training the facial recognition network using the second loss function, deleting, from the sample image, actual key points corresponding to second key points having a predicted label value greater than a label value threshold when there are predicted label values ​​greater than the label value threshold, and continuing to train the facial recognition network based on the remaining first key points when there are remaining first key points in the sample image, where the second key points are first key points having a predicted label value greater than the label value threshold; When all actual key points in the sample image are deleted, it is determined that the training of the facial recognition network is completed.

7. The method according to any one of claims 1 to 6, characterized in that Before performing image segmentation on the sample image using the model to obtain the first mask image, the method further includes: A sample image is obtained by performing at least one of the following processing: Get a preconfigured original image as a sample image; Adding noise to the original image to obtain the sample image; Erasing at least a portion of the original image to obtain the sample image; An image having the same background color as the original image is obtained as the sample image.

8. An image segmentation method, characterized in that: The method comprises: Performing image segmentation on the image to be segmented using a model to obtain a second mask image corresponding to the target object in the image to be segmented, wherein the model is trained based on the model training method according to any one of claims 1 to 7; A preset operation is performed on the second mask image and the image to be segmented to obtain an image of the target object in the image to be segmented.

9. A model training device, characterized in that: The device comprises: A processing module, configured to perform image segmentation on the sample image using an image segmentation model to obtain a first mask image; A determination module is used to calculate the difference between the first mask image and the actual mask image of the sample image to obtain a first loss function; perform key point detection on the first mask image to obtain a first key point; calculate the difference between the first key point and the actual key point of the sample image to obtain a second loss function; and train the image segmentation model based on the first loss function and the second loss function.

10. An image segmentation device, characterized in that: The device comprises: A segmentation module is used to perform image segmentation on the image to be segmented through a model to obtain a second mask image corresponding to the target object in the image to be segmented, wherein the model is trained based on the model training method described in any one of claims 1 to 7; the second mask image is subjected to a preset operation with the image to be segmented to obtain an image of the target object in the image to be segmented.

Images