Image segmentation model training methods, image segmentation methods, devices, and equipment

Abstract

This application discloses a training method, image segmentation method, apparatus, and device for an image segmentation model, relating to the field of image segmentation. The method includes: extracting features from a sample image and a scale image to obtain sample image features of the sample image and scale image features of the scale image; calculating class activation maps to obtain sample class activation maps of the sample image and scale class activation maps of the scale image; calling a decoder to decode the sample image features to obtain sample segmentation results of the sample image, and calling a decoder to decode the scale image features to obtain scale segmentation results of the scale image; calculating class activation map loss and scale loss based on the sample class activation maps, sample segmentation results, scale class activation maps, and scale segmentation results; and training the decoder based on the class activation map loss and scale loss. This method improves the accuracy of image segmentation.

Description

Technical Field

[0001] This application relates to the field of image segmentation, and in particular to a training method for an image segmentation model, an image segmentation method, an apparatus, and a device. Background Technology

[0002] Image segmentation refers to dividing an image into several non-overlapping regions based on features such as grayscale, color, spatial texture, and geometric shape. These features should exhibit consistency or similarity within the same region, while showing significant differences between different regions. Simply put, it's about separating the foreground object from the background in an image.

[0003] In related technologies, thresholding is used for image segmentation. The basic idea of ​​thresholding is to calculate one or more gray-level thresholds based on the gray-level features of the image, compare the gray-level value of each pixel in the image with the threshold, and finally classify the pixels into the appropriate category according to the comparison result. Therefore, the most crucial step of this method is to solve for the optimal gray-level threshold according to a certain criterion function.

[0004] The methods in related technologies require the target to have obvious edges or grayscale differences, and can only be used to determine the target based on shallow features such as image pixel values, resulting in low accuracy. Summary of the Invention

[0005] This application provides a training method for an image segmentation model, an image segmentation method, an apparatus, and a device, which can improve the accuracy of image segmentation. The technical solution is as follows.

[0006] According to one aspect of this application, a method for training an image segmentation model is provided, the image segmentation model including an encoder and a decoder, the method comprising:

[0007] The encoder is invoked to extract features from the sample image and the scale image to obtain the sample image features of the sample image and the scale image features of the scale image. The scale image includes at least one of the following: an image obtained by enlarging the sample image, or an image obtained by shrinking the sample image.

[0008] The sample class activation map of the sample image is obtained by calculating the class activation map based on the sample image features, and the scale class activation map of the scale image is obtained by calculating the class activation map based on the scale image features; the class activation map is used to represent the degree of influence of each pixel in the image on the classification result of the image;

[0009] The decoder is invoked to decode the features of the sample image to obtain the sample segmentation result of the sample image, and the decoder is invoked to decode the features of the scale image to obtain the scale segmentation result of the scale image; the sample segmentation result includes the classification probability value of each pixel in the sample image;

[0010] The class activation map loss is calculated based on the sample class activation map, the sample segmentation result, the scale class activation map, and the scale segmentation result; the scale loss is calculated based on the sample segmentation result and the scale segmentation result; the class activation map loss is used to train the decoder to make the sample segmentation result closer to the sample class activation map and the scale segmentation result closer to the scale class activation map; the scale loss is used to train the decoder to make the sample segmentation result closer to the scale segmentation result.

[0011] The decoder is trained based on the class activation graph loss and the scale loss.

[0012] According to another aspect of this application, a training apparatus for an image segmentation model is provided, the image segmentation model including an encoder and a decoder, the apparatus comprising:

[0013] The encoding module is used to call the encoder to extract features from the sample image and the scale image to obtain the sample image features of the sample image and the scale image features of the scale image. The scale image includes at least one of the following: an image obtained by enlarging the sample image or an image obtained by shrinking the sample image.

[0014] The class activation map module is used to calculate the class activation map based on the features of the sample image to obtain the sample class activation map of the sample image, and to calculate the class activation map based on the features of the scale image to obtain the scale class activation map of the scale image; the class activation map is used to represent the degree of influence of each pixel in the image on the classification result of the image;

[0015] The decoding module is used to call the decoder to decode the features of the sample image to obtain the sample segmentation result of the sample image, and to call the decoder to decode the features of the scale image to obtain the scale segmentation result of the scale image; the sample segmentation result includes the classification probability value of each pixel in the sample image;

[0016] The loss module is used to calculate a class activation map loss based on the sample class activation map, the sample segmentation result, the scale class activation map, and the scale segmentation result, and to calculate a scale loss based on the sample segmentation result and the scale segmentation result; the class activation map loss is used to train the decoder to make the sample segmentation result closer to the sample class activation map and the scale segmentation result closer to the scale class activation map; the scale loss is used to train the decoder to make the sample segmentation result closer to the scale segmentation result.

[0017] The training module is used to train the decoder based on the class activation graph loss and the scale loss.

[0018] According to another aspect of this application, an image segmentation method is provided, the method comprising:

[0019] Obtain the input image;

[0020] The encoder is invoked to extract features from the input image, thereby obtaining the input image features;

[0021] The decoder is invoked to decode the features of the input image to obtain the image segmentation result of the input image. The decoder is trained based on class activation map loss and scale loss. The class activation map loss is used to train the decoder to output segmentation results close to the class activation map. The class activation map is used to represent the degree of influence of each pixel in the image on the classification result of the image. The scale loss is used to train the decoder to output similar segmentation results for multiple images with the same image content but different scales.

[0022] According to another aspect of this application, an image segmentation apparatus is provided, the apparatus comprising:

[0023] The acquisition module is used to acquire the input image;

[0024] The feature extraction module is used to call the encoder to extract features from the input image and obtain the input image features of the input image;

[0025] The image segmentation module is used to call the decoder to decode the features of the input image and obtain the image segmentation result of the input image. The decoder is trained based on class activation map loss and scale loss. The class activation map loss is used to train the decoder to output segmentation results close to the class activation map. The class activation map is used to represent the degree of influence of each pixel in the image on the classification result of the image. The scale loss is used to train the decoder to output similar segmentation results for multiple images with the same image content but different scales.

[0026] According to another aspect of this application, a computer device is provided, the computer device comprising: a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the training method or image segmentation method of the image segmentation model as described above.

[0027] According to another aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the training method or image segmentation method of the image segmentation model as described above.

[0028] According to another aspect of the embodiments of this disclosure, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the training method or image segmentation method of the image segmentation model provided in the above-described optional implementation.

[0029] The beneficial effects of the technical solutions provided in this application include at least the following:

[0030] Based on the idea that the image segmentation result should not change after scaling the same image, a sample image and a scaled version of the sample image are used as inputs to the encoder and decoder respectively for image segmentation, resulting in two image segmentation results. The scale loss can be calculated based on these two results and used to train the encoder, making the two image segmentation results approximate each other, ensuring that the image segmentation result remains unchanged after image size changes. Based on the image features output by the encoder, class activation maps are calculated for both images. Since the class activation map reflects the main pixels used for image classification, the image segmentation result should be close to the class activation map. The class activation map loss is calculated using the image segmentation result and the class activation map, and used to train the encoder. Ultimately, the image segmentation result output by the encoder is close to the class activation map while ensuring that the image segmentation result remains unchanged after image scaling. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a block diagram of a computer device provided in an exemplary embodiment of this application;

[0033] Figure 2 This is a flowchart of a method for training an image segmentation model provided in another exemplary embodiment of this application;

[0034] Figure 3 This is a flowchart of a method for training an image segmentation model provided in another exemplary embodiment of this application;

[0035] Figure 4 This is a flowchart of a method for training an image classification model provided in another exemplary embodiment of this application;

[0036] Figure 5 This is a schematic diagram of a training method for an image classification model provided in another exemplary embodiment of this application;

[0037] Figure 6 This is a schematic diagram of a training method for an image segmentation model provided in another exemplary embodiment of this application;

[0038] Figure 7 This is a flowchart of an image segmentation method provided in another exemplary embodiment of this application;

[0039] Figure 8 This is a block diagram of a training apparatus for an image segmentation model provided in another exemplary embodiment of this application;

[0040] Figure 9 This is a block diagram of an image segmentation apparatus provided in another exemplary embodiment of this application;

[0041] Figure 10 This is a schematic diagram of the server structure provided in another exemplary embodiment of this application;

[0042] Figure 11 This is a block diagram of a terminal provided in another exemplary embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0044] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0045] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0046] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0047] Figure 1 A schematic diagram of a computer device 101 provided in an exemplary embodiment of this application is shown. The computer device 101 may be a terminal or a server.

[0048] The terminal may include at least one of a smartphone, laptop, desktop computer, tablet, smart speaker, or smart robot. In one optional implementation, the terminal has a client installed that supports image segmentation. For example, the client supporting image segmentation may be an image processing application or a video processing application client. For instance, the image processing application provides image segmentation functionality to intelligently identify facial regions in an image for automatic image optimization. Alternatively, the video processing application provides batch target modification functionality to intelligently identify targets (people, plants, animals, objects, etc.) in each frame of a video image and perform uniform image processing on the targets in each frame, such as adjusting color, brightness, and saturation.

[0049] For example, the terminal stores an image segmentation model. When the client needs to use the image segmentation function, it can call the image segmentation model to complete the image segmentation. For instance, when a user needs to optimize a target image, the client calls the image segmentation model to segment the target image, obtain the target region in the target image, and then automatically optimizes the target region.

[0050] The terminal and the server are connected to each other via wired or wireless network.

[0051] For example, the method provided in this application can be executed by a client on a terminal or by a server. That is, the training of the image segmentation model can be completed by the client or by the server. For example, in the application stage after the image segmentation model has been trained, the client can call the locally stored image segmentation model to perform image segmentation; the client can also send an image segmentation request to the server, and the server can call the image segmentation model to perform image segmentation; or the server can call the image segmentation model to perform image segmentation when it needs to segment the received image.

[0052] The terminal includes a first memory and a first processor. The first memory stores an image segmentation model; the image segmentation model is invoked and executed by the first processor to implement the training method of the image segmentation model provided in this application. The first memory may include, but is not limited to, the following: Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM).

[0053] The first processor can consist of one or more integrated circuit chips. Optionally, the first processor can be a general-purpose processor, such as a central processing unit (CPU) or a network processor (NP). Optionally, the first processor can implement the training method of the image segmentation model provided in this application by running programs or code.

[0054] The server includes a second memory and a second processor. The second memory stores an image segmentation model, which is invoked by the second processor to implement the training method for the image segmentation model provided in this application. For example, the server receives and stores user data sent by a terminal, and annotates information objects based on the user data. Optionally, the second memory may include, but is not limited to, the following: RAM, ROM, PROM, EPROM, and EEPROM. Optionally, the second processor may be a general-purpose processor, such as a CPU or NP.

[0055] For example, the image segmentation model stored in the client or server includes an encoder 102 and a decoder 103.

[0056] For example, the computer device calls the decoder 103 to perform feature extraction on the sample image X and the scale image R(X) to obtain the sample image features of the sample image X and the scale image features of the scale image R(X). The scale image R(X) is an image obtained by upsampling the sample image X; that is, the size of the scale image R(X) is twice that of the sample image X.

[0057] The sample class activation map of the sample image is obtained by calculating the class activation map 104 based on the sample image features, and the scale class activation map of the scale image is obtained by calculating the class activation map 104 based on the scale image features. The class activation map 104 is used to represent the degree of influence of each pixel in the image on the classification result of the image.

[0058] Decoder 103 is called to decode the features of the sample image to obtain the sample segmentation result of the sample image, and decoder 103 is called to decode the features of the scale image to obtain the scale segmentation result of the scale image; the sample segmentation result includes the classification probability value of each pixel in the sample image;

[0059] The loss L is calculated based on the sample class activation map, sample segmentation results, scale class activation map, and scale segmentation results. seg Loss L seg This includes class activation map loss and scale loss; class activation map loss is used to train the decoder to make the sample segmentation result closer to the sample class activation map, and to make the scale segmentation result closer to the scale class activation map; scale loss is used to train the decoder to make the sample segmentation result closer to the scale segmentation result.

[0060] The decoder 103 is trained based on class activation graph loss and scale loss.

[0061] Figure 2 A flowchart illustrating a training method for an image segmentation model provided in an exemplary embodiment of this application is shown. This method can be performed by a computer device, such as... Figure 1 The computer device shown is used to perform this action. The method includes the following steps.

[0062] Step 201: Call the encoder to extract features from the sample image and the scale image to obtain the sample image features of the sample image and the scale image features of the scale image. The scale image includes at least one of the following: an image obtained by enlarging the sample image or an image obtained by shrinking the sample image.

[0063] For example, an image segmentation model includes an encoder and a decoder. The image segmentation model is used to segment an input image to obtain the regions containing the classified target. The image segmentation model can be used to identify at least one classified target; for example, it can be used to identify the pixels containing cats, dogs, and people in an image.

[0064] For example, when an image style model is used to segment N categories of objects in an image, the image segmentation model can output N probability values ​​for each pixel in the image belonging to the N categories of objects based on the input image. This results in N classification probability maps for the image belonging to the N categories of objects. The pixel value of each pixel in the i-th classification probability map is the probability value of that pixel belonging to the i-th category of objects, where N is a positive integer and i is a positive integer not greater than N.

[0065] For example, based on N classification probability maps of an image, the classification target of each pixel in the image can be determined. For instance, the classification target with the highest probability value among the N probability values ​​of a pixel is determined as the classification target of that pixel. For example, if a pixel has a probability of 0.1 for a cat, 0.2 for a dog, and 0.7 for a human, then the classification target of this pixel is human. For example, the output of the image segmentation model can be a classification target map of the image, where the value of each pixel represents the classification target to which that pixel belongs. Therefore, the location (pixel) of each classification target can be seen on the classification target map.

[0066] In summary, the segmentation result output by the image segmentation model can be either the N classification probability maps mentioned above or the classification target map mentioned above.

[0067] Since scaling up or down an image does not change the positional distribution of the classified objects within it, the image segmentation result of a scaled-up or scaled-down image should be identical to that of the original image (the segmentation result is scaled to the same size). Therefore, the method provided in this embodiment is based on this idea to train an image segmentation network, making the segmentation results output by the image segmentation network approximate those of the original and scaled images.

[0068] Therefore, when obtaining training samples for an image segmentation model, it is necessary to obtain sample images and corresponding scale images. The scale images include at least one of magnified or reduced sample images. For example, magnifying a sample image can be done by upsampling, and reducing a sample image can be done by downsampling.

[0069] That is, a set of training samples includes at least one of the following three combinations:

[0070] 1) Sample image, and magnified image obtained by upsampling the sample image;

[0071] 2) Sample image, and a scaled-down image obtained by downsampling the sample image;

[0072] 3) Sample image, upsampled sample image to obtain a magnified image, and downsampled sample image to obtain a reduced image.

[0073] For example, the method steps in this embodiment are only used as an example of a set of training samples to illustrate the training method of the image segmentation model provided in this application. Based on the method provided in this embodiment, multiple sets of training samples can be used to iteratively train the image segmentation model.

[0074] For example, after acquiring the sample image and the scale image, the sample image and the scale image are respectively input into the encoder for feature extraction to obtain the sample image features of the sample image and the scale image features of the scale image.

[0075] For example, the encoder used in this embodiment is a pre-trained encoder with high feature extraction capabilities. In the training method of this embodiment, only the decoder of the image segmentation model is trained; the encoder is not trained. This encoder can be any pre-trained encoder from an image classification model. The image classification model is used to output the classification target to which the image belongs based on the input image. For example, the classification target identified by the image classification model is the same as the classification target identified by the image segmentation model; that is, the image classification model is also used to identify N classification targets.

[0076] For example, the encoder is a convolutional neural network (CNN), and its specific network structure can be adjusted according to the image size.

[0077] Step 202: Calculate the class activation map based on the sample image features to obtain the sample class activation map of the sample image, and calculate the class activation map based on the scale image features to obtain the scale class activation map of the scale image; the class activation map is used to represent the degree of influence of each pixel in the image on the classification result of the image.

[0078] For example, since the encoder uses the encoder from a pre-trained image classification model, the image features extracted by the encoder already contain the feature information for classifying the image. Therefore, by calculating the class activation map based on the image features output by the encoder, the pixel points on which the image classification model classifies the image can be obtained. The decoder is then trained using the class activation map so that the image segmentation result output by the decoder closely matches the class activation map.

[0079] Step 203: Call the decoder to decode the features of the sample image to obtain the sample segmentation result of the sample image, and call the decoder to decode the features of the scale image to obtain the scale segmentation result of the scale image; the sample segmentation result includes the classification probability value of each pixel in the sample image.

[0080] For example, the decoder is used to decode the image features output by the encoder to obtain the segmentation result.

[0081] For example, the decoder consists of multiple layers of convolutional neural networks. Typically, the decoder employs a network structure that is the inverse of the encoder. For instance, the encoder includes four convolutional blocks, each consisting of two convolutional layers with a kernel size of 3*3. After each convolutional block, the encoder connects to a pooling layer with a kernel size of 2*2, reducing the image features output by the convolutional block to half their original size. Correspondingly, the decoder can also include four convolutional blocks, each consisting of two convolutional layers with a kernel size of 3*3. Before each convolutional block, the decoder connects to an upsampling layer, doubling the size of the image features. In this way, the image size input to the encoder is the same as the image size output by the decoder.

[0082] Step 204: Calculate the class activation map loss based on the sample class activation map, sample segmentation result, scale class activation map, and scale segmentation result; calculate the scale loss based on the sample segmentation result and scale segmentation result; the class activation map loss is used to train the decoder to make the sample segmentation result closer to the sample class activation map and the scale segmentation result closer to the scale class activation map; the scale loss is used to train the decoder to make the sample segmentation result closer to the scale segmentation result.

[0083] For example, based on the above approach of approximating the class activation map to the segmentation result, the class activation map loss of the segmentation result and the class activation map loss of the segmentation result are calculated. Based on the above approach of keeping the segmentation result of the image unchanged after scale change, the scale loss of the sample segmentation result and the scale segmentation result are calculated.

[0084] Step 205: Train the decoder based on class activation graph loss and scale loss.

[0085] For example, the decoder is trained based on the class activation map loss and scale loss described above, so that the decoder outputs the segmentation result based on the class activation map, and the output result of the decoder ensures that the segmentation result of the image remains unchanged after the scale is changed.

[0086] In summary, the method provided in this embodiment, based on the idea that the image segmentation result should not change after scaling the same image, uses a sample image and a scaled image obtained from the sample image, respectively, as inputs to the encoder and decoder for image segmentation, obtaining two image segmentation results. Based on these two segmentation results, a scale loss can be calculated, and the encoder is trained using this scale loss to make the two image segmentation results approximate each other, ensuring that the image segmentation result remains unchanged after the image size changes. Based on the image features output by the encoder, class activation images are calculated for both images. Since the class activation image reflects the main pixels used for image classification, the image segmentation result should be close to the class activation image. The class activation image loss is calculated using the image segmentation result and the class activation image, and the encoder is trained using this loss. Ultimately, the image segmentation result output by the encoder can be close to the class activation image while ensuring that the image segmentation result obtained after image scaling remains unchanged.

[0087] An exemplary embodiment of computing class activation graphs and losses is provided, for example.

[0088] Figure 3 A flowchart illustrating a training method for an image segmentation model provided in an exemplary embodiment of this application is shown. This method can be performed by a computer device, such as... Figure 1 The terminal or server shown is used to execute the command. Figure 2 Based on the exemplary embodiment shown, step 301 is included before step 202, step 202 includes step 2021, step 204 includes steps 2041 to 2043, and step 205 includes steps 2051 to 2052.

[0089] Step 301: Call the fully connected layer to perform classification prediction on the features of the sample image to obtain the sample classification result of the sample image; call the fully connected layer to perform classification prediction on the features of the scale image to obtain the scale classification result of the scale image.

[0090] For example, the image segmentation model also includes pre-trained fully connected layers.

[0091] For example, Figure 2 In the illustrated embodiment, the image classification model mentioned in step 201 also includes a fully connected layer. That is, the image classification model includes an encoder and a fully connected layer. After feature extraction from the image input encoder, the image passes through the fully connected layer to obtain the classification result. The image segmentation model also uses the fully connected layer of the image classification model to output the classification results of the sample image and the scale image respectively based on the image features output by the encoder.

[0092] For example, if the image segmentation model and the image classification model are models used to identify N classification targets, then the classification result is a vector consisting of N probability values ​​of the image belonging to the N classification targets.

[0093] Step 2021: Calculate the sample class activation map of the sample image based on the sample image features and sample classification results; calculate the scale class activation map of the scale image based on the scale image features and scale classification results.

[0094] For example, the formula for calculating the class activation graph is:

[0095]

[0096] Where c is the c-th classification target out of N classification targets, S c Let N*M be the probability value of class c in the classification results, and let N*M be the size of the image features. Let be the pixel value of the i-th image feature in the k-th row and j-th column.

[0097]

[0098] in, Let A be the class activation map of the image for the c-th classification target; ReLU is the activation function, indicating that the class activation map only focuses on pixels with a probability value greater than 0. i Let i be the i-th image feature.

[0099] For example, when an image segmentation model is used to identify N classification targets, the above calculation formula can be used to calculate N class activation maps of the image for the N classification targets. The i-th class activation map is used to represent the degree of influence of each pixel in the image on the image being classified into the i-th classification target.

[0100] That is, the sample class activation map includes the N class activation maps of the sample image for the N classification targets; the scale class activation map includes the N class activation maps of the scale image for the N classification targets.

[0101] Step 2041: Calculate the sample class activation map loss based on the sample class activation map and the sample segmentation results.

[0102] For example, the sample class activation map loss is calculated based on the error between the sample segmentation result and the sample class activation map. For example, the cross-entropy between the sample class activation map and the sample segmentation result is determined as the sample class activation map loss.

[0103] The formula for calculating the sample class activation map loss is as follows:

[0104] L seg,1 = -y1log(s1)

[0105] Among them, L seg,1y1 is the sample class activation map loss, s1 is the sample class activation map, and s1 is the sample segmentation result.

[0106] For example, if the class activation map is different in size from the segmentation result, it needs to be upsampled to the same size as the segmentation result before calculating the class activation map loss using the formula described above. Since the size of the segmentation result is the same as the size of the original image in the input image segmentation model (i.e., the size of the sample segmentation result is the same as the size of the sample image), and the class activation map is calculated based on the image features output by the encoder, its size is the same as the size of the image features output by the encoder. However, the size of the image features is smaller than the original image; therefore, the size of the class activation map is smaller than the segmentation result. Therefore, the class activation map needs to be upsampled to the size of the segmentation result, and then the upsampled class activation map is substituted into the formula described above to calculate the class activation map loss.

[0107] For example, the above formula is the formula for calculating the sample class activation map loss when the classification target of the image segmentation network is 1. When the classification target of the image segmentation network is N, the formula for calculating the sample class activation map loss is:

[0108]

[0109] Among them, L seg,1 For sample class activation map loss, y 1i Let s be the sample class activation map for the i-th classification target. 1i The sample segmentation result for the i-th classification target.

[0110] Step 2042: Calculate the scale class activation map loss based on the scale class activation map and the scale segmentation results.

[0111] For example, the scale-class activation map loss is calculated based on the error between the scale segmentation result and the scale-class activation map. For example, the cross-entropy between the scale-class activation map and the scale segmentation result is determined as the scale-class activation map loss.

[0112] The formula for calculating the scale-based activation map loss is as follows:

[0113] L seg,2 = -y2log(s2)

[0114] Among them, L seg,2 y2 is the scale-class activation map loss, y2 is the scale-class activation map, and s2 is the scale segmentation result.

[0115] For example, the above formula is the formula for calculating the scale-based activation map loss when the image segmentation network has only one classification target. When the image segmentation network has N classification targets, the formula for calculating the scale-based activation map loss is:

[0116]

[0117] Among them, L seg,2 For scale-based activation map loss, y 2i For the scale class activation map of the i-th classification target, s 2i The scale segmentation result for the i-th classification target.

[0118] For example, when the scaled image includes only one of a downscaled image and a magnified image, the scale-class activation map loss can be calculated using the above formula. When the scaled image includes both a downscaled image and a magnified image, the scale-class activation map loss includes both the downscale activation map loss and the magnified activation map loss. The downscale activation map loss L for the downscaled image can then be calculated using the above formula. seg,2.1 And the scaled-scale class activation map loss L for magnified images seg,2.2 .

[0119] Step 2043: Calculate the scale loss based on the sample segmentation results and the scale segmentation results.

[0120] For example, since the sample image and the scale image are different sizes, the output sample segmentation result and the scale segmentation result are also different sizes. Therefore, in order to compare the differences between the two, it is necessary to scale the sample segmentation result and the scale segmentation result to the same size. For example, since the encoder and decoder have corresponding structures, the output segmentation result and the input image size are the same. Therefore, according to the scale relationship between the sample image and the scale image, the sample segmentation result and the scale segmentation result are scaled to the same scale.

[0121] That is, the computer equipment scales the sample segmentation result to the same size as the scale segmentation result based on the scale relationship between the sample image and the scale image, and obtains the scaled sample segmentation result; the scale loss is calculated based on the error between the scale segmentation result and the scaled sample segmentation result.

[0122] For example, a scaled image is an image obtained by upsampling a sample image and magnifying it by two times. Similarly, the sample segmentation result is upsampled and magnified by two times to obtain a scaled sample segmentation result.

[0123] For example, the first matrix difference between the scaled segmentation result and the scaled sample segmentation result is calculated, and the 2-norm of the first matrix difference is determined as the scale loss.

[0124] The formula for calculating scale loss can be:

[0125] L seg,3 =‖s2-R(s1)‖2

[0126] Among them, L seg,3s2 is the scale loss, s1 is the scaled segmentation result, and R(s1) is the scaled sample segmentation result.

[0127] For example, the above formula is the formula for calculating the scale loss when the image segmentation network has only one classification target. When the image segmentation network has N classification targets, the formula for calculating the scale loss is:

[0128]

[0129] Among them, L seg,3 For scale loss, s 2i For the scale segmentation result of the i-th classification target, s 1i For the sample segmentation result of the i-th classification target, R(s) 1i ) represents the scaled sample segmentation result for the i-th classification target.

[0130] For example, when the scaled image includes only one of a downscaled image and a upscaled image, the above formula can be used to calculate the scale loss. When the scaled image includes two scaled images, a downscaled image and an upscaled image, the formula for calculating the scale loss is:

[0131] L seg,3 =‖s4-R4(s1)‖2+‖s5-R5(s1)‖2

[0132] Among them, L seg,3 For scale loss, s4 is the scaled segmentation result of the scaled image, s5 is the scaled segmentation result of the scaled image, s1 is the sample segmentation result, R4(s1) is the sample segmentation result after scaling down according to the scale relationship between the scaled image and the sample image, and R5(s1) is the sample segmentation result after scaling up according to the scale relationship between the scaled image and the sample image.

[0133] For example, the above formula is the formula for calculating the scale loss when the image segmentation network has only one classification target. When the image segmentation network has N classification targets, the formula for calculating the scale loss is:

[0134]

[0135] Among them, L seg,3 For scale loss, s 4i For the scaled-down segmentation result of the image for the i-th classified object, s 5i For the magnified image, the magnified segmentation result of the i-th classified object, s 1i The sample segmentation result for the i-th classification target; R4(s 1iR5(s) represents the segmentation result of the i-th classification target, which is the segmentation result scaled down according to the scale relationship between the scaled image and the sample image; 1i ) represents the sample segmentation result of the i-th classification target, which is the sample segmentation result after being enlarged according to the scale relationship between the magnified image and the sample image.

[0136] Step 2051: Calculate the weighted sum of the sample class activation map loss, the scale class activation map loss, and the scale loss.

[0137] For example, L seg,1 and L seg,2 This indicates that the segmentation result output by the constraint decoder tends to approach the corresponding class activation map. seg,3 The segmentation result obtained after scaling the image should be consistent with the segmentation result of the original image after the same scaling transformation.

[0138] Based on the three types of loss (sample class activation map loss, scale class activation map loss, and scale loss), the total loss is calculated:

[0139] L seg =L seg,1 +L seg,2 +λL seg,3

[0140] Among them, L seg For the total loss, L seg,1 For sample class activation map loss, L seg,2 For scale-based activation map loss, L seg,3 Let λ be the scaling loss and λ be the weight of the scaling loss.

[0141] For example, the weight of the sample class activation map loss is 1, the weight of the scale class activation map loss is 1, and the weight of the scale loss is 2, that is, λ = 2.

[0142] For example, when the scaled image includes only one of a downscaled image and a upscaled image, the above formula can be used to calculate the total loss. When the scaled image includes both a downscaled image and an upscaled image, the scale-class activation map loss includes both the downscale activation map loss and the upscale activation map loss. The total loss is then:

[0143] L seg =L seg,1 +L seg,2.1 +L seg,2.2 +λL seg,3

[0144] Among them, L seg For the total loss, L seg,1 For sample class activation map loss, L seg,2.1To reduce the scale-based activation map loss, L seg,2.2 To scale up the activation map loss, L seg,3 Let λ be the scaling loss and λ be the weight of the scaling loss.

[0145] Step 2052: Train the decoder based on the weighted sum.

[0146] For example, the decoder is trained based on the calculated weighted sum (total loss), which constrains the segmentation results to approach the class activation map and constrains the segmentation results of the same image at different sizes to remain consistent.

[0147] For example, after training an image segmentation model is completed, the image segmentation model can be used for image segmentation.

[0148] That is, the computer device calls the image segmentation model to perform image segmentation on the input image and obtain the image segmentation result. The image segmentation model includes an encoder and a decoder.

[0149] For example, the encoder is called to extract features from the input image to obtain the input image features; the decoder is called to decode the input image features to obtain the image segmentation result of the input image.

[0150] In summary, the method provided in this embodiment extracts features using an encoder of a pre-trained image classification model, ensuring that the encoder outputs image features with the necessary classification information. The fully connected layer of the image classification model outputs the image classification result, and then the class activation map is calculated using the classification result and image features. Although the class activation map can also output an image segmentation result, changes in image scale significantly affect the class activation map, while scale changes normally do not affect the image segmentation result. Therefore, based on the idea of ​​scale invariance, two scales of the same image are introduced. The decoder is trained using the errors of the two segmentation results at the two scales, enabling the decoder to accurately segment the image while ensuring that the segmentation result remains unchanged after scale transformation.

[0151] For example, the encoder and fully connected layer in the image segmentation model use the encoder and fully connected layer of a pre-trained image classification model. For example, assuming the encoder in the image classification model is a classification encoder and the fully connected layer is a classification fully connected layer, during the initialization phase of the image segmentation model, the computer device sets the encoder parameters based on the parameters of the classification encoder in the pre-trained image classification model. The classification target of the image classification model and the image segmentation model are the same. Similarly, based on the parameters of the classification fully connected layer in the pre-trained image classification model, the parameters of the fully connected layer are set. The image classification model is used to output classification results based on the input image.

[0152] An exemplary embodiment of training an image classification model is given.

[0153] Figure 4 A flowchart illustrating a training method for an image segmentation model provided in an exemplary embodiment of this application is shown. This method can be performed by a computer device, such as... Figure 1 The method is executed using the terminal or server shown. The method includes the following steps.

[0154] Step 401: Obtain data samples.

[0155] Step 402: Call the classification encoder and classification fully connected layer to extract features and classify the data samples, obtain the classification vector of the data samples, and determine the correlation between the data samples and the classification vector; the classification vector includes the category vector and the intra-class style vector.

[0156] The computer device acquires a data sample set. The data sample set includes data samples, which are images. An image classification model is pre-built in the computer device. This model includes an encoder and fully connected layers following the encoder. The image classification model can employ various neural networks. For example, it can use a convolutional neural network, where the convolutional blocks can be adjusted according to the image size. Larger images require larger convolutional blocks. For instance, a 32x32 image might require two convolutional blocks, while a 96x96 image might require four.

[0157] The computer device inputs data samples into an image classification model. The encoder of the image classification model extracts features from the data samples to obtain sample features. The fully connected layer outputs a classification vector for the data sample based on the sample features. The classification vector includes a class vector and a within-class style vector. The elements in the class vector represent the probability that the data sample belongs to each classification target. The within-class style vector describes the within-class style information of the data sample. The computer device can use other network models to train the image classification model. Specifically, the computer device can use a discriminator to determine the correlation between the data sample and the classification vector (the data sample corresponds to the classification vector obtained from the data sample), and use an evaluator to determine the score value of the classification vector according to the prior distribution. The overall network structure diagram for training the image classification model by the computer device can be shown as follows: Figure 5 As shown.

[0158] The discriminator 501 is a deep neural network composed of multiple fully connected layers. For example, it can be a deep neural network composed of three or more fully connected layers.

[0159] The evaluator 502 is a deep neural network composed of multiple fully connected layers. It can be a deep neural network composed of three or more fully connected layers.

[0160] Discriminator 501 determines whether there is a correlation between a data sample and a classification vector, thereby maximizing the mutual information between them. The computer device can simultaneously input data samples and extracted classification vectors into discriminator 501. The data samples include a first sample and a second sample. When the data sample input to discriminator 501 is the first sample, and the extracted classification vector originates from the second sample, and the first and second samples are different, then the first sample and the classification vector are negative samples, and discriminator 501 determines that they are uncorrelated. When the data sample input to discriminator 501 is the first sample, and the extracted classification vector originates from the first sample, then the first sample and the extracted classification vector are positive samples, and discriminator 501 determines that they are correlated. Figure 5 In this algorithm, shoe image 503 can be used as the first sample, and clothing image 504 can be used as the second sample. The encoder and fully connected layer obtain a first classification vector based on the first input sample and a second classification vector based on the second input sample. The first sample is related to the first classification vector, and the first sample is not related to the second classification vector. When the discriminator 501 can correctly determine whether the data sample is related to the classification vector, it means that the classification vector contains information related to the data sample, thereby maximizing mutual information.

[0161] Step 403: Introduce a category prior distribution to the category vector and an intra-class style prior distribution to the intra-class style vector to determine the score value of the classification vector that follows the prior distribution.

[0162] The evaluator introduces a prior distribution for the classification vector. The evaluator is also a deep neural network composed of multiple fully connected layers. It can be a deep neural network consisting of three or more fully connected layers.

[0163] Prior distributions include class prior distribution and within-class style prior distribution. The class prior distribution can be simply referred to as the class distribution, while the within-class style prior distribution can be a Gaussian distribution. The evaluator is the class vector z. c Introducing category distribution For the in-class style vector z s Introducing Gaussian distribution This allows for effective decoupling of category vectors from intra-class style vectors.

[0164] When the classification vector follows a prior distribution, the output category feature part becomes a one-hot vector. The element with the largest value in the one-hot vector can be directly used to represent the category of the data sample, avoiding the need for further classification operations. Simultaneously, it prevents data samples from being clustered into only one or a few categories, ensuring that the required number of categories are reached, such as 10.

[0165] Step 404: Train an image classification model based at least on the relevance and score values.

[0166] Computer equipment can use the correlation between data samples and classification vectors, and the score values ​​of classification vectors following a prior distribution, to perform backpropagation optimization on the network parameters of an image classification model. Backpropagation can be used to optimize the network parameters of the image classification model. For example, backpropagation can employ gradient descent based on Adam. During backpropagation optimization, the weights of the network parameters of the image classification model, discriminator, and evaluator can be updated. During training, the learning rate is 0.0001, and the parameters β1 and β2 controlling the convergence of the loss function are set to 0.5 and 0.9 respectively. The batch size is set to 64. During backpropagation, the same batch of data samples can be used to alternately optimize the evaluator, image classification model, and discriminator. When the loss function of the evaluator begins to converge, it indicates that the classification vectors learned by the image classification model have approached the prior distribution, and training can be stopped.

[0167] In one embodiment, the training method for the image classification model further includes: augmenting the data samples and mapping the augmented classification vectors through the image classification model; the augmented classification vectors include augmented class vectors and augmented intra-class style vectors; determining the class feature differences between the class vectors and the augmented class vectors; training the image classification model based at least on relevance and score values ​​includes: training the image classification model based on relevance, class feature differences, and score values.

[0168] Computer equipment inputs data samples into an image classification model, which then processes them through an encoder and a fully connected layer to obtain corresponding classification vectors. These classification vectors include a category vector and a within-class style vector. The category vector is the vector activated by the Softmax function; its elements represent the probability that a data sample belongs to each classification target, and the vector dimension is set to the number of classification targets. The within-class style vector is the vector after linear activation. This vector describes the within-class style information of the data sample, and its dimension can be a preset number, such as 50. For example, within-class style information refers to the stylistic differences between images belonging to the same classification target. The category vector and the within-class style vector will have different values ​​after different activations, but some information may be mixed together. By introducing a category prior distribution for the category vector and an within-class style prior distribution for the within-class style vector, the category vector and the within-class style vector can be effectively decoupled.

[0169] Since data samples of the same type can have different styles, and changes in style do not alter the classification target of the data sample, this embodiment, based on the phenomenon that specific data augmentation does not change the classification target of a data sample, performs data augmentation processing on the data samples and trains the system so that the augmented classification vector remains unchanged. The computer device performs augmentation processing on the data samples, including random cropping, random horizontal flipping, color jittering, and random combination of color channels. The augmented data samples are input into an image classification model, passing through an encoder and a fully connected layer to obtain the augmented classification vector. The computer device extracts the class vector from the classification vector and the augmented class vector from the augmented classification vector. The class vector and the augmented class vector are input into an evaluator, which identifies the class feature differences between the class vector and the augmented class vector. The elements in the class vector represent the probability values ​​of the data sample belonging to each classification target. The class feature differences between the class vector and the augmented class vector can be measured using divergence.

[0170] Computer devices can backpropagate image classification model network parameters by utilizing the correlation between data samples and classification vectors, the score values ​​of classification vectors following a prior distribution, and the differences in class features between the class vectors and the augmented class vectors. During backpropagation, gradient descent is used to update the weight values ​​corresponding to the network parameters of the image classification model, discriminator, and evaluator. This enables the image classification model to learn that classification vectors are related to data samples, that class vectors can represent the classification target of data samples, and that intra-class style vectors can represent the differences between data samples of the same class. After data augmentation, the class vectors of data samples remain unchanged, meaning that the style of data samples may change to some extent, but they still belong to the same class. Moreover, due to the constraint of the prior distribution, the class vectors can be made as close as possible to the one-hot vectors, that is, most elements have values ​​close to 0, and only one element has a value close to 1, thus enabling the classification target corresponding to the data sample to be directly determined based on the class vectors.

[0171] In one embodiment, the data samples include a first sample and a second sample; determining the correlation between the data samples and the classification vector includes: obtaining the first sample, concatenating the classification vector of the first sample with the first sample vector to generate a concatenated first sample vector (positive sample); concatenating the classification vector of the second sample with the first sample vector to generate a concatenated second sample vector (negative sample); training the discriminator to output a "correlated" result based on the first sample vector and an "irrelevant" result based on the second sample vector.

[0172] The data samples include a first sample and a second sample, which can be completely different or the same. The first sample is input into an image classification model, mapping to a first classification vector corresponding to the first sample. The second sample is input into the image classification model, mapping to a second classification vector corresponding to the second sample. Both the first and second classification vectors can be multi-dimensional vectors, such as 50-dimensional vectors. The computer device converts the first sample into a first sample vector. The computer device concatenates the first classification vector and the first sample vector to generate a concatenated first sample vector. The concatenation can be done by adding the first sample vector after the first classification vector or vice versa. The computer device can also concatenate the second classification vector and the first sample vector using the same method to generate a concatenated second sample vector. The concatenated first sample vector is input into a discriminator. If the discriminator determines that the two are correlated, it outputs 1; if it determines that they are not correlated, it outputs 0. Similarly, the concatenated second sample vector is input into the discriminator. If the discriminator determines that the two are correlated, it outputs 1; if it determines that they are not correlated, it outputs 0. When the discriminator can correctly determine whether there is a correlation between the data sample and the classification vector, it means that the classification vector contains information related to the data sample, thus maximizing mutual information. This enables the classification vector learned by the image classification model to be related to the data sample.

[0173] In one embodiment, introducing a category prior distribution to the category vector and an intra-class style prior distribution to the intra-class style vector to determine the score value of the classification vector following the prior distribution includes: introducing a category prior distribution to the category vector through an evaluator to obtain the category distribution result of the category vector; introducing an intra-class style prior distribution to the intra-class style vector through an evaluator to obtain the intra-class style prior distribution result of the intra-class style vector; and scoring the category distribution result and the intra-class style prior distribution result through an evaluator to obtain the score value of the classification vector following the prior distribution.

[0174] The evaluator introduces a prior distribution for the classification vectors. This prior distribution includes a class prior distribution and a within-class style prior distribution. The class prior distribution can be simply referred to as the class distribution, while the within-class style prior distribution can be a Gaussian distribution. in, Let Cat be the distribution of the class vectors, representing the class distribution (a one-hot vector), K be the number of classification targets, and P be the reciprocal of K. The within-class style vector can be... σ represents the distribution of style vectors within a class, where N is a Gaussian distribution and σ is the standard deviation. σ can be a preset value, such as 0.1.

[0175] The computer device simultaneously inputs the category vector and the within-class style vector into the evaluator. The evaluator outputs the category distribution result corresponding to the category vector and the Gaussian distribution result corresponding to the within-class style vector, respectively. The category distribution result can be a category vector, which can be a one-hot vector. The Gaussian distribution result can be a style vector.

[0176] In one embodiment, scoring the category distribution result and the intra-class style prior distribution result by an evaluator includes: concatenating the category distribution vector of the category vector with the Gaussian distribution vector of the intra-class style vector to generate a prior distribution vector; and scoring the prior distribution vector by an evaluator to obtain a score value for the classification vector following the prior distribution.

[0177] The computer concatenates the category results with the Gaussian distribution results, that is, concatenates the corresponding category vector with the style vector. The concatenation method can be either adding an element of the style vector after the last element of the category vector, or vice versa. The evaluator scores the concatenated vector, obtaining a score that represents the probability that the classification vector follows the prior distribution. A higher probability indicates that the classification vector closely follows the prior distribution. When the classification vector follows the prior distribution, the output category vector is as close as possible to the one-hot vector, allowing the largest element in the one-hot vector to directly represent the category of the data sample, avoiding the need for further classification. Furthermore, following the prior distribution prevents data samples from being classified into only one or a few classes, ensuring that data samples are assigned to the desired classification target.

[0178] In one embodiment, the training method for the image classification model further includes: determining the correlation between data samples and classification vectors through a discriminator; determining the score value of the classification vectors following a prior distribution through an evaluator; and training the image classification model based at least on the correlation and score value by: alternately optimizing the image classification model, the discriminator, and the evaluator based at least on the correlation and score value.

[0179] A discriminator identifies the correlation between data samples and classification vectors. The loss function used by the discriminator to identify the correlation between data samples and classification vectors can be called the mutual information loss function. The discriminator can be trained using the mutual information loss function. The mutual information loss function can be represented as follows:

[0180]

[0181] Where X is the data sample, Z is the classification vector, S is the sigmoid function, E represents the expectation, D is the discriminator used to determine whether X and Z are related, and Q(Z|X) is the posterior distribution of Z obtained by the image classification model mapping; P X Given the prior distribution of the input image, Let Z be the aggregate posterior distribution. Indicate that X and Z follow the order Q(Z|X)P. X The expected value of (X). When X and Z are positive samples. When X and Z are negative samples

[0182] During the training of the discriminator using the mutual information loss function, the smaller the loss function value, the more accurate the correlation judgment. This also means that during backpropagation, the impact on the weights of each layer in the discriminator network is smaller. When the discriminator can correctly determine whether a data sample is correlated with a feature, it indicates that the feature contains information related to the data sample, thus maximizing mutual information.

[0183] The difference in class features between the class vector and the enhanced class vector can be measured by divergence. The divergence can be KL divergence. The corresponding loss function can be called the class difference loss function, and it uses the following formula:

[0184] L Aug =KL(Q(Z) c |X)||Q(Z c |T(X)))

[0185] Where KL is the KL divergence, Q is the image classification model, and Z is the KL divergence. c Let X be the class vector, T be the data sample, and Q(Z) be the data augmentation vector. c |X) is Z c The aggregate posterior distribution, Q(Z) c |T(X)) is the posterior distribution of the enhanced classification vector.

[0186] The smaller the value of the category difference loss function, the smaller the difference in category features between the category vector and the augmented category vector. Consequently, the probability of the category vector changing after data augmentation is smaller.

[0187] An evaluator scores the classification vectors based on a prior distribution. A loss function representing the prior distribution is introduced for the classification vectors; this is called the prior distribution loss function. Different prior distribution loss functions can be defined for the image classification model and the evaluator. The prior distribution loss function ensures that the classification vectors mapped by the image classification model closely approximate the prior distribution. The prior distribution loss function for the image classification model can be as follows:

[0188]

[0189] Where Q is the image classification model, Z is the classification vector of the data sample, C(Z) is the probability value of whether the classification vector follows a prior distribution, and Q ZIt is the aggregate posterior distribution of Z. Let Z satisfy the mathematical expectation of QZ[C(Z)].

[0190] The prior distribution loss function of the evaluator can be shown below:

[0191]

[0192] Where C is the evaluator, P Z For the prior distribution, To obtain from the prior distribution P Z and aggregated posterior distribution Q Z The features sampled are related to the features on the connection line. λ is the gradient penalty term, used to ensure that the evaluator C satisfies the Lipschitz constraint, so that the changes in its evaluation score, i.e. the probability of following the prior distribution, are not too drastic. λ is the coefficient of the gradient penalty term, set to 10.

[0193] In one embodiment, the mutual information loss function, the class difference loss function, and the prior distribution loss function of the image classification model can be used as sub-loss functions to define the total loss function of the image classification model. Each sub-loss function can have corresponding weights. The total loss function of the discriminator can be defined using the mutual information loss function and its corresponding weights. The total loss function of the evaluator can be defined using the prior distribution loss function and its weights.

[0194] The total loss function of the image classification model is as follows:

[0195] The total loss function of the discriminator is as follows:

[0196]

[0197] The total loss function of the discriminator is as follows:

[0198] L D =β MI L MI ;

[0199] The total loss function of the evaluator is as follows:

[0200]

[0201] Among them, L Q L represents the total loss function for the image classification model. MI Let L be the mutual information loss function. Aug For the category difference loss function, Let β be the prior distribution loss function of the image classification model. MI For L MI The weights, β Aug For LAug The weights, β Adv for The weight of β. MI β Adv It can be set to a corresponding fixed value, for example, β. MI Set to 0.5, β Adv Set it to 1. β Aug Related to the dataset of data samples, it can be set up in the following way. Specifically, computer equipment can perform non-linear dimensionality reduction on the classification vectors to generate a corresponding visual dimensionality reduction map, and select the weights of the class difference loss function based on the visual dimensionality reduction map. A visual dimensionality reduction map is the result of reducing high-dimensional data to low-dimensional data, making the result visible. Low-dimensionality can be two-dimensional or three-dimensional. For example, t-SNE can be used to perform non-linear dimensionality reduction on the classification vectors, and a visual dimensionality reduction map, i.e., a t-SNE map, can be generated based on the processing result. In the t-SNE map, the data samples are classified into clusters, and in β... Aug When the value of β is low, the classification clusters of the data samples are relatively dispersed; as β increases... Aug As the value of β increases, features tend to cluster, and class clusters may even overlap. Different data types of datasets will produce different classification results. Taking images as an example, at β... Aug When β = 2, the taxonomic clusters in the t-SNE diagram do not overlap. Aug When β = 3, overlapping clusters appear in the t-SNE diagram. Therefore, the maximum overlap between 2 and 3 can be selected as β. Aug The value of can make the total loss function of the image classification model more accurate, thereby making the classification results of the trained image classification model more accurate.

[0202] Image classification model training can be performed using backpropagation. During backpropagation, the evaluator, image classification model, and discriminator can be optimized alternately. First, the evaluator is optimized, followed by the image classification model and discriminator. Specifically, the evaluator's total loss function is used to backpropagate it, making its probability of classifying vectors following the prior distribution close to 1, and its probability of classifying vectors not following the prior distribution close to 0. Then, the image classification model's total loss function and the discriminator's total loss function are used to backpropagate them, aiming to maximize the scores of the classifying vectors output by the image classification model, i.e., maximizing the probability that the classifying vectors follow the prior distribution. This alternating optimization process is repeated until the image classification model outputs high scores, i.e., the probability that the classifying vectors follow the prior distribution is close to 1, thus ensuring that the model follows the prior distribution.

[0203] In one embodiment, alternating optimization of the image classification model, discriminator, and evaluator based at least on relevance and score values ​​includes: first optimizing the network parameters of the evaluator at least once based on the score values; then optimizing the network parameters of the image classification model at least based on relevance and score values; and optimizing the network parameters of the discriminator based on relevance.

[0204] Specifically, due to the large number of data samples, it is not possible to input all the data samples into the image classification model for training at once. During backpropagation, the data samples can be randomly divided into multiple batches, with each batch using a fixed number of data samples, also known as batch samples. For example, the batch sample can be set to 64 data samples, i.e., the batch size is set to 64.

[0205] During training, the computer determines the score values ​​of the classification vectors according to a prior distribution and the correlation between the data samples and the classification vectors. When alternately optimizing the image classification model, discriminator, and evaluator, the weights corresponding to each network parameter are updated. First, the network parameters of the evaluator are optimized at least once based on the score values ​​of the classification vectors according to the prior distribution and the total loss function of the evaluator. Then, the network parameters of the image classification model are optimized based on the correlation between the data samples and the classification vectors, the score values ​​of the classification vectors according to the prior distribution, the differences in class features, and the total loss function of the image classification model. Similarly, the network parameters of the discriminator are optimized based on the correlation between the data samples and the classification vectors and the total loss function of the discriminator. For example, the evaluator is optimized four times first, followed by one optimization of the image classification model and the discriminator. When performing back-optimization on the image classification model and the discriminator, back-optimization can be performed sequentially or simultaneously.

[0206] When backpropagating the evaluator, for inputs from the prior distribution, the closer its output is to 1, the smaller the loss function value and the smaller the change in parameters during backpropagation. Conversely, for inputs from data samples, the closer its output is to 0, the smaller the loss function value and the smaller the change in parameters during backpropagation. When backpropagating an image classification model, the closer its output is to 1, the smaller the loss function value and the smaller the change in parameters during backpropagation. Prior distribution is not considered when backpropagating an image classification model. During backpropagation, the total loss function of the evaluator indicates the difference between the feature distribution learned by the current image classification model and the prior distribution. When the total loss function of the evaluator begins to converge, it means that the feature distribution learned by the image classification model has approached the prior distribution, and training can be stopped.

[0207] In summary, the method provided in this embodiment does not require additional classification algorithms or the generation of real images for comparison with original images for data samples in classification operations. It determines the correlation between data samples and classification vectors, and introduces a prior distribution for the category vector and a prior distribution for the intra-class style vector to determine the score value that the classification vector follows the prior distribution. By using correlation and score to train the image classification model, the learning of classification vectors by the image classification model can be effectively improved. Since the feature distribution learned by the image classification model is close to the prior distribution, and the category vector and intra-class style vector are effectively decoupled, the classification target corresponding to the data sample can be obtained based on the category vector. This achieves an effective improvement in data classification accuracy without manual annotation. Furthermore, it enables effective training of image segmentation networks without manual annotation.

[0208] For example, such as Figure 6 The present invention provides a schematic diagram illustrating a training method for training an image segmentation model using the image segmentation model provided in this application.

[0209] Figure 6 The encoder 102 in the middle adopts Figure 4 In the embodiment shown, the encoder of the image classification model has a fully connected layer (not shown in the figure) after the encoder 102 during the training phase of the image segmentation model.

[0210] During training, a sample image x1 is first acquired. For the sample image x1, it is upsampled (generally the length and width are magnified to twice the original size) to obtain the scaled image x2 = R(x1).

[0211] The sample image x1 and the scale image x2 are each input into the encoder, and the encoder outputs the sample image features of the sample image x1 and the scale image features of the scale image x2.

[0212] Input sample image features into a fully connected layer to output sample classification results, and input scale image features into a fully connected layer to output scale classification results.

[0213] Based on the sample classification results and sample image features, the sample class activation map C1 of the sample image is calculated according to the formula for calculating class activation maps. Based on the scale classification results and scale image features, the scale class activation map C2 of the scale image is calculated according to the formula for calculating class activation maps.

[0214] The sample image features are input into the decoder 103 to obtain the sample segmentation result s1. The scale image features are input into the decoder 103 to obtain the scale segmentation result s2.

[0215] Two constraints are introduced: first, the segmentation result should be as close as possible to the class activation map; second, the target regions (regions containing the classification target) segmented after scale transformation should remain consistent after the same scale transformation. Based on this approach, the sample class activation map loss is calculated based on the sample class activation map and the sample segmentation result; the scale class activation map loss is calculated based on the scale class activation map and the scale segmentation result; and the scale loss is calculated based on the sample segmentation result and the scale segmentation result. The weighted sum of the sample class activation map loss, the scale activation map loss, and the scale loss yields the total loss L. seg .

[0216] Based on the total loss L seg The network parameters in the decoder are optimized using back propagation. The optimization method employs Adam-based gradient descent with a learning rate of 0.0001. The exponential decay rate β1 for the first moment estimation of Adam's parameters is set to 0.5, and the exponential decay rate β2 for the second moment estimation is set to 0.9. As mentioned earlier, the image classification model is optimized first, then the encoder part is fixed and transferred to the image segmentation model, where the decoder part of the image segmentation model is further optimized.

[0217] For example, after training an image segmentation model, an exemplary embodiment of using the trained image segmentation model for image segmentation is given.

[0218] Figure 7 A flowchart illustrating an exemplary embodiment of the image segmentation method provided in this application is shown. This method can be executed by a computer device, such as... Figure 1 The method is executed using a terminal or server as shown. For example, the computer device executing the training method for the image segmentation model and the computer device executing the image segmentation method can be the same computer device or different computer devices. The method includes the following steps.

[0219] Step 701: Obtain the input image.

[0220] For example, the input image can be any image that needs to be segmented.

[0221] For example, when an image segmentation model is trained to segment faces in an image, the input image can be an image containing faces; when an image segmentation model is trained to segment lesions in an image, the input image can be an image containing lesions. Of course, the input image may not contain the classification target of the image segmentation model; that is, the input image may not contain faces or lesions.

[0222] Step 702: Call the encoder to extract features from the input image to obtain the input image features.

[0223] For example, the encoder is the encoder in the image segmentation model mentioned in any of the above embodiments.

[0224] For example, the source of the encoder varies depending on the classification target of the image segmentation model. When the classification target of the image segmentation model is a face, the encoder parameters are set according to the parameters of the classification encoder of the face classification model, which is used to identify whether the input image contains a face.

[0225] When the classification target of the image segmentation model is a lesion, the encoder parameters are set according to the classification encoder parameters of the lesion classification model, which is used to identify whether the input image contains a lesion.

[0226] By using the classification encoder of the image classification model to set the encoder of the image segmentation model, the encoder of the image segmentation model can accurately extract the features of the image in the classification task. Since the classification target of image segmentation is the same as that of image classification, the image segmentation model can perform accurate image segmentation based on the extracted features, which simplifies the training process of the image segmentation model and improves the segmentation accuracy of the image segmentation model.

[0227] Step 703: Call the decoder to decode the features of the input image to obtain the image segmentation result of the input image. The decoder is trained based on class activation map loss and scale loss. Class activation map loss is used to train the decoder to output segmentation results close to the class activation map. The class activation map is used to represent the degree of influence of each pixel in the image on the classification result of the image. Scale loss is used to train the decoder to output similar segmentation results for multiple images with the same image content but different scales.

[0228] For example, the decoder is the decoder in the image segmentation model mentioned in any of the above embodiments. The training method of the decoder can refer to the above embodiments.

[0229] The segmentation result output by the encoder includes the probability value of each pixel in the input image belonging to each classification target, or the segmentation result output by the encoder includes the classification target to which each pixel in the input image belongs.

[0230] For example, when the classification target of the image segmentation model is a human face, the image segmentation result includes the probability value of each pixel in the input image being a human face, or the image segmentation result includes whether each pixel in the input image is a human face.

[0231] For example, when the classification target of an image segmentation model is a lesion, the image segmentation result includes the probability value of each pixel in the input image being a lesion, or the image segmentation result includes whether each pixel in the input image is a lesion.

[0232] In summary, the method provided in this embodiment, by using a trained image segmentation model to segment the input image and employing a neural network for feature extraction and segmentation, can segment images based on deep features, thus improving the accuracy of image segmentation. Furthermore, because the decoder of the image segmentation model is trained based on scale invariance, the model outputs similar segmentation results for multiple images with the same content but different scales, more closely reflecting the actual situation of image segmentation and further improving the accuracy.

[0233] For example, the image segmentation model provided in this application can be used to segment different classification targets in images in various application scenarios. Taking the aforementioned application scenarios of face recognition and lesion recognition as examples, this application also provides exemplary embodiments of training corresponding image segmentation models for these two application scenarios.

[0234] An exemplary embodiment of an image segmentation model for segmenting face regions in an image is provided, using a training method of the image segmentation model provided in this application.

[0235] First, an image classification model for face recognition is trained. This model consists of a classification encoder and a fully connected classification layer. The image classification model is used for clustering, that is, grouping multiple input images into a class containing faces and a class not containing faces.

[0236] The first step is to obtain data samples.

[0237] The second step involves calling a classification encoder and a fully connected classification layer to extract features and classify the data samples, obtaining classification vectors for each data sample, and determining the correlation between the data sample and the classification vector. The classification vector includes a class vector and an within-class style vector; the class vector describes whether the input image contains a face.

[0238] The third step is to introduce a category prior distribution into the category vector and a René style prior distribution into the within-class style vector to determine the score value of the classification vector that follows the prior distribution.

[0239] The fourth step is to train an image classification model based at least on the relevance and score values.

[0240] Then, an image segmentation model is trained based on the already trained image classification model. The image segmentation model includes an encoder, a fully connected layer, and a decoder during the training phase, and an encoder and a decoder during the application phase.

[0241] The first step is to initialize the encoder based on the parameters of the classification encoder and the fully connected layer based on the parameters of the classification fully connected layer. That is, the encoder and the fully connected layer can accurately classify the input image and identify whether the image contains a human face.

[0242] The second step is to obtain the training dataset, which includes at least one set of sample images and scale images. The sample images are images that may or may not include human faces, and the scale images are images obtained by upsampling the sample images. The size of the scale images is twice that of the sample images.

[0243] The third step involves calling the encoder to extract features from the sample image, obtaining sample image features, and then calling a fully connected layer to classify the sample image features to obtain the sample classification result, which includes the probability value that the sample image contains a face. The encoder is then called to extract features from the scale image to obtain scale image features, and a fully connected layer is called to classify the scale image features to obtain the scale classification result, which includes the probability value that the scale image contains a face.

[0244] The fourth step involves calling the decoder to decode the features of the sample image, obtaining the sample segmentation result. The sample segmentation result includes the probability value of each pixel in the sample image being a face. The next step is to call the decoder to decode the features of the scale image, obtaining the scale segmentation result. The scale segmentation result includes the probability value of each pixel in the scale image being a face.

[0245] The fifth step is to calculate the sample class activation map of the sample image based on the sample image features and sample classification results. The scale class activation map of the scale image is calculated based on the scale image features and scale classification results.

[0246] Step 6: Calculate the sample class activation map loss and the sample segmentation result's sample class activation map loss, the scale class activation map loss and the scale segmentation result's scale loss, and the sample segmentation result's scale loss. Calculate the weighted sum of the sample class activation map loss, scale activation map loss, and scale loss to obtain the total loss.

[0247] Step 7: Train the decoder based on the total loss.

[0248] Step 8: Repeat steps 3 through 7 to iteratively train the decoder and obtain the final image segmentation model.

[0249] The ninth step is to use the trained image segmentation model to segment the face region in the image.

[0250] An exemplary embodiment of an image segmentation model for segmenting lesion regions in an image is provided, using a training method of the image segmentation model provided in this application.

[0251] First, an image classification model for lesion identification is trained. This model includes a classification encoder and a fully connected classification layer. The image classification model is used for clustering, that is, clustering multiple input images into a class containing lesions and a class not containing lesions. For example, the image classification model can also be used to identify several specific types of lesions, such as clustering multiple input images into lesion one, lesion two, lesion three, and normal.

[0252] The first step is to obtain data samples.

[0253] The second step involves calling a classification encoder and a fully connected classification layer to extract features and classify the data samples, obtaining classification vectors for the data samples, and determining the correlation between the data samples and the classification vectors. The classification vectors include a category vector and an within-class style vector. The category vector describes whether the input image contains lesions, while the style vector describes the probability value of the input image belonging to a particular type of lesion.

[0254] The third step is to introduce a category prior distribution into the category vector and a René style prior distribution into the within-class style vector to determine the score value of the classification vector that follows the prior distribution.

[0255] The fourth step is to train an image classification model based at least on the relevance and score values.

[0256] Then, an image segmentation model is trained based on the already trained image classification model. The image segmentation model includes an encoder, a fully connected layer, and a decoder during the training phase, and an encoder and a decoder during the application phase.

[0257] The first step is to initialize the encoder according to the parameters of the classification encoder, and initialize the fully connected layer according to the parameters of the classification fully connected layer. That is, the encoder and the fully connected layer can accurately classify the input image, identify whether the image contains lesions, or identify which type of lesion the image belongs to.

[0258] The second step is to obtain the training dataset, which includes at least one set of sample images and scale images. The sample images are images with or without lesions, and the scale images are images obtained by upsampling the sample images. The size of the scale images is twice that of the sample images.

[0259] The third step involves calling the encoder to extract features from the sample image, obtaining sample image features, and then calling a fully connected layer to classify these features to obtain the sample classification result. The sample classification result includes the probability value that the sample image contains lesion regions, or the probability value that the sample image belongs to each type of lesion. Similarly, the encoder is called to extract features from the scale image, obtaining scale image features, and then a fully connected layer is called to classify these scale image features to obtain the scale classification result. The scale classification result includes the probability value that the scale image contains lesions, or the probability value that the scale image belongs to each type of lesion.

[0260] The fourth step involves calling the decoder to decode the features of the sample image to obtain the sample segmentation result. The sample segmentation result includes the probability value of each pixel in the sample image being a lesion, or the probability value of each pixel in the sample image belonging to each lesion. The same step is repeated to decode the features of the scale image to obtain the scale segmentation result. The scale segmentation result includes the probability value of each pixel in the scale image being a lesion, or the probability value of each pixel in the scale image belonging to each lesion.

[0261] The fifth step is to calculate the sample class activation map of the sample image based on the sample image features and sample classification results. The scale class activation map of the scale image is calculated based on the scale image features and scale classification results.

[0262] Step 6: Calculate the sample class activation map loss and the sample segmentation result's sample class activation map loss, the scale class activation map loss and the scale segmentation result's scale loss, and the sample segmentation result's scale loss. Calculate the weighted sum of the sample class activation map loss, scale activation map loss, and scale loss to obtain the total loss.

[0263] Step 7: Train the decoder based on the total loss.

[0264] Step 8: Repeat steps 3 through 7 to iteratively train the decoder and obtain the final image segmentation model.

[0265] The ninth step is to use the trained image segmentation model to segment the lesion region in the image.

[0266] The following are device embodiments of this application. For details not described in detail in the device embodiments, please refer to the corresponding descriptions in the above method embodiments. They will not be repeated here.

[0267] Figure 8A schematic diagram of a training apparatus for an image segmentation model provided in an exemplary embodiment of this application is shown. This apparatus can be implemented as all or part of a computer device through software, hardware, or a combination of both. The image segmentation model includes an encoder and a decoder. The apparatus includes:

[0268] The encoding module 601 is used to call the encoder to extract features from the sample image and the scale image to obtain the sample image features of the sample image and the scale image features of the scale image. The scale image includes at least one of the following: an image obtained by enlarging the sample image or an image obtained by shrinking the sample image.

[0269] The class activation map module 602 is used to calculate a class activation map based on the features of the sample image to obtain a sample class activation map of the sample image, and to calculate a class activation map based on the features of the scale image to obtain a scale class activation map of the scale image; the class activation map is used to represent the degree of influence of each pixel in the image on the classification result of the image;

[0270] The decoding module 603 is used to call the decoder to decode the features of the sample image to obtain the sample segmentation result of the sample image, and to call the decoder to decode the features of the scale image to obtain the scale segmentation result of the scale image; the sample segmentation result includes the classification probability value of each pixel in the sample image;

[0271] Loss module 604 is used to calculate class activation map loss based on the sample class activation map, the sample segmentation result, the scale class activation map, and the scale segmentation result, and to calculate scale loss based on the sample segmentation result and the scale segmentation result; the class activation map loss is used to train the decoder to make the sample segmentation result closer to the sample class activation map and the scale segmentation result closer to the scale class activation map; the scale loss is used to train the decoder to make the sample segmentation result closer to the scale segmentation result.

[0272] Training module 605 is used to train the decoder based on the class activation map loss and the scale loss.

[0273] In one optional embodiment, the class activation map loss includes sample class activation map loss and scale class activation map loss;

[0274] The loss module 604 is used to calculate the sample class activation map loss based on the sample class activation map and the sample segmentation result;

[0275] The loss module 604 is used to calculate the scale class activation map loss based on the scale class activation map and the scale segmentation result.

[0276] The loss module 604 is used to calculate the scale loss based on the sample segmentation result and the scale segmentation result.

[0277] In an optional embodiment, the loss module 604 is used to scale the sample segmentation result to the same size as the scale segmentation result according to the scale relationship between the sample image and the scale image, so as to obtain a scaled sample segmentation result;

[0278] The loss module 604 is used to calculate the scale loss based on the error between the scale segmentation result and the scaled sample segmentation result.

[0279] In an optional embodiment, the loss module 604 is used to calculate a first matrix difference between the scaled segmentation result and the scaled sample segmentation result, and to determine the 2-norm of the first matrix difference as the scale loss.

[0280] In an optional embodiment, the loss module 604 is used to determine the cross-entropy of the sample class activation map and the sample segmentation result as the sample class activation map loss;

[0281] The loss module 604 is used to determine the cross-entropy of the scale class activation graph and the scale segmentation result as the scale class activation graph loss.

[0282] In an optional embodiment, the loss module 604 is used to calculate a weighted sum of the sample class activation map loss, the scale class activation map loss, and the scale loss;

[0283] The training module 605 is used to train the decoder based on the weighted sum.

[0284] In an optional embodiment, the image segmentation model further includes a pre-trained fully connected layer; the apparatus further includes:

[0285] The classification module 606 is used to call the fully connected layer to perform classification prediction on the features of the sample image to obtain the sample classification result of the sample image; and to call the fully connected layer to perform classification prediction on the features of the scale image to obtain the scale classification result of the scale image;

[0286] The class activation map module 602 is used to calculate the sample class activation map of the sample image based on the sample image features and the sample classification result;

[0287] The class activation map module 602 is used to calculate the scale class activation map of the scale image based on the scale image features and the scale classification result.

[0288] In one alternative embodiment, the encoder is a pre-trained encoder; the device further includes:

[0289] The initialization module 607 is used to set the parameters of the encoder according to the parameters of the classification encoder in the trained image classification model, wherein the classification target of the image classification model and the image segmentation model are the same.

[0290] In an optional embodiment, the device further includes:

[0291] The initialization module 607 is used to set the parameters of the fully connected layer according to the parameters of the fully connected layer in the trained image classification model, wherein the classification target of the image classification model and the image segmentation model are the same.

[0292] In one optional embodiment, the image classification model includes the classification encoder and the classification fully connected layer; the apparatus further includes:

[0293] The classification training module 608 is used to acquire data samples; call the classification encoder and the classification fully connected layer to perform feature extraction and classification on the data samples to obtain the classification vector of the data samples, and determine the correlation between the data samples and the classification vector; the classification vector includes a category vector and a within-class style vector; introduce a category prior distribution for the category vector and an within-class style prior distribution for the within-class style vector to determine the score value of the classification vector that follows the prior distribution; and train the image classification model based at least on the correlation and the score value.

[0294] Figure 9 A schematic diagram of an image segmentation apparatus provided in an exemplary embodiment of this application is shown. This apparatus can be implemented as all or part of a computer device through software, hardware, or a combination of both. The image segmentation model includes an encoder and a decoder. The apparatus includes:

[0295] Module 1001 is used to acquire the input image;

[0296] The feature extraction module 1002 is used to call the encoder to extract features from the input image to obtain the input image features;

[0297] The image segmentation module 1003 is used to call the decoder to decode the features of the input image to obtain the image segmentation result of the input image. The decoder is trained based on class activation map loss and scale loss. The class activation map loss is used to train the decoder to output segmentation results close to the class activation map. The class activation map is used to represent the degree of influence of each pixel in the image on the classification result of the image. The scale loss is used to train the decoder to output similar segmentation results for multiple images with the same image content but different scales.

[0298] In one optional embodiment, the parameters of the encoder are set according to the parameters of the classification encoder of the face classification model, which is used to identify whether the input image contains a face;

[0299] The image segmentation result includes the probability value of each pixel in the input image being a face.

[0300] In one optional embodiment, the parameters of the encoder are set according to the parameters of the classification encoder of the lesion classification model, which is used to identify whether the input image contains lesions;

[0301] The image segmentation result includes the probability value of each pixel in the input image being a lesion.

[0302] Figure 10 This is a schematic diagram of the structure of a server provided in one embodiment of this application. Specifically, the server 800 includes a Central Processing Unit (CPU) 801, a system memory 804 including Random Access Memory (RAM) 802 and Read-Only Memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the CPU 801. The server 800 also includes a basic input / output system (I / O system) 806 that facilitates the transfer of information between various devices within the computer, and a mass storage device 807 for storing the operating system 813, application programs 814, and other program modules 815.

[0303] The basic input / output system 806 includes a display 808 for displaying information and an input device 809 for inputting user account information, such as a mouse or keyboard. Both the display 808 and the input device 809 are connected to the central processing unit 801 via an input / output controller 810 connected to the system bus 805. The basic input / output system 806 may also include the input / output controller 810 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 810 also provides output to a display screen, printer, or other types of output devices.

[0304] Mass storage device 807 is connected to central processing unit 801 via a mass storage controller (not shown) connected to system bus 805. Mass storage device 807 and its associated computer...

[0305] The readable medium provides non-volatile storage for the server 800. That is, the mass storage device 807 may include computer-readable media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0306] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 804 and mass storage device 807 described above can be collectively referred to as memory.

[0307] According to various embodiments of this application, server 800 can also be connected to a remote computer on a network, such as the Internet. That is, server 800 can be connected to network 812 via network interface unit 811 connected to system bus 805, or it can use network interface unit 811 to connect to other types of networks or remote computer systems (not shown).

[0308] This application also provides a terminal, which includes a processor and a memory. The memory stores at least one instruction, which is loaded and executed by the processor to implement the image segmentation model training method or image segmentation method provided in the above-described method embodiments. It should be noted that the terminal can be as follows: Figure 11 The provided terminal.

[0309] Figure 11 This illustration shows a structural block diagram of a terminal 900 provided in an exemplary embodiment of this application. The terminal 900 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 900 may also be referred to as a user account device, portable terminal, laptop terminal, desktop terminal, or other names.

[0310] Typically, terminal 900 includes a processor 901 and a memory 902.

[0311] Processor 901 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0312] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one instruction, which is executed by the processor 901 to implement the image segmentation model training method or image segmentation method provided in the method embodiments of this application.

[0313] In some embodiments, the terminal 900 may also optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, a positioning assembly 908, and a power supply 909.

[0314] Peripheral device interface 903 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 901 and memory 902. In some embodiments, processor 901, memory 902 and peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 901, memory 902 and peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0315] The radio frequency (RF) circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Exemplarily, the RF circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user account identity module card, etc. The RF circuit 904 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0316] Display screen 905 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 905 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 901 for processing. In this case, display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 905, which serves as the front panel of terminal 900; in other embodiments, there may be at least two display screens 905, respectively disposed on different surfaces of terminal 900 or in a folded design; in still other embodiments, display screen 905 may be a flexible display screen, disposed on a curved or folded surface of terminal 900. Furthermore, display screen 905 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 905 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0317] Camera assembly 906 is used to acquire images or videos. Exemplarily, camera assembly 906 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, camera assembly 906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0318] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user account and environment, converting the sound waves into electrical signals that are input to the processor 901 for processing, or input to the radio frequency circuit 904 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 900. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0319] The positioning component 908 is used to determine the current geographic location of the terminal 900 in order to enable navigation or LBS (Location Based Service). The positioning component 908 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, or Galileo system.

[0320] Power supply 909 is used to supply power to the various components in terminal 900. Power supply 909 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 909 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0321] In some embodiments, the terminal 900 further includes one or more sensors 910. The one or more sensors 910 include, but are not limited to: an accelerometer 911, a gyroscope 912, a pressure sensor 913, a fingerprint sensor 914, an optical sensor 915, and a proximity sensor 916.

[0322] Accelerometer 911 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 900. For example, accelerometer 911 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 901 can control display screen 905 to display the user account interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 911. Accelerometer 911 can also be used to collect motion data from games or user accounts.

[0323] The gyroscope sensor 912 can detect the orientation and rotation angle of the terminal 900. The gyroscope sensor 912, in conjunction with the accelerometer sensor 911, can collect 3D motion data from the user account on the terminal 900. Based on the data collected by the gyroscope sensor 912, the processor 901 can perform the following functions: motion sensing (e.g., changing the UI based on the user account's tilting actions), image stabilization during shooting, game control, and inertial navigation.

[0324] The pressure sensor 913 can be installed on the side bezel of the terminal 900 and / or on the lower layer of the display screen 905. When the pressure sensor 913 is installed on the side bezel of the terminal 900, it can detect the user's grip signal on the terminal 900, and the processor 901 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is installed on the lower layer of the display screen 905, the processor 901 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0325] The fingerprint sensor 914 is used to collect the user's fingerprint. The processor 901 identifies the user's account based on the fingerprint collected by the fingerprint sensor 914, or vice versa. When the user's account is identified as a trusted identity, the processor 901 authorizes the user account to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 914 can be located on the front, back, or side of the terminal 900. When the terminal 900 has physical buttons or a manufacturer's logo, the fingerprint sensor 914 can be integrated with the physical buttons or manufacturer's logo.

[0326] An optical sensor 915 is used to collect ambient light intensity. In one embodiment, the processor 901 can control the display brightness of the display screen 905 based on the ambient light intensity collected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the display screen 905 is increased; when the ambient light intensity is low, the display brightness of the display screen 905 is decreased. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 based on the ambient light intensity collected by the optical sensor 915.

[0327] The proximity sensor 916, also known as a distance sensor, is typically located on the front panel of the terminal 900. The proximity sensor 916 is used to detect the distance between the user account and the front of the terminal 900. In one embodiment, when the proximity sensor 916 detects that the distance between the user account and the front of the terminal 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from a screen-on state to a screen-off state; when the proximity sensor 916 detects that the distance between the user account and the front of the terminal 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from a screen-off state to a screen-on state.

[0328] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on terminal 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0329] The memory further includes one or more programs stored in the memory, and the one or more programs include a training method or an image segmentation method for performing the image segmentation model provided in the embodiments of this application.

[0330] This application also provides a computer device, which includes a processor and a memory. The memory storage medium stores at least one instruction, at least one program, code set, or instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the image segmentation model training method or image segmentation method provided in the above-described method embodiments.

[0331] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, which is loaded and executed by a processor to implement the image segmentation model training method or image segmentation method provided in the above-described method embodiments.

[0332] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the training method or image segmentation method of the image segmentation model provided in the above-described optional implementation.

[0333] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0334] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0335] The above are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for an image segmentation model, characterized in that, The image segmentation model includes an encoder and a decoder, wherein the encoder is a pre-trained encoder, and the method includes: Obtain data samples; The data samples are processed by calling a classification encoder and a classification fully connected layer to extract features and classify them, thereby obtaining classification vectors for the data samples and determining the correlation between the data samples and the classification vectors. The classification vectors include category vectors and intra-class style vectors. The elements in the category vectors represent the probabilities of the data samples belonging to each classification target, and the intra-class style vectors describe the intra-class style information of the data samples. A category prior distribution is introduced for the category vector, and an intra-class style prior distribution is introduced for the intra-class style vector to determine the score value of the classification vector that follows the prior distribution; The image classification model is trained based on the correlation and the score value, and the parameters of the encoder are set according to the parameters of the classification encoder in the trained image classification model. The classification target of the image classification model is the same as that of the image segmentation model. The encoder is invoked to extract features from the sample image and the scale image to obtain the sample image features of the sample image and the scale image features of the scale image. The scale image includes at least one of the following: an image obtained by enlarging the sample image and an image obtained by shrinking the sample image. Based on the features of the sample image, a sample class activation map is obtained by calculating the class activation map; based on the features of the scale image, a scale class activation map is obtained by calculating the class activation map; wherein, the sample class activation map includes N class activation maps of the sample image for N classification targets; the scale class activation map includes N class activation maps of the scale image for N classification targets, where N is a positive integer; the class activation map is used to represent the degree of influence of each pixel in the image on the classification result of the image; The decoder is invoked to decode the features of the sample image to obtain the sample segmentation result of the sample image, and the decoder is invoked to decode the features of the scale image to obtain the scale segmentation result of the scale image; the sample segmentation result includes the classification probability value of each pixel in the sample image; The class activation map loss is calculated based on the sample class activation map, the sample segmentation result, the scale class activation map, and the scale segmentation result; the scale loss is calculated based on the sample segmentation result and the scale segmentation result; the class activation map loss is used to train the decoder to make the sample segmentation result closer to the sample class activation map and the scale segmentation result closer to the scale class activation map; the scale loss is used to train the decoder to make the sample segmentation result closer to the scale segmentation result. The decoder is trained based on the class activation graph loss and the scale loss.

2. The method according to claim 1, characterized in that, The class activation graph loss includes sample class activation graph loss and scale class activation graph loss; The calculation of class activation map loss based on the sample class activation map, the sample segmentation result, the scale class activation map, and the scale segmentation result, and the calculation of scale loss based on the sample segmentation result and the scale segmentation result, include: The sample class activation map loss is calculated based on the sample class activation map and the sample segmentation result; The scale class activation map loss is calculated based on the scale class activation map and the scale segmentation result. The scale loss is calculated based on the sample segmentation results and the scale segmentation results.

3. The method according to claim 2, characterized in that, The calculation of the scale loss based on the sample segmentation result and the scale segmentation result includes: Based on the scale relationship between the sample image and the scale image, the sample segmentation result is scaled to the same size as the scale segmentation result to obtain the scaled sample segmentation result; The scale loss is calculated based on the error between the scaled segmentation result and the scaled sample segmentation result.

4. The method according to claim 2, characterized in that, The calculation of the sample class activation map loss based on the sample class activation map and the sample segmentation result includes: The cross-entropy of the sample class activation map and the sample segmentation result is determined as the sample class activation map loss; The calculation of the scale-class activation map loss based on the scale-class activation map and the scale segmentation result includes: The cross-entropy of the scale-class activation graph and the scale segmentation result is determined as the scale-class activation graph loss.

5. The method according to any one of claims 2 to 4, characterized in that, Training the decoder based on the class activation map loss and the scale loss includes: Calculate the weighted sum of the sample class activation map loss, the scale class activation map loss, and the scale loss; The decoder is trained based on the weighted sum.

6. The method according to any one of claims 1 to 4, characterized in that, The image segmentation model further includes a pre-trained fully connected layer; the method further includes: The fully connected layer is invoked to perform classification prediction on the features of the sample image to obtain the sample classification result of the sample image; the fully connected layer is invoked to perform classification prediction on the features of the scale image to obtain the scale classification result of the scale image; The step of calculating the class activation map of the sample image based on the features of the sample image to obtain the sample class activation map includes: The sample class activation map of the sample image is calculated based on the sample image features and the sample classification results; The step of calculating the class activation map based on the scale image features to obtain the scale class activation map of the scale image includes: The scale class activation map of the scale image is calculated based on the scale image features and the scale classification results.

7. The method according to any one of claims 1 to 4, characterized in that, The encoder is a pre-trained encoder; before invoking the encoder to extract features from the sample image and scale image, the process further includes: Based on the parameters of the classification encoder in the trained image classification model, the parameters of the encoder are set, and the classification target of the image classification model and the image segmentation model are the same.

8. The method according to claim 6, characterized in that, The method further includes: Based on the parameters of the fully connected layer in the trained image classification model, the parameters of the fully connected layer are set, and the classification target of the image classification model is the same as that of the image segmentation model.

9. An image segmentation method, characterized in that, The method includes: Obtain the input image; An encoder is invoked to extract features from an input image, obtaining input image features. The encoder is a pre-trained encoder. Data samples are acquired; a classification encoder and a fully connected classification layer are invoked to extract features and classify the data samples, obtaining classification vectors for the data samples, and determining the correlation between the data samples and the classification vectors. The classification vector includes a category vector and a within-class style vector. Elements in the category vector represent the probability that the data sample belongs to each classification target, and the within-class style vector describes the within-class style information of the data sample. A category prior distribution is introduced for the category vector, and an within-class style prior distribution is introduced for the within-class style vector, determining a score value for which the classification vector follows the prior distribution. An image classification model is trained based on the correlation and the score value. The encoder parameters are set according to the parameters of the classification encoder in the trained image classification model. The image classification model and the image segmentation model have the same classification target, and the image segmentation model includes the encoder. The input image features are decoded by a decoder to obtain the image segmentation result of the input image. The decoder is trained based on class activation map loss and scale loss. The class activation map loss is obtained with the goal of making the segmentation result approximate the class activation map and is used to train the decoder to output a segmentation result close to the class activation map. The class activation map is used to represent the degree of influence of each pixel in the image on the classification result of the image. The class activation map includes a sample class activation map of the sample image and a scale class activation map of the scale image. The sample class activation map includes N class activation maps of the sample image for N classification targets. The scale class activation map includes N class activation maps of the scale image for N classification targets, where N is a positive integer. The scale image includes at least one of: an image obtained by enlarging the sample image and an image obtained by shrinking the sample image. The scale loss is obtained with the goal of keeping the segmentation result of the image unchanged after the scale is changed and is used to train the decoder to output similar segmentation results for multiple images with the same image content but different scales.

10. The method according to claim 9, characterized in that, The parameters of the encoder are set according to the parameters of the classification encoder of the face classification model, which is used to identify whether the input image contains a face. The image segmentation result includes the probability value of each pixel in the input image being a face.

11. The method according to claim 9, characterized in that, The parameters of the encoder are set according to the parameters of the classification encoder of the lesion classification model, which is used to identify whether the input image contains lesions; The image segmentation result includes the probability value of each pixel in the input image being a lesion.

12. A training device for an image segmentation model, characterized in that, The image segmentation model includes an encoder and a decoder, wherein the encoder is a pre-trained encoder, and the device includes: An encoding module is used to acquire data samples; call a classification encoder and a fully connected classification layer to extract features and classify the data samples to obtain classification vectors for the data samples, and determine the correlation between the data samples and the classification vectors; the classification vector includes a category vector and a within-class style vector, where the elements in the category vector represent the probability that the data sample belongs to each classification target, and the within-class style vector describes the within-class style information of the data sample; a category prior distribution is introduced for the category vector, and an within-class style prior distribution is introduced for the within-class style vector to determine the score value that the classification vector follows the prior distribution; an image classification model is trained based on the correlation and the score value, and the parameters of the encoder are set based on the parameters of the classification encoder in the trained image classification model, wherein the classification target of the image classification model is the same as that of the image segmentation model; The encoder is invoked to extract features from the sample image and the scale image to obtain the sample image features of the sample image and the scale image features of the scale image. The scale image includes at least one of the following: an image obtained by enlarging the sample image and an image obtained by shrinking the sample image. The class activation map module is used to calculate a sample class activation map based on the features of the sample image to obtain a sample class activation map of the sample image, and to calculate a scale class activation map of the scale image based on the features of the scale image; wherein, the sample class activation map includes N class activation maps of the sample image for N classification targets; the scale class activation map includes N class activation maps of the scale image for N classification targets, where N is a positive integer; the class activation map is used to represent the degree of influence of each pixel in the image on the classification result of the image; The decoding module is used to call the decoder to decode the features of the sample image to obtain the sample segmentation result of the sample image, and to call the decoder to decode the features of the scale image to obtain the scale segmentation result of the scale image; the sample segmentation result includes the classification probability value of each pixel in the sample image; The loss module is used to calculate a class activation map loss based on the sample class activation map, the sample segmentation result, the scale class activation map, and the scale segmentation result, and to calculate a scale loss based on the sample segmentation result and the scale segmentation result; the class activation map loss is used to train the decoder to make the sample segmentation result closer to the sample class activation map and the scale segmentation result closer to the scale class activation map; the scale loss is used to train the decoder to make the sample segmentation result closer to the scale segmentation result. The training module is used to train the decoder based on the class activation graph loss and the scale loss.

13. An image segmentation apparatus, characterized in that, The device includes: The acquisition module is used to acquire the input image; A feature extraction module is used to call an encoder to extract features from an input image, obtaining input image features of the input image. The encoder is a pre-trained encoder. The module includes: acquiring data samples; calling a classification encoder and a fully connected classification layer to extract features and classify the data samples, obtaining classification vectors for the data samples, and determining the correlation between the data samples and the classification vectors; the classification vector includes a category vector and a within-class style vector, where the elements in the category vector represent the probability that the data sample belongs to each classification target, and the within-class style vector describes the within-class style information of the data sample; introducing a category prior distribution for the category vector and an within-class style prior distribution for the within-class style vector, and determining a score value for which the classification vector follows the prior distribution; training an image classification model based on the correlation and the score value; and setting the parameters of the encoder based on the parameters of the classification encoder in the trained image classification model. The image classification model and the image segmentation model have the same classification target, and the image segmentation model includes the encoder. The image segmentation module is used to call the decoder to decode the features of the input image to obtain the image segmentation result of the input image. The decoder is trained based on class activation map loss and scale loss. The class activation map loss is obtained with the goal of making the segmentation result approximate the class activation map and is used to train the decoder to output a segmentation result close to the class activation map. The class activation map is used to represent the degree of influence of each pixel in the image on the classification result of the image. The class activation map includes a sample class activation map of the sample image and a scale class activation map of the scale image. The sample class activation map includes N class activation maps of the sample image for N classification targets. The scale class activation map includes N class activation maps of the scale image for N classification targets, where N is a positive integer. The scale image includes at least one of: an image obtained by enlarging the sample image and an image obtained by shrinking the sample image. The scale loss is obtained with the goal of keeping the segmentation result of the image unchanged after the scale is changed and is used to train the decoder to output similar segmentation results for multiple images with the same image content but different scales.

14. A computer device, the computer device comprising: A processor and a memory, wherein the memory stores at least one program, which is loaded and executed by the processor to implement the training method of the image segmentation model as described in any one of claims 1 to 8, or to implement the image segmentation method as described in any one of claims 9 to 11.

15. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is loaded and executed by a processor to implement the training method of the image segmentation model as described in any one of claims 1 to 8, or to implement the image segmentation method as described in any one of claims 9 to 11.

16. A computer program product, characterized in that, The system includes computer instructions that, when executed by a processor, implement the training method of the image segmentation model as described in any one of claims 1 to 8, or the image segmentation method as described in any one of claims 9 to 11.

Images