Training method and device of image segmentation model, electronic equipment, computer readable storage medium and computer program product

Abstract

The application provides a training method and device of an image segmentation model, an electronic device, a computer readable storage medium and a computer program product, applied to the field of artificial intelligence technology, and the image segmentation model at least includes an encoder, a generator and a discriminator, and the method comprises: obtaining a first image sample carrying a segmentation label; encoding the first image sample through the encoder to obtain a first image vector; performing image reconstruction and segmentation image prediction on the first image sample based on the first image vector through the generator to obtain a first reconstructed image and a first segmentation image; predicting the authenticity of the first reconstructed image and the first segmentation image based on the first image sample and the segmentation label through the discriminator to obtain a first prediction result; determining the value of a loss function of the image segmentation model based on the first prediction result, and updating the model parameters of the image segmentation model based on the value of the loss function. In this way, the accuracy of the segmentation result of the image segmentation model can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for an image segmentation model. Background Technology

[0002] In industrial inspection based on traditional deep learning segmentation algorithms, due to insufficient label data or high acquisition costs, the definition of defects in some parts is very vague, requiring professional quality inspectors to identify them. Therefore, when segmentation methods based on deep neural networks are used to detect some minor defects in industry, the accuracy of the detection results is low and it is difficult to meet the application requirements. Summary of the Invention

[0003] This application provides a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for an image segmentation model, which can improve the accuracy of the segmentation results of the image segmentation model.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a method for training an image segmentation model, wherein the image segmentation model includes at least an encoder, a generator, and a discriminator, comprising: Obtain a first image sample carrying a segmentation label, wherein the segmentation label is a standard segmentation image of the first image sample; The encoder encodes the first image sample to obtain a first image vector corresponding to the first image sample. The generator performs image reconstruction and segmentation prediction on the first image sample based on the first image vector to obtain the first reconstructed image and the first segmented image corresponding to the first image sample. The discriminator predicts the authenticity of the first reconstructed image and the first segmented image based on the first image sample and the segmentation label, respectively, to obtain a first prediction result. Based on the first prediction result, the value of the loss function of the image segmentation model is determined, and the model parameters of the image segmentation model are updated based on the value of the loss function.

[0005] This application provides a training apparatus for an image segmentation model, wherein the image segmentation model includes at least an encoder, a generator, and a discriminator, comprising: The acquisition module is used to acquire a first image sample carrying a segmentation label, wherein the segmentation label is a standard segmentation image of the first image sample; The encoding module is used to encode the first image sample through the encoder to obtain a first image vector corresponding to the first image sample; The generation module is used to perform image reconstruction and segmentation prediction on the first image sample based on the first image vector through the generator, so as to obtain the first reconstructed image and the first segmented image corresponding to the first image sample. The discrimination module is used to predict the authenticity of the first reconstructed image and the first segmented image based on the first image sample and the segmentation label through the discriminator, and obtain a first prediction result; The prediction update module is used to determine the value of the loss function of the image segmentation model based on the first prediction result, and update the model parameters of the image segmentation model based on the value of the loss function.

[0006] In the above scheme, the first prediction result includes a first realism prediction result and a second realism prediction result; the discrimination module is further configured to match the first reconstructed image with the first image sample through the discriminator to obtain a first matching result, and determine a first realism prediction result to indicate the realism of the first reconstructed image based on the first matching result; match the first image pair with the second image pair through the discriminator to obtain a second matching result, and determine a second realism prediction result to indicate the realism of the first segmented image based on the second matching result; wherein, the first image pair consists of the first reconstructed image and the first segmented image, and the second image pair consists of the first image sample and the standard segmented image.

[0007] In the above scheme, the discrimination module is further configured to: match the first reconstructed image with the first image sample pixel by pixel using the discriminator to obtain a first sub-matching result of the first reconstructed image and the first image sample in the pixel dimension; extract features from the first reconstructed image and the first image sample respectively using the discriminator to obtain a first image feature of the first reconstructed image and a second image feature of the first image sample, and match the first image feature with the second image feature to obtain a second sub-matching result of the first reconstructed image and the first image sample in the feature dimension; and perform a weighted summation of the first sub-matching result and the second sub-matching result to obtain a first matching result.

[0008] In the above scheme, the device further includes a matching module, which is used to match the first segmented image with the standard segmented image through the discriminator to obtain a third matching result; the discriminator is also used to perform a weighted summation of the second matching result and the third matching result to obtain a second authenticity prediction result used to indicate the authenticity of the first segmented image.

[0009] In the above scheme, the discrimination module is further configured to extract features from the first reconstructed image and the first segmented image using the discriminator to obtain a first image feature of the first reconstructed image and a second image feature of the first image sample, and to extract features from the first image sample and the standard segmented image to obtain a third image feature of the first image sample and a fourth image feature of the standard segmented image; to concatenate the first image feature and the second image feature using the discriminator to obtain a first concatenated feature, and to concatenate the third image feature and the fourth image feature to obtain a second concatenated feature; and to match the first concatenated feature and the second concatenated feature using the discriminator to obtain a second matching result.

[0010] In the above scheme, the first prediction result includes a first realism prediction result for the first reconstructed image and a second realism prediction result for the first segmented image. The prediction update module is further configured to obtain the first loss function, the second loss function, and the third loss function of the image segmentation model; determine the reconstruction loss of the first image sample by combining the first realism prediction result and the first loss function; determine the segmentation loss of the first image sample by combining the second realism prediction result and the second loss function; determine the adversarial loss of the first image sample by combining the first realism prediction result, the second realism prediction result, and the third loss function; and perform a weighted summation of the reconstruction loss, the segmentation loss, and the adversarial loss of the first image sample to obtain the value of the loss function of the image segmentation model.

[0011] In the above scheme, the value of the loss function includes: a reconstruction loss determined based on a first realism prediction result of the first reconstructed image, a segmentation loss determined based on a second realism prediction result of the first segmented image, and an adversarial loss determined by combining the first realism prediction result and the second realism prediction result; the prediction update module is further configured to fix the parameters of the encoder and the generator, update the parameters of the discriminator based on the adversarial loss; and fix the parameters of the discriminator, update the parameters of the encoder and the generator by combining the reconstruction loss and the segmentation loss.

[0012] In the above scheme, the device further includes a second image sample processing module, which is used to encode a second image sample without segmentation labels using the encoder to obtain a second image vector corresponding to the second image sample; to perform image reconstruction and segmentation image prediction on the second image sample using the generator based on the second image vector to obtain a second reconstructed image and a second segmented image corresponding to the second image sample; to predict the authenticity of the second reconstructed image and the second segmented image using the discriminator based on the second image sample and the segmentation labels to obtain a second prediction result; the prediction update module is further used to determine the value of the loss function of the image segmentation model based on the first prediction result and the second prediction result, and to update the model parameters of the image segmentation model based on the value of the loss function.

[0013] In the above scheme, the second prediction result includes a third realism prediction result and a fourth realism prediction result; the discrimination module is further configured to match the second reconstructed image with the second image sample through the discriminator to obtain a fourth matching result, and determine a third realism prediction result to indicate the realism of the second reconstructed image based on the fourth matching result; match the third image pair with the second image pair through the discriminator to obtain a fifth matching result, and determine a fourth realism prediction result to indicate the realism of the second segmented image based on the fifth matching result; wherein, the third image pair consists of the second reconstructed image and the second segmented image, and the second image pair consists of the first image sample and the standard segmented image.

[0014] In the above scheme, the similarity between the first image sample and the second image sample is lower than the similarity threshold; the discrimination module is further used to obtain the mapping relationship between the fifth matching result and the fourth authenticity prediction result; based on the fifth matching result and the mapping relationship, a fourth authenticity prediction result is determined to indicate the authenticity of the second segmented image; wherein, the authenticity of the second segmented image is negatively correlated with the fifth matching result.

[0015] In the above scheme, the first prediction result includes a first realism prediction result for the first reconstructed image and a second realism prediction result for the first segmented image. The second prediction result includes a third realism prediction result for the second reconstructed image and a fourth realism prediction result for the second segmented image. The prediction update module is further configured to obtain the fourth loss function, a fifth loss function, and a sixth loss function included in the loss function of the image segmentation model; combine the first realism prediction result and the fourth loss function to determine the first reconstruction loss of the first image sample, and combine the third realism prediction result and the fourth loss function to determine the second reconstruction loss of the second image sample; and perform a weighted summation of the first reconstruction loss and the second reconstruction loss to obtain the prediction update result for the first image sample. The total reconstruction loss of the first image sample and the second image sample; the segmentation loss of the first image sample is determined by combining the second realism prediction result and the fifth loss function; the first adversarial loss of the first image sample is determined by combining the first realism prediction result, the second realism prediction result and the sixth loss function, and the second adversarial loss of the second image sample is determined by combining the third realism prediction result, the fourth realism prediction result and the sixth loss function; the first adversarial loss and the second adversarial loss are weighted and summed to obtain the total adversarial loss for the first image sample and the second image sample; the total reconstruction loss, the segmentation loss and the total adversarial loss are weighted and summed to obtain the value of the loss function of the image segmentation model.

[0016] In the above scheme, the generation module is further configured to decode the first image vector through the generator to obtain the first reconstructed image and the first segmented image corresponding to the first image sample.

[0017] In the above scheme, the generation module is further configured to decode the first image vector in the red-green-blue channels respectively to obtain the decoding results corresponding to each color channel, and fuse the decoding results corresponding to each color channel to obtain the first reconstructed image corresponding to the first image sample; and to perform single-channel decoding on the first image vector to obtain the first segmented image corresponding to the first image sample.

[0018] In the above scheme, the device further includes an application module, which is used to acquire the image to be segmented and the trained target image segmentation model; to encode the image to be segmented based on the encoder of the target image segmentation model to obtain the target image vector of the image to be segmented; and to perform segmentation image prediction on the image to be segmented based on the target image vector through the generator of the target image segmentation model to obtain the target segmented image of the image to be segmented.

[0019] This application provides an electronic device, including: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the training method of the image segmentation model provided in the embodiments of this application.

[0020] This application provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor will execute the training method of the image segmentation model provided in this application.

[0021] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image segmentation model training method provided in this application.

[0022] The embodiments of this application have the following beneficial effects: By performing image reconstruction and segmentation prediction on the first image vector obtained from the encoder, a first reconstructed image and a first segmented image are obtained. Then, the realism of the first reconstructed image and the first segmented image are predicted respectively, resulting in a first prediction result. The model parameters of the image segmentation model are then updated based on this first prediction result. In this way, by combining image reconstruction and image segmentation, the image segmentation model is trained. While reconstructing the image using a generator, the overall distribution of image pixels is modeled, and the features of each pixel are extracted, thereby achieving classification and determination of that pixel. This results in high-quality, high-precision semantic segmentation, improving the accuracy of the image segmentation model's segmentation results. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the architecture of the training system 100 for the image segmentation model provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 3 This is a flowchart illustrating the training method of the image segmentation model provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the image segmentation model provided in the embodiments of this application; Figure 5 This is a schematic diagram of a first image sample carrying segmentation labels provided in an embodiment of this application; Figure 6 This is a schematic diagram of the generator provided in the embodiments of this application; Figure 7 This is a flowchart of the process for determining the first prediction result provided in the embodiments of this application; Figure 8 This is a schematic diagram of the discriminator provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the image segmentation model provided in the embodiments of this application; Figure 10 This is a schematic diagram of the discriminator provided in the embodiments of this application; Figure 11 This is a schematic diagram illustrating the pre-training process of the encoder and generator provided in an embodiment of this application; Figure 12 This is a flowchart illustrating the training method of the image segmentation model provided in the embodiments of this application; Figure 13 This is a flowchart of the training method for the image segmentation model provided in the embodiments of this application; Figure 14 This is a schematic diagram of the training process framework for labeled data provided in the embodiments of this application; Figure 15 This is a schematic diagram of the training process framework for unlabeled data provided in the embodiments of this application; Figure 16 This is a schematic diagram of the input and output during the training of the discriminator provided in the embodiments of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0026] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0028] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0029] 1) Semi-supervised learning refers to training a model using both labeled and unlabeled data simultaneously. It is typically applied when labeled data is scarce or when labels are difficult to obtain, thereby improving the model's ability on limited labeled data.

[0030] 2) Client: Also known as user terminal, it refers to the program that provides local services to users in contrast to the server. Except for some applications that can only run locally, it is generally installed on ordinary client machines and needs to cooperate with the server to run. That is, there needs to be a corresponding server and service program in the network to provide the corresponding services. Thus, a specific communication connection needs to be established between the client and the server to ensure the normal operation of the application.

[0031] 3) Semantic segmentation, also known as image segmentation, refers to the process of subdividing a digital image into multiple image sub-regions (sets of pixels) (also called superpixels). It's the technique and process of dividing an image into several specific regions with unique properties and identifying objects of interest. It's a key step from image processing to image analysis. The purpose of semantic segmentation is to simplify or change the representation of an image, making it easier to understand and analyze. Semantic segmentation is commonly used to locate objects and boundaries (lines, curves, etc.) in an image. More precisely, semantic segmentation is a process of labeling each pixel in an image, ensuring that pixels with the same label share certain common visual characteristics.

[0032] For example, in an image, pixels of objects belonging to the same category are grouped together and marked with the same color, which is pixel-by-pixel classification. For instance, in a natural image, all pixels belonging to trees are marked with one color, all people are marked with one color pixel-by-pixel, the sky is marked with one color, and so on.

[0033] 4) Image segmentation, the result image obtained from semantic segmentation.

[0034] 5) Loss is used to measure the gap between the model's actual results and the target results, in order to train and optimize the model.

[0035] 6) Deep Neural Network-Based Segmentation Methods: With the successful application of deep learning in image perception, traditional image segmentation techniques have gradually been replaced by deep learning technologies. Deep neural network-based segmentation methods mainly focus on using convolutional neural networks to extract features from images, and then performing pixel-by-pixel classification based on the extracted features. This process typically involves continuous downsampling during feature extraction and continuous upsampling during the generation of segmented images. It usually requires a large amount of labeled data for training, and the trained model often performs poorly on new data. Furthermore, retraining not only requires data labeling but is also time-consuming.

[0036] 7) The semi-supervised segmentation method based on adversarial training uses a neural network discriminator to determine whether the generated semantic segmentation image is real. Specifically, by using the discriminator, it is not necessary to have real segmentation image labels on unlabeled data. It only distinguishes whether the obtained image is similar to the usual segmentation image, or distinguishes the pixel-by-pixel credibility of the obtained image to identify fakes.

[0037] 8) Perceptual loss (LPIPS, Learned Perceptual Image Patch Similarity), which is the learning of perceptual image patch similarity, is used to measure the difference between two images.

[0038] 9) Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to using cameras and computers to replace human eyes in recognizing and measuring targets, and then performing image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0039] 10) Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0040] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0041] The inventors discovered that most image segmentation techniques in artificial intelligence are based on deep neural network segmentation methods or semi-supervised segmentation methods based on adversarial training.

[0042] For segmentation methods based on deep neural networks, a large amount of labeled data is typically required for training, and manually labeling segmentation data is very time-consuming and labor-intensive. In some application scenarios, labeling data also requires specialized personnel or professional training. For example, in industrial inspection, the definition of defects in certain parts is very vague, requiring identification by professional quality inspectors. Therefore, when labeled data is insufficient or too costly to obtain, traditional deep learning segmentation algorithms often perform unsatisfactorily. Furthermore, segmentation algorithms based entirely on supervised learning typically exhibit unstable performance and poor generalization ability on out-of-domain data, i.e., data not present in the training set. For some minor defects in industry, their performance often fails to meet application requirements.

[0043] As for semi-supervised segmentation methods based on adversarial training, they are essentially still semi-supervised methods based on pseudo-labels, using a discriminator to score the quality of the pseudo-labels. Although they can be trained using unlabeled data, they realistically compare the predicted segmented image with the actual labeled image at each pixel, without exploring the relationships between pixels corresponding to the image to be segmented, or the connection between the segmented image and the image to be segmented. This is somewhat of an overemphasis on some statistical features of the dataset itself, and the model's generalization ability cannot be improved.

[0044] Based on this, this application provides a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for an image segmentation model. While reconstructing the image through a generative network, it models the overall distribution of image pixels, captures the features of each pixel, and thus achieves classification and determination of the pixel, achieving high-quality and high-precision semantic segmentation.

[0045] The solutions provided in this application involve technologies such as image segmentation models in artificial intelligence, and can also be applied to fields such as cloud technology and vehicle networking. The specific implementation is illustrated in the following embodiments.

[0046] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the image segmentation model training system 100 provided in this application embodiment. In order to realize the application scenario of training the image segmentation model, the terminal (terminal 400 is shown as an example) is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The terminal 400 is used for users to use the client 401 and display on the display interface (display interface 401-1 is shown as an example). The terminal 400 and the server 200 are connected to each other through wired or wireless networks.

[0047] The terminal 400 is used to: acquire a first image sample carrying a segmentation label, wherein the segmentation label is a standard segmented image of the first image sample; and send the first image sample carrying the segmentation label to the server 200. Server 200 is configured to: receive a first image sample carrying a segmentation label; encode the first image sample using an encoder to obtain a first image vector corresponding to the first image sample; perform image reconstruction and segmentation image prediction on the first image sample based on the first image vector using a generator to obtain a first reconstructed image and a first segmented image corresponding to the first image sample; predict the authenticity of the first reconstructed image and the first segmented image based on the first image sample and the segmentation label using a discriminator to obtain a first prediction result; determine the value of the loss function of the image segmentation model based on the first prediction result, and update the model parameters of the image segmentation model based on the value of the loss function.

[0048] In some embodiments, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 can be a smartphone, tablet, laptop, desktop computer, set-top box, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, smart speakers, and smartwatches), but is not limited to these. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0049] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. In practical applications, the electronic device can be... Figure 1 The server 200 or terminal 400 shown are described in the following document. Figure 2 , Figure 2 The illustrated electronic device includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0050] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0051] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0052] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0053] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0054] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0055] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., a display screen, a speaker, etc.). The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0056] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A training device 455 for an image segmentation model stored in memory 450 is shown. This device can be software in the form of programs or plugins, and includes the following software modules: an acquisition module 4551, an encoding module 4552, a generation module 4553, a discrimination module 4554, and a prediction update module 4555. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0057] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the training apparatus for the image segmentation model provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method for the image segmentation model provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0058] In some embodiments, the terminal or server can implement the image segmentation model training method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as an instant messaging APP or a web browser APP; it can also be a mini-program, that is, a program that only needs to be downloaded into a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0059] Based on the above description of the image segmentation model training system and electronic device provided in the embodiments of this application, the training method of the image segmentation model provided in the embodiments of this application is described below. In actual implementation, the training method of the image segmentation model provided in the embodiments of this application can be implemented by the terminal or the server alone, or by the terminal and the server working together, so that... Figure 1 The following description uses the example of server 200 executing the training method of the image segmentation model provided in this application embodiment. See also... Figure 3 , Figure 3 This is a flowchart illustrating the training method of the image segmentation model provided in this application embodiment. It should be noted that the image segmentation model includes an encoder, a generator, and a discriminator. (See also...) Figure 4 , Figure 4 This is a schematic diagram of the image segmentation model provided in the embodiments of this application. Below, it will be discussed in conjunction with... Figure 3 and Figure 4 The steps shown are explained.

[0060] Step 101: The server obtains a first image sample carrying a segmentation label, where the segmentation label is the standard segmentation image of the first image sample.

[0061] In practice, the first image sample carrying the segmentation label can be pre-stored locally on the terminal, or it can be obtained by the terminal from external sources (such as the Internet). Alternatively, the first image sample can be captured in real-time by the terminal, for example, through its camera. Then, by labeling the first image sample, the first image sample carrying the segmentation label is obtained. For example, see [link to example]. Figure 5 , Figure 5 This is a schematic diagram of a first image sample carrying segmentation labels provided in an embodiment of this application, based on... Figure 5 The dashed box 501 contains the first image sample, while the dashed box 502 contains the segmentation label corresponding to the first image sample.

[0062] It should be noted that in the embodiments of this application, real-time acquired images and other related data are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0063] Step 102: Encode the first image sample using an encoder to obtain the first image vector corresponding to the first image sample.

[0064] See also Figure 4 ,based on Figure 4 After acquiring the first image sample, the first image sample is encoded by the encoder included in the image segmentation model to obtain the first image vector corresponding to the first image sample.

[0065] It should be noted that before encoding the first image sample to obtain the corresponding first image vector, the first image sample can be preprocessed. Specifically, the first image sample is normalized to obtain a first preprocessed image sample, which is then used for subsequent processing. Here, normalization includes, but is not limited to, smoothing, cropping, segmentation, and normalization based on bilinear interpolation algorithms. Thus, by preprocessing the first image sample, the influence of factors such as illumination and shadows on the image segmentation process is reduced, improving the training efficiency and segmentation accuracy of the image segmentation model.

[0066] Step 103: Using a generator, based on the first image vector, perform image reconstruction and segmentation prediction on the first image sample to obtain the first reconstructed image and the first segmented image corresponding to the first image sample.

[0067] In actual implementation, the process of using a generator to perform image reconstruction and segmentation prediction on the first image sample based on the first image vector to obtain the first reconstructed image and the first segmented image corresponding to the first image sample specifically includes decoding the first image vector using a generator to obtain the first reconstructed image and the first segmented image corresponding to the first image sample.

[0068] It should be noted that the generator here includes a reconstructed image generator and a segmented image generator. Thus, the process of decoding the first image vector through the generator to obtain the first reconstructed image and the first segmented image corresponding to the first image sample specifically includes decoding the first image vector through the reconstructed image generator to obtain the first reconstructed image, and decoding the first image vector through the segmented image generator to obtain the first segmented image.

[0069] For example, see Figure 6 , Figure 6 This is a schematic diagram of the generator structure provided in the embodiments of this application, based on Figure 6 The generator includes a reconstructed image generator and a segmented image generator. After obtaining the first image vector, the reconstructed image generator performs image reconstruction (i.e., decoding) on ​​the first image vector to obtain the first reconstructed image. The segmented image generator performs segmented image prediction (i.e., decoding) on ​​the first image vector to obtain the first segmented image.

[0070] In actual implementation, due to the differences between the reconstructed image generator and the segmented image generator in the generator, the process of decoding the first image vector to obtain the first reconstructed image and the first segmented image corresponding to the first image sample also differs. Specifically, the process of decoding the first image vector to obtain the first reconstructed image and the first segmented image corresponding to the first image sample includes decoding the first image vector in the red-green-blue channels respectively to obtain the decoding results corresponding to each color channel, and fusing the decoding results corresponding to each color channel to obtain the first reconstructed image corresponding to the first image sample; and performing single-channel decoding on the first image vector to obtain the first segmented image corresponding to the first image sample.

[0071] Step 104: Using a discriminator, based on the first image sample and segmentation label, predict the authenticity of the first reconstructed image and the first segmented image respectively, and obtain the first prediction result.

[0072] In practice, the first prediction result includes a first true prediction result and a second true prediction result. Based on this, see [link to relevant documentation]. Figure 7 , Figure 7 This is a flowchart of the process for determining the first prediction result provided in the embodiments of this application, based on Figure 7 Step 104 can be achieved in the following way: Step 1041: The first reconstructed image is matched with the first image sample by the discriminator to obtain the first matching result, and based on the first matching result, a first authenticity prediction result is determined to indicate the authenticity of the first reconstructed image.

[0073] In actual implementation, there are multiple ways to match the first reconstructed image with the first image sample using a discriminator to obtain the first matching result. Next, we will take three of them as examples to explain the process of matching the first reconstructed image with the first image sample using a discriminator to obtain the first matching result.

[0074] In some embodiments, the first reconstructed image can be matched with the first image sample based on the pixel dimension to obtain a first matching result. Specifically, the first reconstructed image and the first image sample are matched pixel by pixel by a discriminator to obtain a matching result between the first reconstructed image and the first image sample in the pixel dimension, and the matching result is used as the first matching result.

[0075] In some embodiments, the first reconstructed image can be matched with the first image sample based on the feature dimension to obtain a first matching result. Specifically, a discriminator is used to extract features from the first reconstructed image and the first image sample respectively to obtain the first image feature of the first reconstructed image and the second image feature of the first image sample. The first image feature and the second image feature are then matched to obtain the matching result of the first reconstructed image and the first image sample in the feature dimension, and the matching result is used as the first matching result.

[0076] In some embodiments, the first reconstructed image can be matched with the first image sample based on the pixel dimension and the feature dimension to obtain a first matching result. Specifically, the discriminator matches the first reconstructed image with the first image sample pixel by pixel to obtain a first sub-matching result of the first reconstructed image with the first image sample in the pixel dimension; the discriminator extracts features from the first reconstructed image and the first image sample respectively to obtain a first image feature of the first reconstructed image and a second image feature of the first image sample, and matches the first image feature with the second image feature to obtain a second sub-matching result of the first reconstructed image with the first image sample in the feature dimension; the first sub-matching result and the second sub-matching result are weighted and summed to obtain the first matching result.

[0077] It should be noted that the matching result between the first reconstructed image and the first image sample is also known as the similarity between the first reconstructed image and the first image sample, which is also the first authenticity prediction result used to indicate the authenticity of the first reconstructed image. Here, authenticity is used to indicate the degree of similarity between the first reconstructed image and the first image sample.

[0078] Step 1042: Determine the first target matching result through the discriminator, and determine the second realism prediction result to indicate the realism of the first segmented image based on the first target matching result.

[0079] The first target matching result includes at least one of the second matching result and the third matching result. The second matching result is the matching result between the first image pair and the second image pair. The first image pair consists of the first reconstructed image and the first segmented image. The second image pair consists of the first image sample and the standard segmented image. The third matching result is the matching result between the first segmented image and the standard segmented image.

[0080] In some embodiments, when the first target matching result includes a second matching result, the process of determining the first target matching result by a discriminator and determining a second realism prediction result to indicate the realism of the first segmented image based on the first target matching result specifically includes matching the first image pair with the second image pair by a discriminator to obtain the second matching result, and determining the second realism prediction result to indicate the realism of the first segmented image based on the second matching result; wherein the first image pair consists of a first reconstructed image and a first segmented image, and the second image pair consists of a first image sample and a standard segmented image.

[0081] It should be noted that the matching result of the first image pair and the second image pair here is also the similarity between the first image pair and the second image pair, which is also the second realism prediction result used to indicate the realism of the first segmented image. Here, realism is used to indicate the degree of similarity between the first image pair and the second image pair.

[0082] In practical implementation, matching the first image pair with the second image pair can be achieved through feature concatenation. Specifically, the process of matching the first image pair with the second image pair using a discriminator to obtain a second matching result includes: using the discriminator to extract features from the first reconstructed image and the first segmented image respectively, obtaining the first image features of the first reconstructed image and the second image features of the first image sample; and using the discriminator to extract features from the first image sample and the standard segmented image respectively, obtaining the third image features of the first image sample and the fourth image features of the standard segmented image; using the discriminator to concatenate the first image features and the second image features to obtain the first concatenated feature; and using the discriminator to concatenate the third image features and the fourth image features to obtain the second concatenated feature; and using the discriminator to match the first concatenated feature and the second concatenated feature to obtain the second matching result. Based on the second matching result, a second realism prediction result used to indicate the realism of the first segmented image is determined. Here, the process of matching the first splicing feature and the second splicing feature with a discriminator to obtain the second matching result is similar to the process of matching the first reconstructed image with the first image sample with a discriminator in step 1041 to obtain the first matching result. Therefore, this embodiment will not elaborate on this aspect.

[0083] In some embodiments, when the first target matching result includes a third matching result, the process of determining the first target matching result using a discriminator and, based on the first target matching result, determining a second realism prediction result to indicate the realism of the first segmented image specifically includes matching the first segmented image with a standard segmented image using a discriminator to obtain a third matching result, and, based on the third matching result, determining a second realism prediction result to indicate the realism of the first segmented image. Here, the process of matching the first segmented image with the standard segmented image using a discriminator to obtain a second matching result is similar to the process in step 1041 of matching the first reconstructed image with the first image sample using a discriminator to obtain a first matching result; therefore, this embodiment will not elaborate further on this aspect.

[0084] It should be noted that the matching result between the first segmented image and the standard segmented image is also known as the similarity between the first segmented image and the standard segmented image. In other words, it is the second realism prediction result used to indicate the realism of the first segmented image. Here, realism is used to indicate the degree of similarity between the first segmented image and the standard segmented image.

[0085] In some embodiments, when the first target matching result includes a second matching result and a third matching result, the process of determining the first target matching result by a discriminator and determining a second realism prediction result for indicating the realism of the first segmented image based on the first target matching result specifically includes: matching the first image pair with the second image pair by a discriminator to obtain a second matching result; matching the first segmented image with a standard segmented image by a discriminator to obtain a third matching result; and determining a second realism prediction result for indicating the realism of the first segmented image based on the second matching result and the third matching result. Here, the process of determining the second realism prediction result for indicating the realism of the first segmented image based on the second matching result and the third matching result may be a weighted summation of the second matching result and the third matching result to obtain the second realism prediction result for indicating the realism of the first segmented image.

[0086] It should be noted that the discriminator here includes a first discriminator and a second discriminator. Therefore, the process of predicting the authenticity of the first reconstructed image and the first segmented image based on the first image sample and the segmentation label by the discriminator to obtain the first prediction result specifically includes: predicting the authenticity of the first reconstructed image by the first discriminator to obtain the first authenticity prediction result, and predicting the authenticity of the first segmented image by the second discriminator to obtain the second authenticity prediction result.

[0087] For example, see Figure 8 , Figure 8 This is a schematic diagram of the discriminator provided in the embodiments of this application, based on Figure 8 The discriminator includes a first discriminator and a second discriminator. After acquiring the first reconstructed image and the first segmented image, the first discriminator predicts the authenticity of the first reconstructed image, that is, determines the similarity between the first reconstructed image and the first image sample, and obtains a first authenticity prediction result. The second discriminator predicts the authenticity of the first segmented image, that is, determines the similarity between the first segmented image and the standard segmented image and / or the similarity between the first image pair and the second image pair, and obtains a second authenticity prediction result.

[0088] Step 105: Based on the first prediction result, determine the value of the loss function of the image segmentation model, and update the model parameters of the image segmentation model based on the value of the loss function.

[0089] In practical implementation, the first prediction result includes a first ground truth prediction result for the first reconstructed image and a second ground truth prediction result for the first segmented image. Therefore, the process of determining the value of the loss function of the image segmentation model based on the first prediction result specifically includes obtaining the first loss function, the second loss function, and the third loss function included in the image segmentation model's loss function. ...Formula (1); ...Formula (2); ...Formula (3-1); ...Formula (3-2); ...Formula (3-3); Among them, China As the first image sample, For encoder, For generator, For discriminator, For separating tags, For the first reconstructed image, For the first segmented image, H represents the cross-entropy loss, and DC represents the dice loss. This indicates the calculation of expectation. This represents the expected value of the first true prediction result. , as well as This represents the expected value of the corresponding second-realism prediction result.

[0090] It should be noted that the first loss function is Equation (1), the second loss function is Equation (2), and the third loss function is one of Equations (3-1), (3-2), and (3-3). Specifically, when the second true prediction result is determined by the second matching result, the third loss function is Equation (3-1); when the second true prediction result is determined by the third matching result, the third loss function is Equation (3-2); and when the second true prediction result is determined by both the second and third matching results, the third loss function is Equation (3-3).

[0091] Then, after obtaining the first loss function, second loss function, and third loss function of the image segmentation model, the reconstruction loss of the first image sample is determined by combining the first realism prediction result and the first loss function; the segmentation loss of the first image sample is determined by combining the second realism prediction result and the second loss function; and the adversarial loss of the first image sample is determined by combining the first realism prediction result, the second realism prediction result, and the third loss function. The reconstruction loss, segmentation loss, and adversarial loss of the first image sample are then weighted and summed to obtain the value of the loss function of the image segmentation model, i.e. ...Formula (4) in, To rebuild the losses, To divide the loss, To counteract the loss, the model parameters of the image segmentation model are updated based on the value of the loss function.

[0092] In some embodiments, in addition to updating the model parameters of the image segmentation model based on the value of the loss function, when the value of the loss function includes the reconstruction loss determined based on the first realism prediction result of the first reconstructed image, the segmentation loss determined based on the second realism prediction result of the first segmented image, and the adversarial loss determined by combining the first realism prediction result and the second realism prediction result, the parameters of the encoder, generator, and discriminator can also be updated separately. Specifically, the process of updating the model parameters of the image segmentation model based on the value of the loss function can be: fixing the parameters of the encoder and generator, and updating the parameters of the discriminator based on the adversarial loss; fixing the parameters of the discriminator, and updating the parameters of the encoder and generator by combining the reconstruction loss and the segmentation loss.

[0093] In some embodiments, see Figure 9 , Figure 9 This is a schematic diagram of the structure of the image segmentation model provided in the embodiments of this application, based on Figure 9In addition to training the image segmentation model based on a first image sample carrying segmentation labels, the model can also be trained based on both the first image sample carrying segmentation labels and a second image sample without segmentation labels. Specifically, while training the image segmentation model based on the first image sample as described above, a second image sample without segmentation labels can also be obtained. An encoder encodes the second image sample without segmentation labels to obtain a second image vector corresponding to the second image sample. A generator performs image reconstruction and segmentation prediction on the second image sample based on the second image vector to obtain a second reconstructed image and a second segmented image corresponding to the second image sample. A discriminator predicts the authenticity of the second reconstructed image and the second segmented image based on the second image sample and segmentation labels, respectively, to obtain a second prediction result. Then, based on the first prediction result, the value of the loss function of the image segmentation model is determined, and the model parameters of the image segmentation model are updated based on the value of the loss function. This process specifically includes determining the value of the loss function of the image segmentation model based on the first and second prediction results, and updating the model parameters of the image segmentation model based on the value of the loss function.

[0094] It should be noted that the second image sample without segmentation labels can be pre-stored locally on the terminal, obtained by the terminal from the outside world (such as the Internet), or collected in real time by the terminal, such as through the terminal's camera.

[0095] In practical implementation, the second prediction result includes a third realism prediction result and a fourth realism prediction result. Therefore, the process of predicting the realism of the second reconstructed image and the second segmented image based on the second image samples and segmentation labels using a discriminator to obtain the second prediction result specifically includes: matching the second reconstructed image with the second image samples using a discriminator to obtain a fourth matching result; and determining a third realism prediction result to indicate the realism of the second reconstructed image based on the fourth matching result; determining a second target matching result using a discriminator; and determining a fourth realism prediction result to indicate the realism of the second segmented image based on the second target matching result. The second target matching result includes at least one of a fifth matching result and a sixth matching result. The fifth matching result is the matching result between the second image pair and the third image pair, where the third image pair consists of the second reconstructed image and the second segmented image; the second image pair consists of the first image sample and the standard segmented image; and the sixth matching result is the matching result between the second segmented image and the standard segmented image.

[0096] It should be noted that the process of matching the second reconstructed image with the second image sample using the discriminator to obtain the fourth matching result is similar to the process of matching the first reconstructed image with the first image sample using the discriminator in step 1041 to obtain the first matching result, and will not be elaborated further. Here, the matching result between the second reconstructed image and the second image sample is also the similarity between the second reconstructed image and the second image sample, which is also the third realism prediction result used to indicate the realism of the second reconstructed image. Here, realism is used to indicate the degree of similarity between the second reconstructed image and the second image sample.

[0097] The process involves determining the second target matching result through a discriminator, and then determining the fourth realism prediction result to indicate the realism of the second segmented image based on the second target matching result.

[0098] In some embodiments, when the second target matching result includes a fifth matching result, the process of determining the second target matching result by a discriminator and determining a fourth realism prediction result to indicate the realism of the second segmented image based on the second target matching result specifically includes matching the second image pair with the third image pair by a discriminator to obtain a fifth matching result, and determining a fourth realism prediction result to indicate the realism of the second segmented image based on the fifth matching result; wherein the third image pair consists of a second reconstructed image and a second segmented image, and the second image pair consists of a first image sample and a standard segmented image.

[0099] It should be noted that the matching result between the third image pair and the second image pair here is also the similarity between the third image pair and the second image pair, which is also the fourth realism prediction result used to indicate the realism of the second segmented image. Here, realism is used to indicate the degree of similarity between the third image pair and the second image pair. The process of matching the third image pair with the second image pair through feature concatenation is similar to the process of matching the first image pair with the second image pair through feature concatenation, as described above, and will not be elaborated upon here.

[0100] In some embodiments, when the second target matching result includes a sixth matching result, the process of determining the second target matching result using a discriminator and, based on the second target matching result, determining a fourth realism prediction result to indicate the realism of the second segmented image specifically includes matching the second segmented image with a standard segmented image using a discriminator to obtain a sixth matching result, and, based on the sixth matching result, determining a fourth realism prediction result to indicate the realism of the second segmented image. Here, the process of matching the second segmented image with the standard segmented image using a discriminator to obtain a sixth matching result is similar to the process in step 1041 of matching the first reconstructed image with the first image sample using a discriminator to obtain a first matching result; therefore, this embodiment will not elaborate further.

[0101] It should be noted that the matching result between the second segmented image and the standard segmented image is also known as the similarity between the second segmented image and the standard segmented image. In other words, it is the fourth realism prediction result used to indicate the realism of the second segmented image. Here, realism is used to indicate the degree of similarity between the second segmented image and the standard segmented image.

[0102] In some embodiments, when the second target matching result includes a fifth matching result and a sixth matching result, the process of determining the second target matching result by a discriminator and determining a fourth realism prediction result to indicate the realism of the second segmented image based on the second target matching result specifically includes: matching the third image pair with the second image pair by a discriminator to obtain a fifth matching result; matching the second segmented image with a standard segmented image by a discriminator to obtain a sixth matching result; and determining a fourth realism prediction result to indicate the realism of the second segmented image based on the fifth matching result and the sixth matching result. Here, the process of determining the fourth realism prediction result to indicate the realism of the second segmented image based on the fifth matching result and the sixth matching result may be a weighted sum of the fifth matching result and the sixth matching result to obtain the fourth realism prediction result to indicate the realism of the second segmented image.

[0103] It should be noted that the discriminator here also includes a first discriminator and a second discriminator. Thus, the process of predicting the authenticity of the second reconstructed image and the second segmented image based on the second image sample and the segmentation label by the discriminator to obtain the second prediction result specifically includes: predicting the authenticity of the second reconstructed image by the first discriminator to obtain the third authenticity prediction result, and predicting the authenticity of the second segmented image by the second discriminator to obtain the fourth authenticity prediction result.

[0104] For example, see Figure 10 , Figure 10 This is a schematic diagram of the discriminator provided in the embodiments of this application, based on Figure 10 The discriminator includes a first discriminator and a second discriminator. After acquiring the second reconstructed image and the second segmented image, the first discriminator predicts the authenticity of the second reconstructed image, that is, determines the similarity between the second reconstructed image and the second image sample, and obtains a third authenticity prediction result. The second discriminator predicts the authenticity of the second segmented image, that is, determines the similarity between the second segmented image and the standard segmented image and / or the similarity between the third image pair and the second image pair, and obtains a fourth authenticity prediction result.

[0105] It should be noted that the similarity between the first image sample and the second image sample is lower than the similarity threshold. For example, the image content of the first image sample and the second image sample are different. The similarity threshold can be preset, such as 0.3.

[0106] It should be noted that the similarity between the first image sample and the second image sample is lower than the similarity threshold. Therefore, the process of determining the fourth realism prediction result to indicate the realism of the second segmented image based on the obtained fifth matching result and / or sixth matching result specifically includes: obtaining the mapping relationship between the fifth matching result and / or sixth matching result and the fourth realism prediction result; determining the fourth realism prediction result to indicate the realism of the second segmented image based on the fifth matching result and / or sixth matching result and the mapping relationship; wherein the realism of the second segmented image is negatively correlated with the fifth matching result and / or sixth matching result.

[0107] For example, when the fifth matching result of the third image pair and the second image pair, that is, the similarity between the third image pair and the second image pair is 0.2, the mapping relationship between the fifth matching result and the fourth realism prediction result is obtained; based on the fifth matching result and the mapping relationship, the fourth realism prediction result used to indicate the realism of the second segmented image is determined, such as 0.8, that is, 1-0.2.

[0108] In practical implementation, the first prediction result includes a first realism prediction result for the first reconstructed image and a second realism prediction result for the first segmented image. The second prediction result includes a third realism prediction result for the second reconstructed image and a fourth realism prediction result for the second segmented image. Therefore, the process of determining the value of the loss function of the image segmentation model based on the first and second prediction results specifically includes obtaining the fourth loss function, the fifth loss function (i.e., formula (2), and the sixth loss function included in the loss function of the image segmentation model. ...Formula (5); ...Formula (6-1); ...Formula (6-2); ...Formula (6-3); in, For the second image sample, For encoder, For generator, For discriminator, For separating tags, For the second reconstructed image, For the second segmented image, H represents the cross-entropy loss, and DC represents the dice loss. This indicates the calculation of expectation. This represents the expected value of the corresponding third-degree accuracy prediction result. , as well as This represents the expected value of the fourth true prediction result.

[0109] It should be noted that the fourth loss function is Equation (5), and the fifth loss function is one of Equations (6-1), (6-2), and (6-3). Specifically, when the fourth truth prediction result is determined by the fifth matching result, the fifth loss function is Equation (6-1); when the fourth truth prediction result is determined by the sixth matching result, the fifth loss function is Equation (6-2); and when the fourth truth prediction result is determined by both the fifth and sixth matching results, the fifth loss function is Equation (6-3).

[0110] Then, after obtaining the fourth, fifth, and sixth loss functions included in the image segmentation model's loss functions, and combining the first and third ground truth prediction results with the fourth loss function, the total reconstruction loss for the first and second image samples is determined. Combining the second realism prediction result and the fifth loss function, the segmentation loss of the first image sample is determined. Combining the first, second, third, and fourth realism prediction results with the sixth loss function, the total adversarial loss for the first and second image samples is determined. The total reconstruction loss, segmentation loss, and total adversarial loss are weighted and summed to obtain the value of the loss function of the image segmentation model, i.e. ...Formula (7); Therefore, the model parameters of the image segmentation model are updated based on the value of the loss function.

[0111] In practical implementation, the process of determining the total reconstruction loss for the first image sample and the second image sample by combining the first realism prediction result, the third realism prediction result, and the fourth loss function specifically includes determining the first reconstruction loss for the first image sample by combining the first realism prediction result and the fourth loss function. ; and combining the third realism prediction result and the fourth loss function, the second reconstruction loss of the second image sample is determined, namely Then, the first reconstruction loss and the second reconstruction loss are weighted and summed to obtain the total reconstruction loss for the first image sample and the second image sample. .

[0112] In practice, the process of determining the total adversarial loss for the first and second image samples by combining the first, second, third, and fourth realism prediction results with the sixth loss function specifically includes determining the first adversarial loss for the first image sample by combining the first, second, and sixth realism prediction results with the sixth loss function. Based on the third and fourth realism prediction results and the sixth loss function, the second adversarial loss for the second image sample is determined. The first adversarial loss and the second adversarial loss are weighted and summed to obtain the total adversarial loss for the first image sample and the second image sample. .

[0113] It should be noted that, in addition to updating the model parameters of the image segmentation model based on the value of the loss function, when the value of the loss function includes the total reconstruction loss determined based on the first and third ground truth prediction results, the segmentation loss determined based on the second ground truth prediction result, and the total adversarial loss determined based on the first, second, third, and fourth ground truth prediction results, the parameters of the encoder, generator, and discriminator can also be updated separately. Specifically, the process of updating the model parameters of the image segmentation model based on the value of the loss function can be to fix the parameters of the encoder and generator, and update the parameters of the discriminator based on the total adversarial loss, i.e. ...Formula (8); Then, with the discriminator parameters fixed, the encoder and generator parameters are updated by combining the total reconstruction loss and segmentation loss. ...Formula (9); in, and These represent the model parameters related to the encoder and generator, respectively. This represents the model parameters associated with the discriminator.

[0114] In some embodiments, before acquiring the first image sample carrying the segmentation label, the encoder and generator can be pre-trained. Specifically, training image samples are acquired, and the encoder encodes the training image samples to obtain the training image vectors corresponding to the training image samples; the generator reconstructs the training image samples based on the training image vectors to obtain the training reconstructed images of the training image samples; the encoder encodes the training reconstructed images to obtain the reconstructed image vectors corresponding to the training reconstructed images; the differences between the training image samples and the training reconstructed images, as well as the differences between the training image vectors and the reconstructed image vectors, are acquired, and the parameters of the encoder and the generator of the image segmentation model are updated based on the differences.

[0115] For example, see Figure 11 , Figure 11 This is a schematic diagram illustrating the pre-training process of the encoder and generator provided in an embodiment of this application, based on... Figure 11 The sample image is encoded to obtain an image vector w, and then the reconstructed image based on w is obtained. The image vector w' is re-encoded and its consistency in the latent space is sought, thereby training the encoder and generator according to formula (4), i.e. ...Formula (10); in, To train image samples and to differentiate the reconstructed images, These are the weight parameters. Thus, by pre-training the encoder and decoder, better image reconstruction results are obtained. Based on the optimization of image reconstruction quality, better segmentation results are achieved, accelerating the training efficiency and segmentation accuracy of the image segmentation model.

[0116] In some embodiments, after the image segmentation model is trained, that is, after the model parameters of the image segmentation model are updated based on the value of the loss function, it can also be used for practical applications. Specifically, the image to be segmented and the trained target image segmentation model are obtained; the image to be segmented is encoded based on the encoder of the target image segmentation model to obtain the target image vector of the image to be segmented; and the target image segmentation image is predicted based on the target image vector by the generator of the target image segmentation model to obtain the target segmented image of the image to be segmented.

[0117] In some embodiments, when there are certain differences between the image to be segmented and the first image sample, i.e., differences in data distribution, the segmentation effect of the image segmentation model based on the image to be segmented will usually decrease. In this case, the image to be segmented can be used as the second image sample to fine-tune the image segmentation model for better results.

[0118] The training method for the image segmentation model provided in the embodiments of this application will be described below. See also... Figure 12 , Figure 12 This is a flowchart illustrating the training method of the image segmentation model provided in this application embodiment, based on... Figure 12 The training method for the image segmentation model provided in this application embodiment is implemented collaboratively by the client and the server.

[0119] Step 201: In response to the upload operation of the first image sample carrying segmentation labels and the second image sample without segmentation labels, the client obtains the first image sample carrying segmentation labels and the second image sample without segmentation labels.

[0120] In practice, the client can be a training client for the image segmentation model set on the terminal. The first image sample and the second image sample can be uploaded by the user through the human-computer interaction interface of the client. The user can then upload the first image sample and the second image sample from the terminal based on the image selection interface, so that the client can obtain the uploaded first image sample and the second image sample.

[0121] In some embodiments, the first image sample and the second image sample may also be captured by a camera that is connected to the terminal. After capturing the first image sample and the second image sample, the camera transmits the first image sample and the second image sample to the terminal and the terminal automatically uploads them to the client.

[0122] Step 202: The client sends the first image sample and the second image sample to the server.

[0123] Step 203: The server inputs the received first image sample and second image sample into the image segmentation model.

[0124] Step 204: Based on the image segmentation model, determine the first reconstructed image and the first segmented image corresponding to the first image sample, and the second reconstructed image and the second segmented image corresponding to the second image sample.

[0125] Step 205: Based on the first image sample and segmentation label, predict the authenticity of the first reconstructed image and the first segmented image respectively to obtain the first prediction result; and based on the second image sample and segmentation label, predict the authenticity of the second reconstructed image and the second segmented image respectively to obtain the second prediction result.

[0126] Step 206: Based on the first prediction result and the second prediction result, determine the value of the loss function of the image segmentation model, and update the model parameters of the image segmentation model based on the value of the loss function.

[0127] In practice, the server iterates through the above training process until the loss function converges, thus completing the training of the image segmentation model.

[0128] Step 207: The server generates a notification message indicating that the image segmentation model training is complete.

[0129] Step 208: Send a notification message to the client.

[0130] Step 209: The client responds to the upload operation of the image to be segmented and obtains the image to be segmented.

[0131] In practice, the image to be segmented can be pre-stored locally on the terminal or obtained by the terminal from the outside world (such as the Internet).

[0132] Step 210: In response to the segmentation instruction for the image to be segmented, the client sends the image to be segmented to the server.

[0133] In practice, the segmentation instructions for the image to be segmented can be automatically generated by the client under certain triggering conditions. For example, the client can automatically generate segmentation instructions for the image to be segmented after acquiring it. Alternatively, the instructions can be sent to the client by other devices that are connected to the terminal. Or, the instructions can be generated by the user based on the client's human-computer interaction interface after triggering the corresponding confirmation function.

[0134] Step 211: The server inputs the received image to be segmented into the image segmentation model so that the image segmentation model can perform image segmentation on the image to be segmented to obtain the target segmented image.

[0135] Step 212: Send the target segmented image to the client.

[0136] Step 213: The client outputs the target segmented image.

[0137] In practice, the client can present the target segmentation image in its human-computer interaction interface, save the target segmentation image to the local terminal, and send the target segmentation image to other devices that are connected to the terminal.

[0138] By applying the embodiments described above, image reconstruction and segmentation prediction are performed on the first image vector obtained based on the encoder to obtain a first reconstructed image and a first segmented image. Then, the realism of the first reconstructed image and the first segmented image are predicted respectively to obtain a first prediction result. Based on the first prediction result, the model parameters of the image segmentation model are updated. Thus, by combining image reconstruction and image segmentation, the image segmentation model is trained. While reconstructing the image using a generator, the overall distribution of image pixels is modeled, and the features of each pixel are extracted, thereby achieving classification and determination of that pixel. This results in high-quality, high-precision semantic segmentation, improving the accuracy of the segmentation results of the image segmentation model.

[0139] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0140] The practical application of artificial intelligence largely lies in intelligent visual perception. Accurate target segmentation plays a crucial role in visual perception tasks. Traditional deep learning-based methods require vast amounts of labeled data for training. However, in some scenarios, acquiring large amounts of labeled data is extremely difficult, partly because segmentation annotation is more expensive than general annotation, and partly because the data itself is hard to obtain. For example, in medical AI scenarios, accurately segmenting lesion areas and making diagnoses requires professional doctors to annotate the data, which is very challenging. In industrial AI quality inspection scenarios, some parts have highly variable defect morphologies, and some defects initially have very few images, making comprehensive data collection difficult. These situations are not well-suited to fully supervised frameworks.

[0141] Based on this, this application provides a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for an image segmentation model, which can be applied to the above-mentioned scenarios. It models the distribution relationship of the image itself based on image reconstruction to make segmentation decisions. It can be applied to situations where the shape of part defects is ever-changing. During the image reconstruction process, it can capture new defect image distributions and adjust itself to achieve better results. Moreover, the reconstruction process can capture the features of each pixel, so it can also achieve good results for some small defect features. It can be applied to the segmentation of any visual scene, especially for artificial scenes such as medical and industrial settings, where the boundaries between targets and between targets and the background are not as clear as in natural images.

[0142] It should be noted that in the process of image segmentation and image reconstruction, due to the pixel-level distribution modeling capability of generative networks (generators), the network itself naturally has the ability to perform pixel-level semantic segmentation while reconstructing high-quality images. Therefore, generative networks (generators) need to capture the information of each pixel and its relationship with surrounding pixels, that is, they need to model the overall pixel distribution of the image in order to realize the image segmentation process of this application.

[0143] Specifically, the image to be segmented is first encoded into a string of attribute codes (a first image vector or a second image vector) in a feature space, and then input into a generative network. The generative network then reconstructs the image using the attribute feature encoding. Simultaneously, during the reconstruction, a semantic segmentation image (a first segmentation image or a second segmentation image) at the same scale is generated. Thus, the generative network outputs both the reconstructed image of the image to be segmented and the semantic segmentation image. As training progresses, the generative network continuously improves the quality of the reconstructed image, and its ability to model each pixel and capture its overall distribution also continuously improves, resulting in increasingly accurate semantic segmentation images. Since this improvement in segmentation capability is mainly based on the image reconstruction process, the number of semantic segmentation labeled samples required during training is not large. Therefore, this application can be used not only for supervised semantic segmentation but also for semi-supervised semantic segmentation, thereby solving the problem of defect segmentation and detection with small samples in scenarios such as industrial inspection. Therefore, for defect images in industrial scenarios, some defects are relatively few and difficult to collect. In this application, since the improvement of model capabilities depends on the reconstruction quality of its own images, the demand for segmentation annotation (labels) is not very large. Thus, a large amount of unlabeled data can be used for semi-supervised training to solve the problem of insufficient labeled data.

[0144] Furthermore, during model application and inference, the encoding process of reconstructed images can be optimized based on specific images to achieve higher quality image and semantic segmentation image generation. In other words, the image segmentation model of this application has the ability to adapt to its own application scenarios, thereby achieving better model generalization and segmentation performance on out-of-domain data.

[0145] Next, the technical solution of this application will be described from the product side. Here, the technical solution of this application can be applied to scenarios requiring image perception and segmentation, especially for fixed scenarios with small targets, such as industrial AI quality inspection, where the area of ​​the defect can be used to determine whether a part is qualified. For example, in medical images, relevant lesions can be detected and segmented for medical assessment. Simultaneously, it can also be applied to conventional autonomous driving image perception and facial recognition segmentation products.

[0146] Next, the technical solution of this application will be described from a technical perspective.

[0147] First, see Figure 13 , Figure 13 This is a flowchart illustrating the training method for the image segmentation model provided in this application embodiment, based on... Figure 13 This application accepts both labeled data (first image sample) and unlabeled data (second image sample) as input. Its main framework is an adversarial generative network, comprising a generator network (generator) and a discriminator. First, the 2D images (first and second image samples) in the input data are processed by an encoder, converting them into attribute codes W (first and second image vectors). Then, the generator network, using the attribute codes as input, reconstructs the original 2D images (first and second reconstructed images) and generates corresponding segmented images (first and second segmented images) under given attribute coding conditions. The discriminator evaluates the generator's output (first reconstructed image, second reconstructed image, first segmented image, and second segmented image) and tries to distinguish them from the real data pairs (second image pairs) and the real images (first and second image samples). The generator, on the other hand, continuously generates higher-quality images (first and second reconstructed images) and segmented images (first and second segmented images) to deceive the discriminator, making it difficult for it to distinguish between the generated and real data. Through the continuous competition between the generator and the discriminator, the generator's ability to model images is constantly improved, enabling accurate modeling and feature extraction of the overall distribution of the input image. Ultimately, it can reconstruct a sufficiently realistic image from the input image and generate a corresponding high-quality segmented image.

[0148] Next, the training process of the model will be explained. Here, the encoder and generative adversarial network of the entire model are trained jointly. Generally, in one iteration of training, the discriminator is trained first to enable it to distinguish between generated images and real input images, and then the encoder and generative network are trained. During the training process, the model mainly includes labeled data-related loss and unlabeled data-related loss.

[0149] For labeled data, see Figure 14 , Figure 14 This is a schematic diagram of the training process framework for labeled data provided in the embodiments of this application, based on Figure 14The generator can generate the corresponding reconstructed image (first reconstructed image) and the predicted segmented image (first segmented image). Therefore, the difference between the reconstructed image (first reconstructed image) and the original image (first image sample), as well as the difference between the predicted segmented image (first segmented image) and the real label segmentation label (segmentation label), can be calculated as supervision to guide the optimization of the model. When calculating the difference between the reconstructed image and the original image (first reconstruction loss), this method uses mean squared error and LPIPS loss as measures, as shown in Equation (1). Mean squared error mainly calculates the difference between the reconstructed image and the original image at each pixel. LPIPS loss calculates the difference between the reconstructed image and the original image on the convolutional feature layer, which is closer to the result of human visual discrimination. Its calculation requires inputting the two images to be compared into a pre-trained convolutional neural network, and then extracting the features of a certain layer to calculate the difference between the two images. When calculating the difference between the predicted segmented image and the real label segmented image (segmentation loss), this method uses pixel-wise cross-entropy loss and dice loss to evaluate the similarity between the two samples, as shown in Equation (2). Compared to cross-entropy loss, which focuses only on the prediction result at a single pixel, dice loss can capture the regional relationships around a pixel and also solve the problem of imbalanced positive and negative samples. Besides these two direct supervisions, labeled training is also supervised by the discriminator, which is the adversarial training loss (first adversarial loss). Let the labeled image data be denoted as... Its segmentation label is The encoder is The generator is The discriminator is Then there is a label data related loss ( As shown in formula (4).

[0150] It should be noted that, in (That is, in Formula 3-1), the first part means to make the discriminator classify the reconstructed image as true as possible, that is, the reconstructed image is close enough to the original image. The second part means to make the discriminator classify the reconstructed image as true as possible. and They are identified as a pair, meaning that the generated segmented image (first segmented image) and the reconstructed image (first reconstructed image) have the same correspondence as the original image (first image sample) and the label (segmentation label).

[0151] For unlabeled data, see Figure 15 , Figure 15 This is a schematic diagram of the training process framework for unlabeled data provided in an embodiment of this application, based on Figure 15 Unlabeled data is Since unlabeled data lacks pre-labeled segmentation labels, its supervised loss primarily consists of 2D image reconstruction loss (second reconstruction loss) and adversarial training loss (second adversarial loss). In other words, it evaluates the network's ability to reconstruct unlabeled images (second reconstructed images) and whether the generated segmented images (second segmentation images) form a pairing relationship with the images (second reconstructed images). The loss is calculated as shown in formulas (11), (12), and (13), where the meaning and calculation of each symbol are the same as for labeled data.

[0152] ...Formula (11); ...Formula (12); ...Formula (13); In actual training, see Figure 16 , Figure 16 This is a schematic diagram of the input and output during discriminator training provided in an embodiment of this application, based on... Figure 16 First, the discriminator needs to be trained to distinguish real images. and reconstructed images Distinguish between real and false data and the generated data pair Therefore, the discriminator needs to accept both real and generated data as input, and predict the generated data label as 0 and the real data label as 1. The discriminator training loss is shown in equation (6-1). At the same time, based on the definition of model loss, the process of training the discriminator can be shown in equation (8). Next, after training the discriminator, the next step is to train the encoder and generator. According to the aforementioned definition of the loss related to labeled and unlabeled data, this training process can be expressed as equation (9), that is, minimizing the correlation loss between unlabeled and labeled data. Here, in one round of model training, the discriminator and generator successively train the model and update the parameters. The end signal of the entire model training is when the model converges or the number of training rounds reaches the preset number.

[0153] In some embodiments, the training process of the image segmentation model provided in this application can also be optimized and adapted. Specifically, in the model inference stage, given an image (training image sample), the model can directly predict its segmentation label (segmentation image), or it can optimize the model on the target data based on the image reconstruction loss, making the model more adaptable to the target data domain.

[0154] Specifically, the inference optimization process for single image segmentation, such as Figure 11As shown, inference optimization for a single image mainly refers to optimizing the image's attribute encoding vector, resulting in better reconstruction performance after the image passes through the encoder and generator. Generally, a better reconstruction result corresponds to a better segmentation result. This is achieved by reconstructing the image... Re-encode into attribute code (w`) and seek and Consistency in the latent space can lead to better image reconstruction results, as shown in formula (10).

[0155] Regarding the process of target domain adaptation, this refers to the situation where, on the application dataset, there is a difference between the application data and the training data—that is, a difference in data distribution. Directly applying the model to this dataset often results in a decline in performance. In this case, the application data can be treated as unlabeled data, allowing for fine-tuning of the model to achieve better results. This is a beneficial effect of the semi-supervised framework in this technical solution compared to fully supervised models.

[0156] In some embodiments, the encoder and generator can be trained separately. For example, when training the generator alone, a random code z can be used as input for generation, and then the encoder can be trained together with the generator. The encoding space of the encoder is constrained, and the training samples and the random code z are used together as input to the generator.

[0157] It should be noted that the generative network used in this application can be StyleGAN, ProgressiveGAN, etc., and the specific architecture of the generative network used in this application is not limited in the embodiments of this application. Furthermore, the discriminator may not distinguish between reconstructed images and generated data pairs, but may use only one discriminator, such as only discriminating between generated data pairs. In addition, the output format of the discriminator can also be varied, not limited to a simple true / false judgment such as 0 / 1; for example, additional classification outputs may be added. Furthermore, based on the technical solution of this application, in addition to image segmentation, detection functions can also be performed, such as recognizing components, instruments, or faces included in the image.

[0158] By applying the embodiments described above, image reconstruction and segmentation prediction are performed on the first image vector obtained based on the encoder to obtain a first reconstructed image and a first segmented image. Then, the realism of the first reconstructed image and the first segmented image are predicted respectively to obtain a first prediction result. Based on the first prediction result, the model parameters of the image segmentation model are updated. Thus, by combining image reconstruction and image segmentation, the image segmentation model is trained. While reconstructing the image using a generator, the overall distribution of image pixels is modeled, and the features of each pixel are extracted, thereby achieving classification and determination of that pixel. This results in high-quality, high-precision semantic segmentation, improving the accuracy of the segmentation results of the image segmentation model.

[0159] The following description continues to illustrate the exemplary structure of the training device 455 for the image segmentation model provided in this application embodiment as a software module. The image segmentation model includes at least an encoder, a generator, and a discriminator. In some embodiments, such as... Figure 2 As shown, the software modules in the training device 455 for the image segmentation model stored in the memory 450 may include: The acquisition module 4551 is used to acquire a first image sample carrying a segmentation label, wherein the segmentation label is a standard segmentation image of the first image sample; The encoding module 4552 is used to encode the first image sample through the encoder to obtain a first image vector corresponding to the first image sample; The generation module 4553 is used to perform image reconstruction and image segmentation prediction on the first image sample based on the first image vector through the generator, so as to obtain the first reconstructed image and the first segmented image corresponding to the first image sample. The discrimination module 4554 is used to predict the authenticity of the first reconstructed image and the first segmented image based on the first image sample and the segmentation label through the discriminator, and obtain a first prediction result. The prediction update module 4555 is used to determine the value of the loss function of the image segmentation model based on the first prediction result, and update the model parameters of the image segmentation model based on the value of the loss function.

[0160] In some embodiments, the first prediction result includes a first realism prediction result and a second realism prediction result; the discrimination module 4554 is further configured to match the first reconstructed image with the first image sample through the discriminator to obtain a first matching result, and determine a first realism prediction result indicating the realism of the first reconstructed image based on the first matching result; match the first image pair with the second image pair through the discriminator to obtain a second matching result, and determine a second realism prediction result indicating the realism of the first segmented image based on the second matching result; wherein the first image pair consists of the first reconstructed image and the first segmented image, and the second image pair consists of the first image sample and the standard segmented image.

[0161] In some embodiments, the discrimination module 4554 is further configured to: match the first reconstructed image with the first image sample pixel by pixel using the discriminator to obtain a first sub-matching result of the first reconstructed image and the first image sample in the pixel dimension; extract features from the first reconstructed image and the first image sample respectively using the discriminator to obtain a first image feature of the first reconstructed image and a second image feature of the first image sample, and match the first image feature with the second image feature to obtain a second sub-matching result of the first reconstructed image and the first image sample in the feature dimension; and perform a weighted summation of the first sub-matching result and the second sub-matching result to obtain a first matching result.

[0162] In some embodiments, the apparatus further includes a matching module, which is configured to match the first segmented image with the standard segmented image through the discriminator to obtain a third matching result; the discriminator module 4554 is further configured to perform a weighted summation of the second matching result and the third matching result to obtain a second authenticity prediction result for indicating the authenticity of the first segmented image.

[0163] In some embodiments, the discrimination module 4554 is further configured to: extract features from the first reconstructed image and the first segmented image using the discriminator to obtain a first image feature of the first reconstructed image and a second image feature of the first image sample; extract features from the first image sample and the standard segmented image to obtain a third image feature of the first image sample and a fourth image feature of the standard segmented image; concatenate the first image feature and the second image feature using the discriminator to obtain a first concatenated feature; concatenate the third image feature and the fourth image feature to obtain a second concatenated feature; and match the first concatenated feature and the second concatenated feature using the discriminator to obtain a second matching result.

[0164] In some embodiments, the first prediction result includes a first realism prediction result for the first reconstructed image and a second realism prediction result for the first segmented image. The prediction update module 4555 is further configured to obtain a first loss function, a second loss function, and a third loss function included in the loss function of the image segmentation model; determine the reconstruction loss of the first image sample by combining the first realism prediction result and the first loss function; determine the segmentation loss of the first image sample by combining the second realism prediction result and the second loss function; determine the adversarial loss of the first image sample by combining the first realism prediction result, the second realism prediction result, and the third loss function; and perform a weighted summation of the reconstruction loss, the segmentation loss, and the adversarial loss of the first image sample to obtain the value of the loss function of the image segmentation model.

[0165] In some embodiments, the value of the loss function includes: a reconstruction loss determined based on a first realism prediction result of the first reconstructed image, a segmentation loss determined based on a second realism prediction result of the first segmented image, and an adversarial loss determined by combining the first realism prediction result and the second realism prediction result; the prediction update module 4555 is further configured to fix the parameters of the encoder and the generator, update the parameters of the discriminator based on the adversarial loss; fix the parameters of the discriminator, and update the parameters of the encoder and the generator by combining the reconstruction loss and the segmentation loss.

[0166] In some embodiments, the apparatus further includes a second image sample processing module, which is configured to encode a second image sample without segmentation labels using the encoder to obtain a second image vector corresponding to the second image sample; perform image reconstruction and segmentation image prediction on the second image sample based on the second image vector using the generator to obtain a second reconstructed image and a second segmented image corresponding to the second image sample; and predict the authenticity of the second reconstructed image and the second segmented image based on the second image sample and the segmentation labels using the discriminator to obtain a second prediction result; the prediction update module 4555 is further configured to determine the value of the loss function of the image segmentation model based on the first prediction result and the second prediction result, and update the model parameters of the image segmentation model based on the value of the loss function.

[0167] In some embodiments, the second prediction result includes a third realism prediction result and a fourth realism prediction result; the discrimination module 4554 is further configured to match the second reconstructed image with the second image sample through the discriminator to obtain a fourth matching result, and determine a third realism prediction result to indicate the realism of the second reconstructed image based on the fourth matching result; match the third image pair with the second image pair through the discriminator to obtain a fifth matching result, and determine a fourth realism prediction result to indicate the realism of the second segmented image based on the fifth matching result; wherein the third image pair consists of the second reconstructed image and the second segmented image, and the second image pair consists of the first image sample and the standard segmented image.

[0168] In some embodiments, the similarity between the first image sample and the second image sample is lower than a similarity threshold; the discrimination module 4554 is further configured to obtain the mapping relationship between the fifth matching result and the fourth authenticity prediction result; based on the fifth matching result and the mapping relationship, determine the fourth authenticity prediction result used to indicate the authenticity of the second segmented image; wherein the authenticity of the second segmented image is negatively correlated with the fifth matching result.

[0169] In some embodiments, the first prediction result includes a first realism prediction result for the first reconstructed image and a second realism prediction result for the first segmented image. The second prediction result includes a third realism prediction result for the second reconstructed image and a fourth realism prediction result for the second segmented image. The prediction update module 4555 is further configured to obtain a fourth loss function, a fifth loss function, and a sixth loss function included in the loss function of the image segmentation model; combine the first realism prediction result and the fourth loss function to determine a first reconstruction loss for the first image sample; combine the third realism prediction result and the fourth loss function to determine a second reconstruction loss for the second image sample; and perform a weighted summation of the first reconstruction loss and the second reconstruction loss to obtain... The total reconstruction loss for the first image sample and the second image sample is obtained; the segmentation loss for the first image sample is determined by combining the second realism prediction result and the fifth loss function; the first adversarial loss for the first image sample is determined by combining the first realism prediction result, the second realism prediction result and the sixth loss function, and the second adversarial loss for the second image sample is determined by combining the third realism prediction result, the fourth realism prediction result and the sixth loss function; the first adversarial loss and the second adversarial loss are weighted and summed to obtain the total adversarial loss for the first image sample and the second image sample; the total reconstruction loss, the segmentation loss and the total adversarial loss are weighted and summed to obtain the value of the loss function of the image segmentation model.

[0170] In some embodiments, the generation module 4553 is further configured to decode the first image vector through the generator to obtain a first reconstructed image and a first segmented image corresponding to the first image sample.

[0171] In some embodiments, the generation module 4553 is further configured to decode the first image vector in the red-green-blue channels respectively to obtain the decoding results corresponding to each color channel, and to fuse the decoding results corresponding to each color channel to obtain the first reconstructed image corresponding to the first image sample; and to perform single-channel decoding on the first image vector to obtain the first segmented image corresponding to the first image sample.

[0172] In the above scheme, the device further includes an application module, which is used to acquire the image to be segmented and the trained target image segmentation model; to encode the image to be segmented based on the encoder of the target image segmentation model to obtain the target image vector of the image to be segmented; and to perform segmentation image prediction on the image to be segmented based on the target image vector through the generator of the target image segmentation model to obtain the target segmented image of the image to be segmented.

[0173] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the image segmentation model training method described in this application.

[0174] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the training method for the image segmentation model provided in this application. For example, ... Figure 3 The training method for the image segmentation model is shown.

[0175] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0176] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0177] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0178] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0179] In summary, the embodiments of this application have the following beneficial effects: (1) By combining image reconstruction and image segmentation, the image segmentation model is trained. While reconstructing the image through the generator, the overall distribution of image pixels is modeled and the features of each pixel are extracted, thereby realizing the classification judgment of the pixel and achieving high-quality and high-precision semantic segmentation, which improves the accuracy of the segmentation results of the image segmentation model.

[0180] (2) During the inference stage of the model, the image encoding can be optimized based on the reconstruction effect of the image, thus enabling it to adapt to certain image scenes during application. It also has good generalization ability when the image changes.

[0181] (3) By first preprocessing the first image sample, the influence of factors such as lighting and shadow on the image segmentation process is reduced, thereby improving the training efficiency and segmentation accuracy of the image segmentation model.

[0182] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A training method for an image segmentation model, characterized in that, The image segmentation model includes at least an encoder, a generator, and a discriminator; the method includes: Obtain a first image sample carrying a segmentation label, wherein the segmentation label is a standard segmentation image of the first image sample; The encoder encodes the first image sample to obtain a first image vector corresponding to the first image sample; and encodes the second image sample without segmentation labels to obtain a second image vector corresponding to the second image sample. The generator performs image reconstruction and segmentation prediction on the first image sample based on the first image vector to obtain a first reconstructed image and a first segmented image corresponding to the first image sample; and performs image reconstruction and segmentation prediction on the second image sample based on the second image vector to obtain a second reconstructed image and a second segmented image corresponding to the second image sample. The discriminator predicts the authenticity of the first reconstructed image and the first segmented image based on the first image sample and the segmentation label, respectively, to obtain a first prediction result; and predicts the authenticity of the second reconstructed image and the second segmented image based on the second image sample and the segmentation label, respectively, to obtain a second prediction result. Based on the first prediction result and the second prediction result, the value of the loss function of the image segmentation model is determined, and the model parameters of the image segmentation model are updated based on the value of the loss function.

2. The method as described in claim 1, characterized in that, The first prediction result includes a first realism prediction result and a second realism prediction result; the step of using the discriminator to predict the realism of the first reconstructed image and the first segmented image based on the first image sample and the segmentation label to obtain the first prediction result includes: The discriminator matches the first reconstructed image with the first image sample to obtain a first matching result, and based on the first matching result, determines a first authenticity prediction result to indicate the authenticity of the first reconstructed image. The discriminator matches the first image pair with the second image pair to obtain a second matching result, and based on the second matching result, determines a second realism prediction result to indicate the realism of the first segmented image. The first image pair consists of the first reconstructed image and the first segmented image, and the second image pair consists of the first image sample and the standard segmented image.

3. The method as described in claim 2, characterized in that, The step of matching the first reconstructed image with the first image sample through the discriminator to obtain a first matching result includes: The discriminator performs pixel-by-pixel matching between the first reconstructed image and the first image sample to obtain the first sub-matching result between the first reconstructed image and the first image sample in the pixel dimension. The discriminator extracts features from the first reconstructed image and the first image sample to obtain the first image feature of the first reconstructed image and the second image feature of the first image sample. The first image feature and the second image feature are then matched to obtain the second sub-matching result of the first reconstructed image and the first image sample in the feature dimension. The first sub-matching result and the second sub-matching result are weighted and summed to obtain the first matching result.

4. The method as described in claim 2, characterized in that, The method further includes: The discriminator matches the first segmented image with the standard segmented image to obtain a third matching result. The step of determining a second realism prediction result, based on the second matching result, to indicate the realism of the first segmented image includes: The second matching result and the third matching result are weighted and summed to obtain a second authenticity prediction result used to indicate the authenticity of the first segmented image.

5. The method as described in claim 2, characterized in that, The step of matching the first image pair with the second image pair using the discriminator to obtain a second matching result includes: The discriminator extracts features from the first reconstructed image and the first segmented image to obtain the first image features of the first reconstructed image and the second image features of the first segmented image. It also extracts features from the first image sample and the standard segmented image to obtain the third image features of the first image sample and the fourth image features of the standard segmented image. The discriminator concatenates the first image features and the second image features to obtain a first concatenated feature, and concatenates the third image features and the fourth image features to obtain a second concatenated feature. The discriminator matches the first splicing feature with the second splicing feature to obtain the second matching result.

6. The method as described in claim 1, characterized in that, The first prediction result includes a first ground truth prediction result for the first reconstructed image and a second ground truth prediction result for the first segmented image. Determining the value of the loss function of the image segmentation model based on the first prediction result and the second prediction result includes: The loss functions of the image segmentation model include the first loss function, the second loss function, and the third loss function; Based on the first realism prediction result and the first loss function, the reconstruction loss of the first image sample is determined; By combining the second realism prediction result and the second loss function, the segmentation loss of the first image sample is determined; By combining the first realism prediction result, the second realism prediction result, and the third loss function, the adversarial loss of the first image sample is determined; The reconstruction loss, segmentation loss, and adversarial loss of the first image sample are weighted and summed to obtain the value of the loss function of the image segmentation model.

7. The method as described in claim 1, characterized in that, The value of the loss function includes: reconstruction loss determined based on the first realism prediction result of the first reconstructed image, segmentation loss determined based on the second realism prediction result of the first segmented image, and adversarial loss determined by combining the first realism prediction result and the second realism prediction result. Updating the model parameters of the image segmentation model based on the value of the loss function includes: With the parameters of the encoder and the generator fixed, the parameters of the discriminator are updated based on the adversarial loss; With the parameters of the discriminator fixed, the parameters of the encoder and the generator are updated by combining the reconstruction loss and the segmentation loss.

8. The method as described in claim 1, characterized in that, The second prediction result includes a third realism prediction result and a fourth realism prediction result; the second prediction result is obtained by predicting the realism of the second reconstructed image and the second segmented image based on the second image sample and the segmentation label, respectively, including: The discriminator matches the second reconstructed image with the second image sample to obtain a fourth matching result, and based on the fourth matching result, determines a third realism prediction result to indicate the realism of the second reconstructed image. The discriminator matches the third image pair with the second image pair to obtain a fifth matching result, and based on the fifth matching result, determines a fourth realism prediction result to indicate the realism of the second segmented image. The third image pair consists of the second reconstructed image and the second segmented image, and the second image pair consists of the first image sample and the standard segmented image.

9. The method as described in claim 8, characterized in that, The similarity between the first image sample and the second image sample is lower than the similarity threshold; The step of determining a fourth realism prediction result, based on the fifth matching result, to indicate the realism of the second segmented image includes: Obtain the mapping relationship between the fifth matching result and the fourth authenticity prediction result; Based on the fifth matching result and the mapping relationship, a fourth authenticity prediction result is determined to indicate the authenticity of the second segmented image; wherein the authenticity of the second segmented image is negatively correlated with the fifth matching result.

10. The method as described in claim 8, characterized in that, The first prediction result includes a first realism prediction result for the first reconstructed image and a second realism prediction result for the first segmented image. The second prediction result includes a third realism prediction result for the second reconstructed image and a fourth realism prediction result for the second segmented image. Determining the value of the loss function of the image segmentation model based on the first prediction result and the second prediction result includes: The loss functions of the image segmentation model include the fourth, fifth, and sixth loss functions; Combining the first realism prediction result and the fourth loss function, the first reconstruction loss of the first image sample is determined, and combining the third realism prediction result and the fourth loss function, the second reconstruction loss of the second image sample is determined; the first reconstruction loss and the second reconstruction loss are weighted and summed to obtain the total reconstruction loss for the first image sample and the second image sample. Combining the second realism prediction result and the fifth loss function, the segmentation loss of the first image sample is determined; By combining the first realism prediction result, the second realism prediction result, and the sixth loss function, the first adversarial loss of the first image sample is determined, and by combining the third realism prediction result, the fourth realism prediction result, and the sixth loss function, the second adversarial loss of the second image sample is determined; the first adversarial loss and the second adversarial loss are weighted and summed to obtain the total adversarial loss for the first image sample and the second image sample. The total reconstruction loss, the segmentation loss, and the total adversarial loss are weighted and summed to obtain the value of the loss function of the image segmentation model.

11. The method as described in claim 1, characterized in that, After updating the model parameters of the image segmentation model based on the value of the loss function, the method further includes: Obtain the image to be segmented, and the trained target image segmentation model; The encoder based on the target image segmentation model encodes the image to be segmented to obtain the target image vector of the image to be segmented; The generator of the target image segmentation model predicts the segmentation image of the image to be segmented based on the target image vector, thereby obtaining the target segmentation image of the image to be segmented.

12. A training device for an image segmentation model, characterized in that, The image segmentation model includes at least an encoder, a generator, and a discriminator, and the device includes: The acquisition module is used to acquire a first image sample carrying a segmentation label, wherein the segmentation label is a standard segmentation image of the first image sample; The encoding module is used to encode the first image sample through the encoder to obtain a first image vector corresponding to the first image sample; and to encode the second image sample without segmentation labels to obtain a second image vector corresponding to the second image sample. The generation module is used to perform image reconstruction and segmentation prediction on the first image sample based on the first image vector through the generator to obtain a first reconstructed image and a first segmented image corresponding to the first image sample; and to perform image reconstruction and segmentation prediction on the second image sample based on the second image vector to obtain a second reconstructed image and a second segmented image corresponding to the second image sample. The discrimination module is used to predict the authenticity of the first reconstructed image and the first segmented image based on the first image sample and the segmentation label, respectively, to obtain a first prediction result; and to predict the authenticity of the second reconstructed image and the second segmented image based on the second image sample and the segmentation label, respectively, to obtain a second prediction result. The prediction update module is used to determine the value of the loss function of the image segmentation model based on the first prediction result and the second prediction result, and update the model parameters of the image segmentation model based on the value of the loss function.

13. The apparatus according to claim 12, characterized in that, The first prediction result includes a first true prediction result and a second true prediction result; The discrimination module is further configured to match the first reconstructed image with the first image sample through the discriminator to obtain a first matching result, and based on the first matching result, determine a first authenticity prediction result for indicating the authenticity of the first reconstructed image. The discriminator matches the first image pair with the second image pair to obtain a second matching result, and based on the second matching result, determines a second realism prediction result to indicate the realism of the first segmented image. The first image pair consists of the first reconstructed image and the first segmented image, and the second image pair consists of the first image sample and the standard segmented image.

14. The apparatus according to claim 13, characterized in that, The discrimination module is further configured to match the first reconstructed image with the first image sample pixel by pixel through the discriminator to obtain a first sub-matching result between the first reconstructed image and the first image sample in the pixel dimension. The discriminator extracts features from the first reconstructed image and the first image sample to obtain the first image feature of the first reconstructed image and the second image feature of the first image sample. The first image feature and the second image feature are then matched to obtain the second sub-matching result of the first reconstructed image and the first image sample in the feature dimension. The first sub-matching result and the second sub-matching result are weighted and summed to obtain the first matching result.

15. The apparatus according to claim 13, characterized in that, The device further includes: The matching module is used to match the first segmented image with the standard segmented image through the discriminator to obtain a third matching result; The discrimination module is further configured to perform a weighted summation of the second matching result and the third matching result to obtain a second authenticity prediction result used to indicate the authenticity of the first segmented image.

16. The apparatus according to claim 13, characterized in that, The discrimination module is further configured to extract features from the first reconstructed image and the first segmented image through the discriminator to obtain the first image features of the first reconstructed image and the second image features of the first segmented image, and to extract features from the first image sample and the standard segmented image to obtain the third image features of the first image sample and the fourth image features of the standard segmented image. The discriminator concatenates the first image features and the second image features to obtain a first concatenated feature, and concatenates the third image features and the fourth image features to obtain a second concatenated feature. The discriminator matches the first splicing feature with the second splicing feature to obtain the second matching result.

17. The apparatus according to claim 12, characterized in that, The first prediction result includes a first realism prediction result for the first reconstructed image and a second realism prediction result for the first segmented image; The prediction update module is also used to obtain the first loss function, the second loss function and the third loss function of the image segmentation model; Based on the first realism prediction result and the first loss function, the reconstruction loss of the first image sample is determined; By combining the second realism prediction result and the second loss function, the segmentation loss of the first image sample is determined; By combining the first realism prediction result, the second realism prediction result, and the third loss function, the adversarial loss of the first image sample is determined; The reconstruction loss, segmentation loss, and adversarial loss of the first image sample are weighted and summed to obtain the value of the loss function of the image segmentation model.

18. The apparatus according to claim 12, characterized in that, The value of the loss function includes: reconstruction loss determined based on the first realism prediction result of the first reconstructed image, segmentation loss determined based on the second realism prediction result of the first segmented image, and adversarial loss determined by combining the first realism prediction result and the second realism prediction result. The prediction update module is also used to fix the parameters of the encoder and the generator, and update the parameters of the discriminator based on the adversarial loss; With the parameters of the discriminator fixed, the parameters of the encoder and the generator are updated by combining the reconstruction loss and the segmentation loss.

19. The apparatus according to claim 12, characterized in that, The second prediction result includes the third and fourth true prediction results; The discrimination module is further configured to match the second reconstructed image with the second image sample through the discriminator to obtain a fourth matching result, and based on the fourth matching result, determine a third realism prediction result to indicate the realism of the second reconstructed image; The discriminator matches the third image pair with the second image pair to obtain a fifth matching result, and based on the fifth matching result, determines a fourth realism prediction result to indicate the realism of the second segmented image. The third image pair consists of the second reconstructed image and the second segmented image, and the second image pair consists of the first image sample and the standard segmented image.

20. The apparatus according to claim 19, characterized in that, The similarity between the first image sample and the second image sample is lower than the similarity threshold; The discrimination module is also used to obtain the mapping relationship between the fifth matching result and the fourth authenticity prediction result; Based on the fifth matching result and the mapping relationship, a fourth authenticity prediction result is determined to indicate the authenticity of the second segmented image; wherein the authenticity of the second segmented image is negatively correlated with the fifth matching result.

21. The apparatus according to claim 19, characterized in that, The first prediction result includes a first realism prediction result for the first reconstructed image and a second realism prediction result for the first segmented image. The second prediction result includes a third realism prediction result for the second reconstructed image and a fourth realism prediction result for the second segmented image. The prediction update module is also used to obtain the fourth loss function, the fifth loss function and the sixth loss function included in the loss function of the image segmentation model; Combining the first realism prediction result and the fourth loss function, the first reconstruction loss of the first image sample is determined, and combining the third realism prediction result and the fourth loss function, the second reconstruction loss of the second image sample is determined; the first reconstruction loss and the second reconstruction loss are weighted and summed to obtain the total reconstruction loss for the first image sample and the second image sample. Combining the second realism prediction result and the fifth loss function, the segmentation loss of the first image sample is determined; By combining the first realism prediction result, the second realism prediction result, and the sixth loss function, the first adversarial loss of the first image sample is determined, and by combining the third realism prediction result, the fourth realism prediction result, and the sixth loss function, the second adversarial loss of the second image sample is determined; the first adversarial loss and the second adversarial loss are weighted and summed to obtain the total adversarial loss for the first image sample and the second image sample. The total reconstruction loss, the segmentation loss, and the total adversarial loss are weighted and summed to obtain the value of the loss function of the image segmentation model.

22. The apparatus according to claim 12, characterized in that, The device further includes: The application module is used to acquire the image to be segmented and the trained target image segmentation model; The encoder based on the target image segmentation model encodes the image to be segmented to obtain the target image vector of the image to be segmented; The generator of the target image segmentation model predicts the segmentation image of the image to be segmented based on the target image vector, thereby obtaining the target segmentation image of the image to be segmented.

23. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the training method of the image segmentation model according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that, It stores executable instructions for causing a processor to execute, thereby implementing the training method of the image segmentation model according to any one of claims 1 to 11.

25. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the training method of the image segmentation model according to any one of claims 1 to 11.

Images