Training method of image segmentation model and related device

Abstract

The application provides a training method of an image segmentation model and related devices, applied to the field of artificial intelligence, comprising: obtaining a first image sample without carrying a segmentation label and at least one second image sample carrying a segmentation label; performing image segmentation on the first image sample and each second image sample respectively through an initial image segmentation model to obtain a first segmentation result of the first image sample and a second segmentation result of the second image sample; determining a value of a first loss function based on a probability graph of the first segmentation result in the pixel dimension; determining a value of a second loss function based on the difference between each second segmentation result and the segmentation label; and training the initial image segmentation model by combining the value of the first loss function and the value of each second loss function to obtain a target image segmentation model. Through the application, the accuracy of the segmentation result of the image segmentation model can be improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and in particular to a training method and related apparatus for an image segmentation model. Background Technology

[0002] Target localization has wide applications in industrial automation, target detection, and video viewing. However, current methods for locating targets in images often suffer from poor accuracy and precision in complex real-world situations (such as deformed targets), or require extensive image data annotation, leading to high annotation costs and consequently poor segmentation accuracy in image segmentation models. Summary of the Invention

[0003] This application provides a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for an image segmentation model, which can improve the accuracy of the segmentation results of the image segmentation model.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a method for training an image segmentation model, including: Obtain a joint training sample set, which includes a first image sample without segmentation labels and at least one second image sample with segmentation labels, wherein the segmentation label is the standard segmentation image of the second image sample; An initial image segmentation model is obtained, and the first image sample is segmented using the initial image segmentation model to obtain a first segmentation result. Then, each of the second image samples is segmented to obtain a second segmentation result. Obtain a probability map of the first segmentation result in the pixel dimension, and determine the value of the first loss function based on the probability map. The probability map is used to indicate the probability that each pixel in the first image sample belongs to each segmentation region in the first segmentation result. Obtain the difference between each of the second segmentation results and the segmentation label, and determine the value of the second loss function based on the difference; By combining the values ​​of the first loss function and each of the second loss functions, the initial image segmentation model is trained to obtain the target image segmentation model.

[0005] This application provides a training apparatus for an image segmentation model, comprising: The acquisition module is used to acquire a joint training sample set, which includes a first image sample without segmentation labels and at least one second image sample with segmentation labels, wherein the segmentation label is a standard segmentation image of the second image sample. An initial segmentation module is used to obtain an initial image segmentation model, and to perform image segmentation on the first image sample using the initial image segmentation model to obtain a first segmentation result, and to perform image segmentation on each of the second image samples to obtain a second segmentation result; The first determining module is used to obtain a probability map of the first segmentation result in the pixel dimension, and determine the value of the first loss function based on the probability map. The probability map is used to indicate the probability that each pixel in the first image sample belongs to each segmentation region in the first segmentation result. The second determining module is used to obtain the difference between each of the second segmentation results and the segmentation label, and to determine the value of the second loss function based on the difference; The training module is used to train the initial image segmentation model by combining the value of the first loss function and the value of each of the second loss functions to obtain the target image segmentation model.

[0006] This application provides an electronic device, including: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the training method of the image segmentation model provided in the embodiments of this application.

[0007] This application provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor will execute the training method of the image segmentation model provided in this application.

[0008] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image segmentation model training method provided in this application.

[0009] The embodiments of this application have the following beneficial effects: Applying the embodiments of this application, firstly, an initial image segmentation model is used to determine the first segmentation result of a first image sample without segmentation labels, and the second segmentation result of a second image sample with segmentation labels. Then, the value of a second loss function is determined based on the difference between the second segmentation result and the segmentation labels, and the value of a first loss function is determined based on the probability map of the first segmentation result. Thus, the initial image segmentation model is trained by combining the values ​​of the two determined loss functions to obtain the target image segmentation model. In this way, based on the probability map, accurate classification is achieved for each pixel, thereby realizing high-quality and high-precision semantic segmentation and improving the accuracy of the segmentation results of the image segmentation model. Attached Figure Description

[0010] Figure 1A-1B This is a schematic diagram of the architecture of the training system 100 for the image segmentation model provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of an electronic device 500 that implements the training method of the image segmentation model according to an embodiment of this application; Figure 3 This is a flowchart illustrating the training method of the image segmentation model provided in the embodiments of this application; Figure 4 This is a schematic diagram of a second image sample carrying segmentation labels provided in an embodiment of this application; Figures 5A-5B This is a model structure diagram of the initial image segmentation model provided in the embodiments of this application; Figure 6 This is a schematic diagram of the probability graph provided in the embodiments of this application; Figure 7 This is a flowchart illustrating the method for determining the value of the first loss function provided in an embodiment of this application; Figure 8 This is a schematic diagram illustrating the method for obtaining historical probability maps provided in an embodiment of this application; Figure 9 This is a flowchart illustrating the method for determining the first area provided in an embodiment of this application; Figure 10 This is a schematic diagram of the normalization results provided in the embodiments of this application; Figure 11 This is another schematic diagram illustrating the method for determining the value of the first loss function provided in the embodiments of this application; Figure 12 This is a schematic diagram of the training process of the initial image segmentation model provided in the embodiments of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0013] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0015] It should be noted that in the embodiments of this application, real-time acquired images and other related data are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0016] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0017] 1) Semantic segmentation: Also known as image segmentation, it refers to the process of subdividing a digital image into multiple image sub-regions (sets of pixels) (also called superpixels). It's the technique and process of dividing an image into several specific regions with unique properties and identifying objects of interest. It's a key step from image processing to image analysis. The purpose of semantic segmentation is to simplify or change the representation of an image, making it easier to understand and analyze. Semantic segmentation is commonly used to locate objects and boundaries (lines, curves, etc.) in an image. More precisely, semantic segmentation is a process of labeling each pixel in an image, ensuring that pixels with the same label share certain common visual characteristics.

[0018] For example, in an image, pixels of objects belonging to the same category are grouped together and marked with the same color, which is pixel-by-pixel classification. For instance, in a natural image, all pixels belonging to trees are marked with one color, all people are marked with one color pixel-by-pixel, the sky is marked with one color, and so on.

[0019] 2) Image segmentation: The resulting image obtained from semantic segmentation is also called a segmentation mask.

[0020] 3) Cross-entropy: Cross-entropy measures the difference between two different probability distributions of the same random variable. In machine learning, it represents the difference between the true probability distribution and the predicted probability distribution. The smaller the cross-entropy value, the better the model's prediction performance. Cross-entropy is often used in conjunction with the softmax classifier to process the output so that the sum of the predicted values ​​of multiple classes is 1, and then the loss is calculated using cross-entropy.

[0021] 4) Semi-supervised learning refers to training a model using both labeled and unlabeled data simultaneously. It is typically applied when labeled data is scarce or when labels are difficult to obtain, thereby improving the model's ability on limited labeled data.

[0022] 5) Computer Vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to using cameras and computers to replace human eyes in recognizing and measuring targets, and then performing image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0023] 6) Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0024] 7) COCO Dataset: Short for Microsoft Common Objects in Context (MS COCO), this is a large and rich dataset for object detection, segmentation, and captioning. It primarily extracts objects from images in complex everyday scenes and assigns their locations through precise segmentation. The dataset includes 91 object classes, 328,000 images, and 2,500,000 labels. It is the largest dataset to date for semantic segmentation, providing 80 classes and over 330,000 images, of which 200,000 are labeled. The total number of individuals in the dataset exceeds 1.5 million.

[0025] Based on the above explanation of the nouns and terms used in the embodiments of this application, the training system for the image segmentation model provided in the embodiments of this application is described below. See also Figure 1A , Figure 1A This is a schematic diagram of the architecture of the training system 100 for the image segmentation model provided in this application embodiment. In order to support an exemplary application, the terminal (terminal 400-1 and terminal 400-2 are shown as examples) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two, and data transmission is achieved using wireless or wired links.

[0026] In some embodiments, the terminal (such as terminal 400-1 and terminal 400-2) is used as a human interaction interface for a client (such as client 410-1 and client 410-2) based on image segmentation. Upon receiving a trigger operation for image segmentation of the image to be segmented, the terminal sends an image segmentation request carrying the image to be segmented to the server 200. In some embodiments, server 200 is configured to receive an image segmentation request sent by a terminal, and in response to the request, return a segmentation result for the image to be segmented to the terminal using a trained target image segmentation model; Before receiving an image segmentation request for the image to be segmented, server 200 also performs a training process for an initial image segmentation model: The server obtains a joint training sample set, which includes a first image sample without segmentation labels and at least one second image sample with segmentation labels, where the segmentation label is the standard segmentation image of the second image sample; it obtains an initial image segmentation model and uses this model to segment the first image sample to obtain a first segmentation result, and then segments each second image sample to obtain a second segmentation result; it obtains a probability map of the first segmentation result in the pixel dimension, and determines the value of a first loss function based on the probability map, where the probability map indicates the probability that each pixel in the first image sample belongs to a segmentation region in the first segmentation result; it determines the value of a second loss function based on the difference between each second segmentation result and the segmentation label; and it trains the initial image segmentation model by combining the values ​​of the first loss function and each second loss function to obtain a target image segmentation model.

[0027] In some embodiments, server 200 may be a server cluster or a distributed system consisting of multiple servers. Taking a blockchain system as an example, multiple servers may form a blockchain network, and server 200 may be a node on the blockchain network.

[0028] The following example illustrates an exemplary application of blockchain networks, using the example of multiple servers connecting to a blockchain network to train a speech model.

[0029] In some embodiments, see Figure 1B , Figure 1B This application provides an embodiment of the architecture diagram of the image segmentation model training system 100. Multiple servers involved in the speech model jointly execute the training process of the image segmentation model, such as terminal 600 and terminal 700. After obtaining authorization from the blockchain management platform 900, the client 610 of terminal 600 and the client 710 of terminal 700 can access the blockchain network 800.

[0030] Terminal 600 sends an image segmentation model request to the blockchain management platform 900 (terminal 700 sends an image segmentation model request to the blockchain management platform 900). The blockchain management platform 900 generates a corresponding update operation based on the image segmentation model request. The update operation specifies the smart contract to be called to implement the update / query operation, as well as the parameters to be passed to the smart contract. The transaction also carries a digital signature of the webpage signing and sends the update operation to the blockchain network 800.

[0031] When nodes 210-1, 210-2, and 210-3 in blockchain network 800 receive an update operation, they verify the digital signature of the update operation. If the digital signature verification is successful, they confirm whether client 610 has the necessary access rights based on the identity of client 610 carried in the update operation. Either verification of the digital signature or the access rights will result in failure. After successful verification, node 210 signs its own digital signature (e.g., by encrypting the transaction digest using node 210-1's private key) and continues broadcasting it within blockchain network 800.

[0032] After receiving a successfully verified acquisition request, nodes 210-1, 210-2, and 210-3 in the blockchain network 800, which have sorting functions, fill the acquisition request into a new block and broadcast it to the nodes in the blockchain network 800 that provide consensus services.

[0033] In blockchain network 800, nodes providing consensus services reach an agreement on new blocks. Nodes providing ledger functionality append the new block to the end of the blockchain and execute retrieval requests within the new block: for submitted speech model requests, they update the key-value pairs corresponding to the speech model in the state database; for speech model retrieval requests, they query the key-value pairs corresponding to the speech model from the state database and send the corresponding speech model to the terminal. After receiving the initial image segmentation model returned by blockchain network 800, terminals 600 and 700 train the image segmentation model to obtain the trained image segmentation model and display a training success message in graphical interfaces 610-1 and 710-1. Terminals 600 and 700 send the trained image segmentation model to blockchain network 800, which then uses the trained image segmentation model to perform image segmentation processing based on the image to be segmented, obtaining the segmentation result of the image to be segmented.

[0034] In practical applications, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals (such as terminals 400-1 and 400-2) can be smartphones, tablets, laptops, desktop computers, smart speakers, smart TVs, smartwatches, etc., but are not limited to these. Terminals (such as terminals 400-1 and 400-2) and server 200 can be directly or indirectly connected via wired or wireless communication; this application does not impose any restrictions on this connection.

[0035] The electronic device that implements the training method of the image segmentation model provided in the embodiments of this application will now be described. See also Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device 500 that implements the training method for an image segmentation model according to an embodiment of this application. The electronic device 500 can be... Figure 1A The server 200 and electronic device 500 shown can also be terminals capable of implementing the training method for the image segmentation model provided in this application, with electronic device 500 as the... Figure 1A Taking the server shown as an example, an electronic device implementing the image segmentation model training method of this application embodiment will be described. The electronic device 500 provided in this application embodiment includes: at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It is understood that the bus system 540 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 540.

[0036] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0037] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0038] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0039] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0040] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0041] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.; presentation module 553 is used to enable the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 associated with user interface 530 (e.g., display screen, speaker, etc.); input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0042] In some embodiments, the training apparatus for the image segmentation model provided in this application can be implemented in software. Figure 2A training device 555 for an image segmentation model stored in memory 550 is shown. It can be software in the form of programs and plug-ins, including the following software modules: acquisition module 5551, initial segmentation module 5552, first determination module 5553, second determination module 5554, and training module 5555. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0043] In other embodiments, the training device for the image segmentation model provided in this application can be implemented using a combination of hardware and software. As an example, the training device for the image segmentation model provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method for the image segmentation model provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0044] In some embodiments, the terminal or server can implement the image segmentation model training method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as an instant messaging APP or a web browser APP; it can also be a mini-program, that is, a program that only needs to be downloaded into a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0045] Based on the above description of the image segmentation model training system and electronic device provided in the embodiments of this application, the training method of the image segmentation model provided in the embodiments of this application is described below. In actual implementation, the training method of the image segmentation model provided in the embodiments of this application can be implemented by the terminal or the server alone, or by the terminal and the server working together, so that... Figure 1A (or Figure 1B The following description uses the example of server 200 in the embodiment of this application executing the training method of the image segmentation model separately. See also Figure 3 , Figure 3 This is a flowchart illustrating the training method of the image segmentation model provided in this application embodiment, which will be combined with... Figure 3 The steps shown are explained.

[0046] In step 101, the server obtains a joint training sample set, which includes a first image sample without segmentation labels and at least one second image sample with segmentation labels, the segmentation label being the standard segmentation image of the second image sample.

[0047] In practice, the first image sample is an image without segmentation labels, and the second image sample is an image with segmentation labels. The segmentation labels are the correct standard segmentation images of the second image sample. In the standard segmentation image, each pixel has a corresponding label, which indicates the category to which the pixel belongs. Pixels with the same label share certain visual characteristics (such as the same color). See also Figure 4 , Figure 4 This is a schematic diagram of a second image sample carrying segmentation labels provided in an embodiment of this application. The second image sample is shown in number 1, and the standard segmentation image of the second image sample is shown in number 2. Objects of the same category in the standard segmentation image are identified by the same visual characteristics, such as "vehicle" shown in number 2-1 being identified by "black" and "sidewalk" shown in number 2-2 being identified by "white".

[0048] In step 102, an initial image segmentation model is obtained, and the first image sample is segmented using the initial image segmentation model to obtain a first segmentation result. Then, each second image sample is segmented to obtain a second segmentation result.

[0049] In practice, the server can refine the initial image segmentation model based on the acquired first image sample and multiple second image samples. F Training is then performed. This includes the initial image segmentation model. F It is an image segmentation model that has been pre-trained on a publicly available image dataset (such as the COCO dataset), not an untrained image segmentation model. This improves the training efficiency of the initial image segmentation model. The pre-training process for the initial image segmentation model is actually semantic segmentation training on a publicly available dataset based on cross-entropy loss.

[0050] For the initial image segmentation model F To explain, in actual implementation, the initial image segmentation model FIt can include an encoding layer, a decoding layer, an embedding layer, and a classification layer. The encoding layer encodes (downsamples) the image samples to obtain a compressed image (i.e., dimensionality reduction of the image samples through the encoding layer); the decoding layer decodes (upsamples) the compressed image to obtain a reconstructed image of the same size as the image samples (i.e., dimensionality increase of the compressed image through the decoding layer); the embedding layer performs semantic mapping on the reconstructed image to obtain a semantic image; and the classification layer (also called the pixel classification layer) performs pixel-by-pixel segmentation on the semantic image to obtain a segmentation result, which is then compared with... Figure 4 The standard segmentation images shown are similar, meaning that objects of the same category in the segmentation results use the same visual feature identifiers. It should be noted that the reason for using an embedding layer for semantic mapping is that during the decoding process, some irrelevant features in the compressed image may be amplified, or an important feature may be scattered. Therefore, the embedding layer maps the reconstructed image to a new space, resulting in a semantically clearer image. Furthermore, in practical applications, the embedding layer can also be used as part of the classification layer. When training the initial image segmentation model using a joint training sample set from the target domain, the first image sample from the joint training samples is input into the initial image segmentation model. After processing through the aforementioned encoding layer, decoding layer, embedding layer, and classification layer, the first segmentation result is obtained. Simultaneously, at least one second image sample from the joint training samples is input into the initial image segmentation model, and after processing through the encoding layer, decoding layer, embedding layer, and classification layer, the corresponding second segmentation result is obtained. The server processes the above process in parallel or serial mode depending on the actual availability of computing resources. If available computing resources are sufficient, parallel processing can be used; if available computing resources are below a preset threshold, serial processing can be used. For example, Figures 5A-5B This is a model structure diagram of the initial image segmentation model provided in the embodiments of this application. See [link / reference]. Figure 5A The initial image segmentation model shown in the figure is a single-branch serial structure, including an encoding layer, a decoding layer, an embedding layer, and a classification layer. The server sequentially inputs the first image sample and the second image sample into the initial image segmentation model and performs image segmentation serially. Parameters are shared throughout the process (if the first image sample is processed first, the parameters of the initial image segmentation model after processing the first image sample can be used as the initial parameters of the initial image segmentation model when processing the second image sample). See also Figure 5B The initial image segmentation model shown in the figure is a two-branch parallel structure. Each branch includes an encoding layer, a decoding layer, an embedding layer, and a classification layer. The branch structure shown in number 1 is used to process the second image sample, and the branch structure shown in number 2 is used to process the first image sample. The server uses a parallel approach to implement the segmentation operation for the first and second image samples. During the processing, the parameters of the corresponding layers in the two branch structures can be shared.

[0051] The pre-training process of the initial image segmentation model is explained. To improve the segmentation accuracy of the image segmentation model, the server can use public datasets (such as the COCO dataset) to train the initial image segmentation model. F Model pre-training, the initial image segmentation model in the pre-training process is denoted as , and F Same model structure, different model parameters 。 exist During model pre-training, the training loss can be determined using a pre-set loss function (such as the cross-entropy loss function), and updates can be made based on the training loss. The parameters are specified. The image sample set used for model pre-training is selected from a publicly available dataset. The input image is The corresponding segmentation label is The output predicted segmentation result is ,in, It is the height and width of the image. This represents the number of image samples in the image sample set D. Initial image segmentation model. F or The mapping relationship is shown in formula (1): (1) in, These are the model parameters for the initial image segmentation model. It is the coding layer. It is the decoding layer. It is a classification layer. For the coding layer in the image segmentation model For image samples The resulting compressed image is obtained through encoding processing. Decoding layer in an image segmentation model Decode the compressed image to obtain the image sample. Reconstructed images of the same size are generated; then, semantic mapping is performed on the reconstructed images through an embedding layer to obtain semantic images. For the classification layer in the image segmentation model The segmentation result obtained by performing pixel-by-pixel segmentation on the semantic image is that, for each category in the image sample, each pixel in the semantic image corresponds to a probability value. The category indicated by the highest probability value is taken as the target category to which the pixel belongs. The loss of the initial image segmentation model can be determined by one or more pre-defined loss functions. Since the image samples in the public dataset are labeled samples, the loss function during model pre-training can include the cross-entropy loss function. That is, the training loss during model pre-training of the initial image segmentation model is determined based on the cross-entropy loss function. The formula is as follows: (2) in, Let cross-entropy be the loss function. The initial image segmentation model is based on image samples. The segmentation result after image segmentation. The corresponding standard segmentation results. After the above multiple rounds of training, an initial image segmentation model with basic image segmentation capabilities is obtained. It should be noted that the model structure of the initial image segmentation model can also be in other forms, and this embodiment does not limit it.

[0052] In practice, the server obtains an initial image segmentation model based on a publicly available dataset pre-trained. F Afterwards, the initial image segmentation model can be further refined using a joint training sample set. F Formal model training is then conducted. In each round of training, the initial image segmentation model is used... F The process of segmenting a first image sample from the joint training sample set involves a first loss; the process of segmenting one or more second image samples from the joint training sample set using the initial image segmentation model involves a second loss. The first loss characterizes the value of the first loss function, and the second loss characterizes the value of the second loss function. The server can train the initial image segmentation model based on the first and second losses corresponding to each joint training sample in the joint training sample set. F， The final target image segmentation model is obtained.

[0053] In some embodiments, the output accuracy of the encoding and decoding layers of the initial image segmentation model pre-trained on a large number of labeled public datasets is high. To reduce the consumption of server computing resources and improve the training efficiency of the model based on real-world project datasets in the target domain, the server can first fine-tune the classification layer of the initial image segmentation model using multiple second image samples from the joint training sample set to obtain an image segmentation model suitable for the target domain. , and FThe model structures are the same, with identical parameters for the encoding and decoding layers, but different parameters for the classification layer. Then, in order to improve... For the segmentation accuracy of a small number of first image samples in the target domain (such as image samples of deformed target objects), the segmentation can be further improved based on the first image samples. Training can be conducted using methods such as: server fixed coding layer and decoding layer The parameters are based on the first image sample. ,right Classification layer in conduct The next iteration update yields At this time, the image segmentation model Updated to ,Right now .in, The model structures are the same, the model parameters for the encoding and decoding layers are the same, but the model parameters for the classification layer are different. For the first image sample, the integer is a positive integer greater than zero. When the first The prediction results With the The prediction results When the similarity reaches a similarity threshold, or when both are identical, the classification layer... Training is complete. (Regarding the classification layer...) During the iterative update process, The loss includes a second loss corresponding to multiple second image samples and a first loss corresponding to the first image samples. For example, taking the first image sample as an example... The classification layer obtained in the next iteration Initial classification layer For example, to illustrate from Iterative updates to The process. First, through... Predict each second image sample The second segmentation result During iterative training of the model, the second loss in each round is... ; Model predicts the first image sample First segmentation result During model iterative training, the first loss in each round is: ,in, , Indicates the first Pixels through The prediction result obtained by making the prediction (i.e., the first prediction) The probability value of a pixel belonging to the corresponding category. It is the first The next iteration The model corresponds to the prediction result. The server trains the image segmentation model using the first loss and the second loss. The target image segmentation model is obtained.

[0054] It's important to note that the initial image segmentation model pre-trained on a public dataset can be transferred to specific target domains (such as device detection in industrial automation or lesion detection in medical images). The initial image segmentation model can then be retrained on a target image dataset specific to that domain, resulting in an image segmentation model suitable for that domain. This target image dataset consists of N (N≥1 and N is a positive integer) labeled images, each carrying a segmentation label. Essentially, the pre-training process for the initial image segmentation model is based on a public dataset, while the retraining of the pre-trained model is based on a target image dataset specific to the target domain. The main difference between these two training processes lies in the source of the image samples. In essence, it involves transferring the initial image segmentation model trained on a public dataset to the target domain's image dataset. This significantly improves the training efficiency of the image segmentation model in the target domain, enabling the rapid acquisition of a trained image segmentation model.

[0055] The first and second segmentation results are explained below. The first segmentation result is a segmented image of the first image sample, also known as a segmentation mask. Regions of the same category in the segmentation mask use the same visual features (e.g., regions of the same category are identified by the same color). For example, assuming only the foreground and background of the image are segmented, two colors can be seen on the segmentation mask, such as black representing the background and white representing the foreground. If different categories in the image are distinguished, each category includes its own outline and its region in the segmentation mask (or its area or area ratio on the segmentation mask). In other words, visually appealing segmentation results are typically distinguished by different colors, while segmentation results read by electronic devices are often processed at the pixel level. For example, an H×W image can be viewed as an H-row, W-column pixel matrix, where each pixel includes a value representing its color. Correspondingly, the segmentation result for an image of this size can also be viewed as an H-row, W-column pixel matrix. For ease of calculation, pixel-related data can be stored in various data formats (such as classes, structs, tuples, etc.). A tuple could be as follows: {pixel row, pixel column, target probability value of the pixel, pixel category, color of the category}. Here, the pixel probability value represents the probability that the pixel belongs to the corresponding category. For example, if the segmentation result represents an image with 5 categories, each pixel has 5 probability values, each corresponding to the probability that the pixel belongs to the corresponding category. The sum of the 5 probability values ​​is 1, and the largest probability value among the 5 is taken as the target probability value of the pixel in the above tuple.

[0056] In step 103, a probability map of the first segmentation result in the pixel dimension is obtained, and the value of the first loss function is determined based on the probability map. The probability map is used to indicate the probability that each pixel in the first image sample belongs to each segmentation region in the first segmentation result.

[0057] It should be noted that after the server pre-trains the initial image segmentation model based on the public dataset, there is a first loss when performing the segmentation process on the first image sample in the joint training sample set using the initial image segmentation model; there is a second loss when performing the segmentation process on one or more second image samples in the joint training sample set using the initial image segmentation model. The first loss characterizes the value of the first loss function, and the second loss characterizes the determination of the second loss function. The server can train the initial image segmentation model based on the first loss and the second loss included in each joint training sample in the joint training sample set. The first loss function and the first loss are explained in detail in this step, and the second loss function and the second loss are explained in subsequent step 104.

[0058] In practice, since the first image sample does not carry a segmentation label, the accuracy of the region area of ​​each category shown in the first segmentation result of the first image sample is low. In order to obtain a more accurate region corresponding to the category, the server can determine the first loss of the first image sample based on the probability map corresponding to the first segmentation result.

[0059] The probability map is explained as follows: For the first segmentation result, the server determines the probability map of the current first segmentation result from the pixel dimension. The size of the probability map is the same as the size of the first segmentation result. As described above, each pixel in the first segmentation result has a corresponding target probability value. The server obtains the target probability values ​​of each pixel pixel by pixel, thus forming the probability map corresponding to the first segmentation result. For example, see [link to example]. Figure 6 , Figure 6 This is a schematic diagram of a probability map provided in an embodiment of this application. The diagram shows a 5×5 probability map, where the value at each pixel represents the probability that the pixel belongs to the target category. It should be noted that each probability value in the current probability map is specific to itself relative to each category in the image, and the probability values ​​in the probability map are completely independent of each other. The server obtains a first loss function based on the pixel dimension and determines the value of the first loss function, i.e., the first loss, based on each probability value in the probability map. It can be understood that the first loss function is used to constrain the area of ​​the corresponding category in the first segmentation result from the pixel dimension; the smaller the value of the first loss function, the more accurate the area of ​​each category in the segmentation result.

[0060] The method for determining the value of the first loss function will continue to be explained. In some embodiments, see [link to documentation]. Figure 7 , Figure 7 This is a flowchart illustrating the method for determining the value of the first loss function provided in an embodiment of this application, based on... Figure 3 Step 103, which involves obtaining the value of the first loss function based on the probability graph, can be implemented by steps 201-205.

[0061] Step 201: The server obtains the size of the first image sample, including its length and width.

[0062] In practice, the server reads the size of the first image sample (including length and width). It should be noted that, in terms of pixels, the first image sample is actually a pixel matrix. The number of rows in the pixel matrix is ​​the width of the first image sample, and the number of columns in the pixel matrix is ​​the height of the first image sample. For example, a first image sample of size H×W can be regarded as a pixel matrix of H rows and W columns.

[0063] Step 202: Obtain the historical probability map corresponding to the first image sample.

[0064] In practice, to obtain the first loss of the first segmentation result in the current training epoch of the initial image segmentation model, it is also necessary to obtain the probability map of the first image sample obtained from the previous training of the initial image segmentation model, i.e., the historical probability map. This allows for a more accurate determination of the region area of ​​each category in the first segmentation result from a pixel perspective. The method for obtaining the historical probability map is related to the training epoch of the initial image segmentation model.

[0065] The method for obtaining historical probability graphs is explained below. In some embodiments, see [link to documentation]. Figure 8 , Figure 8 This is a schematic diagram illustrating the method for obtaining historical probability maps provided in this application embodiment, combined with... Figure 8 The steps shown will be implemented in 2021-2022.

[0066] Step 2021: When the initial image segmentation model has completed its first training round, the server randomly perturbs the probability map to obtain a random probability map as the historical probability map.

[0067] In practice, the first round of training of the initial image segmentation model yields the probability map corresponding to the first segmentation result of the first image sample. To determine the value of the first loss function corresponding to the first segmentation result, a perturbation factor can be determined through random perturbation (such as scaling down, increasing the probability value of odd rows, decreasing the probability value of even rows, etc., with the aim of obtaining a probability map different from the probability map obtained in the first round of training). Based on these perturbation factors, the probability values ​​corresponding to the pixels in the probability map corresponding to the first segmentation result are randomly perturbed to obtain a random probability map as the historical probability map.

[0068] Step 2022: When the training round of the initial image segmentation model is not the first round, obtain the probability map corresponding to the first image sample obtained in the previous round of training the initial image segmentation model as the historical probability map.

[0069] In practice, after the server performs non-first round training on the initial image segmentation model, the server can obtain the probability map of the first image sample obtained in the previous training round for the current training round, as a historical probability map.

[0070] Step 203: Based on the size and probability map, determine the first area occupied by the foreground in the first image sample in the probability map.

[0071] In practice, the first image sample is used to constrain the area of ​​each category in the segmentation result during training. Since a generalized distinction based on categories—that is, for the first segmentation result—can distinguish only between foreground and background without further distinguishing specific categories within the foreground, the server can determine the area occupied by the foreground in the probability map (i.e., the first area) through the probability map and its size. This means obtaining the size of the foreground region, which is essentially the area of ​​all other regions in the segmentation result besides the background in the probability map.

[0072] The method for determining the first area is described below. In some embodiments, see [link to documentation]. Figure 9 , Figure 9 This is a flowchart illustrating the method for determining the first area provided in an embodiment of this application, combined with... Figure 9 Steps 2031-2032 are explained below.

[0073] Step 2031: The server normalizes the probability map based on the size to obtain a standard probability map.

[0074] In practice, since the probability value of each pixel in the probability map is independently determined and there is no correlation between the probability values ​​of pixels, the server can establish the correlation between each pixel through normalization in order to calculate the value of the loss function within a unified area space. The server, taking into account the size, performs a normalization operation on the probability map where the pixel probability values ​​are uncorrelated, resulting in a standard probability map. In this standard probability map, the sum of the new probability values ​​of all pixels (derived from the transformation of the original probability values) equals 1, which can also be understood as the area of ​​the standard probability map being 1. The specific normalization method is as follows: (3) In the above formula, H×W represents the size of the probability graph, i.e., it contains H rows and W columns. This represents the probability value of each pixel in the probability map. It's understandable that the normalization operation on the probability map is performed pixel-by-pixel. After normalization, the sum of the new probability values ​​of all pixels in the standard probability map is 1. See also... Figure 10 , Figure 10 This is a schematic diagram of the normalization results provided in the embodiments of this application.

[0075] Step 2032: Based on the standard probability map, determine the area occupied by the foreground in the standard probability map as the first area.

[0076] In practice, based on the standard probability map obtained by the above normalization method, the normalized probability values ​​of the pixels belonging to the foreground in the standard probability map are obtained and summed to obtain the area occupied by the foreground in the standard probability map, i.e., the first area.

[0077] Step 204: Based on the size and the historical probability map, determine the second area occupied by the foreground in the historical probability map.

[0078] Accordingly, in some embodiments, the server may also determine the second area by: the server normalizing the historical probability map in combination with the size to obtain a historical standard probability map; and based on the historical standard probability map, determining the area occupied by the foreground in the historical standard probability map as the second area.

[0079] In practice, the historical standard probability map corresponding to the historical probability map can be obtained using the normalization method described above. The normalized probability values ​​of the foreground pixels in the historical standard probability map are then obtained and summed to obtain the area occupied by the foreground in the historical standard probability map, i.e., the second area. There is a difference between the first area and the second area.

[0080] Step 205: Determine the value of the first loss function based on the first area and the second area.

[0081] In practice, the server determines the value of the first loss function by the ratio of the first foreground area to the second foreground area. The aim is to stabilize this ratio through multiple iterations to obtain the accurate area for each category. It should be noted that the formula for determining the value of the first loss function (i.e., the first loss) is as follows: (4) In the above formula, j For the number of training rounds, j ≥2, j It is an integer. For the first j It's time to get the first area of ​​the foreground. For the first j -1 round (previous round) yields the second area of ​​the foreground. During model training, try to make... It should be noted that in the actual calculation process, the probability value of pixels belonging to the background can be set to zero. This is understandable. It is used to refine the area of ​​each category in the probability map in the segmentation prediction results of the previous round, and to achieve accurate segmentation in the pixel dimension through continuous iteration, so as to determine the accurate area of ​​each category in the probability map.

[0082] Continuing with the explanation of how the value of the first loss function is determined, in some embodiments, the server can determine the value of the first loss function in the following way: the server obtains the ratio of the first area to the second area; the server performs a logarithmic operation on the ratio to obtain the logarithmic result; and the server determines the product of the logarithmic result and the first area as the value of the first loss function.

[0083] In actual implementation, the server determines the value of the first loss function according to the above formula (4). First, it obtains the ratio of the first area to the second area, then calculates the logarithm (log) of the ratio, and multiplies the result of the log operation with the first area to obtain the corresponding value of the first loss function.

[0084] The method for determining the value of the first loss function will continue to be explained. In some embodiments, see [link to documentation]. Figure 11 , Figure 11 This is another schematic diagram illustrating the method for determining the value of the first loss function provided in the embodiments of this application, combined with... Figure 11 Steps 301-306 shown are explained.

[0085] Step 301: The server obtains the size of the first image sample, including its length and width.

[0086] In practical implementation, as described above, the size of the first image sample can be used to identify the number of rows and columns of the pixel matrix. The number of rows indicates the width, and the number of columns indicates the length.

[0087] Step 302: Obtain the historical probability map corresponding to the first image sample.

[0088] In actual implementation, the server obtains the historical probability map of the first image sample according to the aforementioned method of obtaining the historical probability map.

[0089] Step 303: Based on the size and probability map, determine the third area occupied by each category in the probability map.

[0090] In practice, the foreground of the image may include multiple different categories, each corresponding to a region of pixels. Therefore, the server can also determine the area (i.e., the third area) occupied by each category in the probability map based on the size and probability map. The method for obtaining the third area of ​​each category based on the size and probability map is similar to step 203 mentioned above, and will not be repeated here.

[0091] Step 304: Based on the size and historical probability map, determine the fourth area occupied by each category in the historical probability map.

[0092] In practice, the foreground of the image may include multiple different categories, each corresponding to a region of pixels. Therefore, the server can also determine the area (i.e., the fourth area) occupied by each category in the historical probability map based on the size and historical probability map. The method for obtaining the fourth area of ​​each category based on the size and historical probability map is similar to step 204 above, and will not be repeated here.

[0093] Step 305: Based on the third area and the corresponding fourth area of ​​each category, determine the sub-value of each category relative to the first loss function.

[0094] In actual implementation, the server uses the formula (4) above to calculate the value (or sub-value) of each category relative to the first loss function based on the third area of ​​each category in the probability graph and the fourth area of ​​each category in the historical probability graph. One category corresponds to one sub-value of the first loss function. For example, if the first segmentation result has 5 categories, then there are 5 sub-values ​​of the first loss function.

[0095] Step 306: Obtain the weight of each category, and perform a weighted sum based on the sub-values ​​of each category. Use the weighted sum as the value of the first loss function.

[0096] In practice, the server obtains the corresponding sub-values ​​of the first loss function for each category and obtains the weights set for each category according to the actual situation (in this way, larger weights can be set for categories that need special attention). Then, based on the weights, the sub-values ​​of each category are weighted and summed, and the weighted sum is used as the value of the first loss function. It should be noted that the weight of each category can be set to 1.

[0097] In some embodiments, the server may determine the sub-value of each category in the first image sample relative to the first loss function in the following manner: For each category, the server performs the following processing: The server determines the ratio of the third area of ​​the current category to the corresponding fourth area; the server performs a logarithmic operation on the comparison value and determines the product of the logarithmic result and the third area of ​​the category, and uses the product as the sub-value of the current category relative to the first loss function.

[0098] In practice, the server determines the ratio of the third area to the fourth area for each category, then takes the logarithm of the ratio, and multiplies the logarithm result by the third area of ​​the category as a sub-value of the first loss function.

[0099] In step 104, the difference between each second segmentation result and the corresponding segmentation label is obtained, and the value of the second loss function is determined based on the difference.

[0100] In practice, the second loss function is used to constrain the contours of each category in the image segmentation result. The smaller the value of the second loss function, the more accurate the contours of each category in the image segmentation result. The second loss function is the loss function used in the fully supervised (also known as supervised) model training process, and it is determined based on the segmentation results of the image samples carrying segmentation labels (i.e., the second image samples) and the corresponding segmentation labels.

[0101] The method for determining the value of the second loss function is explained below. In some embodiments, the server may determine the value of the second loss function by: obtaining the cross-entropy between each second segmentation result and the corresponding segmentation label; summing each cross-entropy to obtain a summation result; obtaining the number of second image samples, and determining the ratio of the summation result to the number of second image samples as the value of the second loss function.

[0102] In practice, since the second image sample carries segmentation labels, supervised training can be used during the image segmentation model training process. Therefore, the cross-entropy function can be used as the loss function for training the second image sample. The server obtains the cross-entropy between the second segmentation result and the segmentation label of the second image sample. Since the number of second image samples input in each training round can be one or more, the average value of the cross-entropy corresponding to all second image samples in the current training round can be determined as the value of the second loss function.

[0103] For example, the value of the second loss function is determined for each second image sample. The formula is as follows: (5) Where N is the number of second image samples, N≥1, N=1, The image segmentation model training method provided in this application embodiment, which uses cross-entropy loss, can also obtain accurate data. In other words, this method can be applied to situations with a small number of samples.

[0104] In step 105, the initial image segmentation model is trained by combining the value of the first loss function and the value of each second loss function to obtain the target image segmentation model.

[0105] In practice, the server can constrain the area of ​​each category in the segmentation result of the first image sample without segmentation labels based on the first loss function, making the area loss close to zero. Based on the second loss function, the server can constrain the contour of each category in the segmentation result of the second image sample with segmentation labels, making the contour loss close to zero. Therefore, during the training process of the initial image segmentation model, the server can combine the values ​​of the first loss function and the second loss function to jointly train the initial segmentation model. In this way, even when there are few image samples (small samples), accurate segmentation can be achieved.

[0106] The joint training process for the initial image segmentation model is described below. In some embodiments, see [link to documentation]. Figure 12 , Figure 12 This is a schematic diagram of the training process method of the initial image segmentation model provided in the embodiments of this application, based on Figure 3 Step 105 can be implemented by steps 1051-1053.

[0107] Step 1051: The server obtains the weights of the first loss function and the weights of each second loss function.

[0108] In practice, for each training iteration of the initial image segmentation model, the input information includes at least one second image and one first image. Therefore, each second image sample has a corresponding value for a second loss function during training. Weights can be assigned to both the first and second loss functions. Assuming there are N (N≥1 and N is an integer) second image samples, in one training iteration, N values ​​for the second loss function (referred to as the second loss) and one value for the first loss function (referred to as the first loss) are generated. These N+1 losses each have corresponding weights. When N=1, the weights of the first and second losses can be equal, and the weights of the N second losses can also be equal. The server can determine the total loss for the current training iteration based on each loss and its corresponding weight.

[0109] The method for determining the value of the first loss function is explained. In some embodiments, the server may determine the weight of the first loss function in the following way: the server obtains the initial weight of the first loss function and the number of second image samples; the initial weight is adjusted according to the number to obtain the weight of the first loss function; wherein, the weight of the first loss function is negatively correlated with the number.

[0110] In practice, the server obtains the initial weights of the first loss function and adjusts them based on the number of second image samples to obtain the weights of the first loss function. It can be understood that during training, the weights of the first loss function are related to the number of second image samples. That is, the weight of the first loss function during training decreases as the number of second image samples increases. The more second image samples there are, the greater the proportion of the second loss function, ensuring that the value of the second loss function (which affects the accuracy of the contour) used for segmentation dominates, while the value of the first loss function, which limits the area, serves as an auxiliary.

[0111] Step 1052: Based on the weights of the first loss function and the weights of each second loss function, perform a weighted summation of the values ​​of the first loss function and each second loss function to obtain a weighted summation result.

[0112] In practice, the server combines the weights of the first loss function and the weights of each second loss function, and performs a weighted summation of the values ​​of the first loss function and each second loss function to obtain the weighted summation result as the total loss for each training session.

[0113] For example, assuming that each training input consists of N second image samples and one first image sample, the value of the first loss function is determined based on the above formula (3). Based on the above formula (5), the value of the second loss function is determined to be... One possible total loss The formula for determining it is as follows: (6) in, 1 represents the weight of the first loss function, and 1 represents the weight of each second loss function.

[0114] Step 1053: Based on the weighted summation result, update the model parameters of the initial image segmentation model to obtain the target image segmentation model.

[0115] In practice, the server determines the total loss using the formula (6) above during each training round. (i.e., the weighted summation result) is used to update the model parameters of the initial image segmentation model in reverse, that is, to train the initial image segmentation model and obtain the target segmentation model. The training of the target segmentation model can be achieved by only a small amount of data annotation (annotation of less than 10 images, with a minimum of only 1 image).

[0116] In some embodiments, the server can perform image segmentation operations on the image to be segmented in the following ways: the server obtains the image to be segmented; the image to be segmented is input into the target image segmentation model, and the segmentation result is output.

[0117] In practice, the server trains the initial image segmentation model using a small number of second image samples carrying segmentation labels and first image samples in the target domain that do not carry segmentation labels. After obtaining the trained target image segmentation model, the server can obtain the corresponding segmentation result by receiving the image to be segmented in the target domain through the target image segmentation model.

[0118] For example, in the field of industrial vision automated inspection, the server receives an image to be segmented containing a target device that is deformed. The server obtains a target image segmentation model trained on a second image with a small amount of labeling information (i.e., segmentation labels) based on the field and a first image that is not deformed and is not labeled. The server then identifies the received image to be segmented and can accurately label the outline and region of the target device on the segmentation result (segmentation mask).

[0119] Applying the above embodiments of this application, firstly, based on the idea of ​​transfer learning, the initial image segmentation model is pre-trained on a publicly available large-scale dataset; then, a small amount of real-world project data is used to fine-tune the pre-trained model, thereby improving the efficiency of training the model. At the same time, the initial image segmentation model is jointly trained by constraining the second loss determined by at least one second image sample with segmentation labels using the first loss determined by the first image sample without segmentation labels. Thus, only a small amount of data is needed to achieve high region segmentation performance, improving the localization of target objects in images under target object deformation scenarios.

[0120] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0121] Related technologies employ image matching-based target localization and deep learning-based semantic segmentation localization to locate targets in images. Image matching-based localization extracts the object's contour as a matching feature; contour extraction is obtained by calculating and analyzing image gradients. Therefore, this method suffers from poor accuracy in complex real-world scenarios. While deep learning-based semantic segmentation localization achieves high accuracy in complex situations, it requires extensive image data annotation, resulting in significant annotation costs.

[0122] Based on this, this application provides a training method for an image segmentation model, which is also a region localization method based on few-sample semantic segmentation. This method can achieve localization in scenarios with object deformation and requires only a small amount of data annotation (less than 10 images, with a minimum of only 1 image). First, based on the idea of ​​transfer learning, the model (i.e., the image segmentation model mentioned above) is pre-trained on a publicly available large-scale dataset; then, a small amount of real-world project (specific application scenario) data (i.e., few samples) is used to fine-tune the pre-trained model. Furthermore, this application proposes two loss mechanisms for few-sample fine-tuning training, enabling the network to achieve high region segmentation performance with only a small amount of data.

[0123] Next, from the product side, it will be explained that in current industrial vision-based automated inspection systems, accurate location of target devices plays a crucial role in subsequent detection, recognition, classification, and measurement steps. Therefore, the target device localization module is widely used as a fundamental module. This technology can be applied to the stable localization of target devices under varying lighting, rotation, and deformation conditions, requiring only a small amount of labeled data. The image segmentation model in this embodiment, as a region localization model based on few-shot semantic segmentation, is first pre-trained on a public dataset. Even with very little training data in practical applications, the model still achieves high segmentation accuracy. When using this few-shot segmentation model, users only need to collect a small amount of labeled data (less than 10 images, with a minimum of only 1 image) for training, and the product can complete the segmentation and localization of online image targets.

[0124] Next, combine again Figure 5B Taking the image segmentation model structure shown and the image segmentation model F shown in the aforementioned formula (1) as an example, we will explain the model pre-training and model training of the image segmentation model.

[0125] First, in the model pre-training stage, an image sample set is selected from the publicly available COCO dataset to pre-train the image segmentation model F. During the pre-training process, since the image samples in the image sample set carry segmentation labels, the cross-entropy loss of each image sample can be determined using the aforementioned formula (2) to perform semantic segmentation training on the image segmentation model. After model pre-training for the image segmentation model, an initial image segmentation model pre-trained on the publicly available dataset is obtained. Then, during the model training phase, the image segmentation model is reinitialized. Classification layer in To obtain the image segmentation model , The model structure is the same, the parameters of the encoding and decoding layers are the same, but the parameters of the classification layer are different, and the image sample set of the actual project is used. ,train Among them, the project dataset Include A carrying a segmentation label Image samples (i.e., the second image sample mentioned above), through the image segmentation model For each image sample Perform image segmentation to obtain the corresponding segmentation results. The corresponding cross-entropy loss is obtained. (i.e., the second loss mentioned above), the image sample set is calculated using the aforementioned formula (2). The loss, namely: Next, for the image to be segmented... (i.e., the first image sample mentioned above) Input image segmentation model In the process of image segmentation, it is necessary to consider the image segmentation model. conduct The next iteration update yields This leads to the trained image segmentation model. The loss in each iteration (i.e., the first loss mentioned above) can be obtained according to the aforementioned formula (4): , in, , Indicates the first Pixels through The prediction results obtained from the prediction process It is the first The next iteration Model corresponding results.

[0126] Therefore, the total iterative loss of the image segmentation model in one round of training is The image segmentation model is updated using this total loss to obtain the final image segmentation model.

[0127] Applying the above embodiments of this application, firstly, based on the idea of ​​transfer learning, the initial image segmentation model is pre-trained on a publicly available large-scale dataset; then, a small amount of actual project data is used to fine-tune the pre-trained model, thus improving the efficiency of training the model. At the same time, the initial image segmentation model is jointly trained by constraining the second loss determined by at least one second image sample with segmentation label through the first loss determined by the first image sample without segmentation label. In this way, only a small amount of data (less than 10 labeled images, with a minimum of only 1 labeled image) is needed to achieve high region segmentation performance, improving the localization of target objects in images under target object deformation scenarios.

[0128] The following description continues to illustrate the exemplary structure of the image segmentation model training device 555 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 3 As shown, the software modules in the training device 555 for the image segmentation model stored in the memory 550 may include:

[0129] The acquisition module 5551 is used to acquire a joint training sample set, which includes a first image sample without segmentation label and at least one second image sample with segmentation label, wherein the segmentation label is the standard segmentation image of the second image sample. The initial segmentation module 5552 is used to obtain an initial image segmentation model, and to perform image segmentation on the first image sample using the initial image segmentation model to obtain a first segmentation result, and to perform image segmentation on each of the second image samples to obtain a second segmentation result; The first determining module 5553 is used to obtain a probability map of the first segmentation result in the pixel dimension, and determine the value of the first loss function based on the probability map. The probability map is used to indicate the probability that each pixel in the first image sample belongs to each segmentation region in the first segmentation result. The second determining module 5554 is used to obtain the difference between each of the second segmentation results and the segmentation label, and to determine the value of the second loss function based on the difference; The training module 5555 is used to train the initial image segmentation model by combining the value of the first loss function and the value of each of the second loss functions to obtain the target image segmentation model.

[0130] In some embodiments, the first determining module is further configured to: obtain the size of the first image sample, the size including length and width; obtain a historical probability map corresponding to the first image sample; determine a first area occupied by the foreground in the first image sample in the probability map based on the size and the probability map; determine a second area occupied by the foreground in the historical probability map based on the size and the historical probability map; and determine the value of a first loss function based on the first area and the second area.

[0131] In some embodiments, the first determining module is further configured to: when the training round of the initial image segmentation model is the first round, randomly perturb the probability map to obtain a random probability map as the historical probability map; when the training round of the initial image segmentation model is not the first round, obtain the probability map corresponding to the first image sample obtained in the previous round of training the initial image segmentation model as the historical probability map.

[0132] In some embodiments, the first determining module is further configured to normalize the probability map in conjunction with the size to obtain a standard probability map; and based on the standard probability map, determine the area occupied by the foreground in the standard probability map as the first area; In some embodiments, the first determining module is further configured to normalize the historical probability map in conjunction with the size to obtain a historical standard probability map; and based on the historical standard probability map, determine the area occupied by the foreground in the historical standard probability map as the second area.

[0133] In some embodiments, the first determining module is further configured to obtain the ratio of the first area to the second area; take the logarithm of the ratio to obtain the logarithmic result; and determine the product of the logarithmic result and the first area as the value of the first loss function.

[0134] In some embodiments, the foreground of the first image sample includes multiple categories. The first determining module is further configured to: obtain the size of the first image sample, the size including length and width; obtain a historical probability map corresponding to the first image sample; determine a third area occupied by each category in the probability map based on the size and the probability map; determine a fourth area occupied by each category in the historical probability map based on the size and the historical probability map; determine a sub-value of each category relative to a first loss function based on the third area and the corresponding fourth area of ​​each category; obtain the weight of each category; perform a weighted summation based on the sub-value of each category; and use the weighted summation result as the value of the first loss function.

[0135] In some embodiments, the first determining module is further configured to perform the following processing for each category: determining the ratio of the third area of ​​the category to the corresponding fourth area; taking the logarithm of the ratio and determining the product of the logarithm result and the third area of ​​the category, and using the product as a sub-value of the category relative to the first loss function.

[0136] In some embodiments, the training module is further configured to obtain the weights of the first loss function and the weights of each of the second loss functions; based on the weights of the first loss function and the weights of each of the second loss functions, perform a weighted summation on the values ​​of the first loss function and each of the second loss functions to obtain a weighted summation result; and based on the weighted summation result, update the model parameters of the initial image segmentation model to obtain the target image segmentation model.

[0137] In some embodiments, the training module is further configured to obtain the initial weights of the first loss function and the number of the second image samples; adjust the initial weights according to the number to obtain the weights of the first loss function; wherein the weights of the first loss function are negatively correlated with the number.

[0138] In some embodiments, the second determining module is further configured to obtain the cross-entropy between each of the second segmentation results and the corresponding segmentation label; sum each of the cross-entropies to obtain a summation result; obtain the number of the second image samples, and determine the ratio of the summation result to the number of the second image samples as the value of the second loss function.

[0139] In some embodiments, the trained target image segmentation model is further used to acquire the image to be segmented; input the image to be segmented into the target image segmentation model, and output the segmentation result obtained.

[0140] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the image segmentation model training method described in this application.

[0141] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the training method for the image segmentation model provided in this application. For example, ... Figure 7 The training method for the image segmentation model is shown.

[0142] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0143] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0144] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0145] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0146] In summary, the embodiments of this application have the following beneficial effects: First, based on the idea of ​​transfer learning, the initial image segmentation model is pre-trained on a publicly available large-scale dataset; then, the pre-trained model is fine-tuned using a small amount of real-world project data, which improves the efficiency of training the model. At the same time, the initial image segmentation model is jointly trained by constraining the second loss determined by at least one second image sample with segmentation label using the first loss determined by the first image sample without segmentation label. Thus, only a small amount of data (less than 10 labeled images, with a minimum of only 1 labeled image) is needed to achieve high region segmentation performance, improving the localization of target objects in images under target object deformation scenarios.

[0147] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for training an image segmentation model, characterized in that, The method includes: Obtain a joint training sample set, which includes a first image sample without segmentation labels and at least one second image sample with segmentation labels, wherein the segmentation label is the standard segmentation image of the second image sample; An initial image segmentation model is obtained, and the first image sample is segmented using the initial image segmentation model to obtain a first segmentation result. Then, each of the second image samples is segmented to obtain a second segmentation result. Obtain the probability map of the first segmentation result in the pixel dimension, and obtain the size of the first image sample, the size including length and width; Obtain the historical probability map corresponding to the first image sample; Based on the size and the probability map, determine the first area occupied by the foreground in the first image sample in the probability map; Based on the size and the historical probability map, determine the second area occupied by the foreground in the historical probability map; Based on the first area and the second area, the value of the first loss function is determined; the probability map is used to indicate the probability that each pixel in the first image sample belongs to each segmentation region in the first segmentation result; Obtain the difference between each of the second segmentation results and the segmentation label, and determine the value of the second loss function based on the difference; By combining the values ​​of the first loss function and each of the second loss functions, the initial image segmentation model is trained to obtain the target image segmentation model.

2. The method of claim 1, wherein, The step of obtaining the historical probability map corresponding to the first image sample includes: When the initial image segmentation model is in its first training round, the probability map is randomly perturbed to obtain a random probability map, which is then used as the historical probability map. When the training round of the initial image segmentation model is not the first round, the probability map corresponding to the first image sample obtained in the previous round of training the initial image segmentation model is used as the historical probability map.

3. The method of claim 1, wherein, Determining the first area occupied by the foreground in the first image sample in the probability map based on the size and the probability map includes: By taking into account the size, the probability map is normalized to obtain a standard probability map; Based on the standard probability map, the area occupied by the foreground in the standard probability map is determined to be the first area; Determining the second area occupied by the foreground in the historical probability map based on the size and the historical probability map includes: By combining the aforementioned size, the historical probability map is normalized to obtain a historical standard probability map; Based on the historical standard probability map, the area occupied by the foreground in the historical standard probability map is determined as the second area.

4. The method as described in claim 1, characterized in that, Determining the value of the first loss function based on the first area and the second area includes: Obtain the ratio of the first area to the second area; Take the logarithm of the ratio to obtain the logarithmic result; The product of the logarithm result and the first area is determined as the value of the first loss function.

5. The method as described in claim 1, characterized in that, The foreground of the first image sample includes multiple categories, and the method further includes: Based on the size and the probability map, determine the third area occupied by each category in the probability map; Based on the size and the historical probability map, determine the fourth area occupied by each category in the historical probability map; Based on the third area and the corresponding fourth area of ​​each category, a sub-value of each category relative to the first loss function is determined; Obtain the weight of each category, and perform a weighted sum based on the sub-values ​​of each category, using the weighted sum as the value of the first loss function.

6. The method as described in claim 5, characterized in that, Determining a sub-value of each category relative to the first loss function based on the third area and the corresponding fourth area for each category includes: For each of the categories, perform the following processing: Determine the ratio of the third area of ​​the category to the corresponding fourth area; The ratio is logarithmically calculated, and the product of the logarithmic result and the third area of ​​the category is determined. This product is then used as a sub-value of the category relative to the first loss function.

7. The method as described in claim 1, characterized in that, The step of training the initial image segmentation model by combining the values ​​of the first loss function and each of the second loss functions to obtain the target image segmentation model includes: Obtain the weights of the first loss function and the weights of each of the second loss functions; Based on the weights of the first loss function and the weights of each of the second loss functions, the values ​​of the first loss function and each of the second loss functions are summed in a weighted manner to obtain a weighted summation result. Based on the weighted summation result, the model parameters of the initial image segmentation model are updated to obtain the target image segmentation model.

8. The method as described in claim 7, characterized in that, Obtaining the weights of the first loss function includes: Obtain the initial weights of the first loss function and the number of the second image samples; Based on the quantity, the initial weights are adjusted to obtain the weights of the first loss function; The weights of the first loss function are negatively correlated with the quantity.

9. The method as described in claim 1, characterized in that, The step of obtaining the difference between each of the second segmentation results and the segmentation label includes: Obtain the cross-entropy between each of the second segmentation results and the corresponding segmentation label; Determining the value of the second loss function based on the difference includes: Summing each of the cross-entropies yields the summation result; The number of the second image samples is obtained, and the ratio of the summation result to the number of the second image samples is determined as the value of the second loss function.

10. The method as described in claim 1, characterized in that, The method further includes: Obtain the image to be segmented; The image to be segmented is input into the target image segmentation model, and the segmentation result is output.

11. A training device for an image segmentation model, characterized in that, The device includes: The acquisition module is used to acquire a joint training sample set, which includes a first image sample without segmentation labels and at least one second image sample with segmentation labels, wherein the segmentation label is a standard segmentation image of the second image sample. The segmentation module is used to obtain an initial image segmentation model, and to perform image segmentation on the first image sample using the initial image segmentation model to obtain a first segmentation result, and to perform image segmentation on each of the second image samples to obtain a second segmentation result; A first determining module is configured to: acquire a probability map of the first segmentation result in the pixel dimension; acquire the size of the first image sample, the size including length and width; acquire a historical probability map corresponding to the first image sample; determine a first area occupied by the foreground in the first image sample in the probability map based on the size and the probability map; determine a second area occupied by the foreground in the historical probability map based on the size and the historical probability map; and determine the value of a first loss function based on the first area and the second area. The probability map is used to indicate the probability that each pixel in the first image sample belongs to each segmentation region in the first segmentation result. The second determining module is used to obtain the difference between each of the second segmentation results and the segmentation label, and to determine the value of the second loss function based on the difference; The training module is used to train the initial image segmentation model by combining the value of the first loss function and the value of each of the second loss functions to obtain the target image segmentation model.

12. The apparatus according to claim 11, characterized in that, The first determining module is further configured to: when the training round of the initial image segmentation model is the first round, randomly perturb the probability map to obtain a random probability map as the historical probability map; when the training round of the initial image segmentation model is not the first round, obtain the probability map corresponding to the first image sample obtained in the previous round of training the initial image segmentation model as the historical probability map.

13. The apparatus according to claim 11, characterized in that, The first determining module is further configured to normalize the probability map by combining the size to obtain a standard probability map; determine the area occupied by the foreground in the standard probability map as the first area based on the standard probability map; normalize the historical probability map by combining the size to obtain a historical standard probability map; and determine the area occupied by the foreground in the historical standard probability map as the second area based on the historical standard probability map.

14. The apparatus according to claim 11, characterized in that, The first determining module is further configured to obtain the ratio of the first area to the second area; The logarithm of the ratio is taken to obtain the logarithmic result; the product of the logarithmic result and the first area is determined as the value of the first loss function.

15. The apparatus according to claim 11, characterized in that, The foreground of the first image sample includes multiple categories; The first determining module is further configured to determine a third area occupied by each category in the probability map based on the size and the probability map; determine a fourth area occupied by each category in the historical probability map based on the size and the historical probability map; and determine a sub-value of each category relative to the first loss function based on the third area and the corresponding fourth area of ​​each category. Obtain the weight of each category, and perform a weighted sum based on the sub-values ​​of each category, using the weighted sum as the value of the first loss function.

16. The apparatus according to claim 15, characterized in that, The first determining module is further configured to perform the following processing for each category: determine the ratio of the third area of ​​the category to the corresponding fourth area; take the logarithm of the ratio and determine the product of the logarithm result and the third area of ​​the category, and use the product as a sub-value of the category relative to the first loss function.

17. The apparatus according to claim 11, characterized in that, The training module is further configured to obtain the weights of the first loss function and the weights of each of the second loss functions; based on the weights of the first loss function and the weights of each of the second loss functions, perform a weighted summation on the values ​​of the first loss function and each of the second loss functions to obtain a weighted summation result; and based on the weighted summation result, update the model parameters of the initial image segmentation model to obtain the target image segmentation model.

18. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the training method of the image segmentation model according to any one of claims 1 to 10.

19. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the training method of the image segmentation model according to any one of claims 1 to 10.

20. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by the processor, the training method of the image segmentation model according to any one of claims 1 to 10 is implemented.

Images